Hand Gesture Recognition Using Compact CNN Via Surface Electromyography Signals

MVMD-TCCA: A method for gesture classification based on surface electromyographic signals

Gesture recognition plays a fundamental role in enabling nonverbal communication and interaction, as well as assisting individuals with motor impairments in performing daily tasks. Surface electromyographic (sEMG) signals, which can effectively detect and predict motor intentions, are integral to achieving accurate gesture classification. This paper proposes a method, the multivariate variational mode decomposition and the two-channel convolutional neural network with added attention mechanism (MVMD-TCCA), to enhance the accuracy of gesture classification for motor intention recognition. The MVMD technique is utilized to decompose and fuse sEMG signals, enriching signal content and improving feature representation. To further optimize gesture classification performance, the convolutional block attention module (CBAM) and CrissCross attention mechanism are integrated into the neural network, enabling superior learning of local and spatial features. The experimental results show that the MVMD-TCCA method achieves an average classification accuracy of 85.09 % on the NinaPro DB2 dataset, representing a 13.46 % improvement compared to the use of the original signal, and an average classification accuracy of 97.90 % on the dataset collected from 15 subjects, reflecting a 1.70 % improvement over the original signal. These findings underscore the critical role of accurate gesture classification in facilitating daily task assistance for cerebral infarction patients, demonstrating the potential of the proposed approach.

Multi-scale EMG classification with spatial-temporal attention for prosthetic hands

A classification framework for hand gestures using Electromyography (EMG) signals in prosthetic hands is presented. Leveraging the multi-scale characteristics and temporal nature of EMG signals, a Convolutional Neural Network (CNN) is used to extract multi-scale features and classify them with spatial-temporal attention. A multi-scale coarse-grained layer introduced into the input of one-dimensional CNN (1D-CNN) facilitates multi-scale feature extraction. The multi-scale features are fed into the attention layer and subsequently given to the fully connected layer to perform classification. The proposed model achieves classification accuracies of 93.4%, 92.8%, 91.3%, and 94.1% for Ninapro DB1, DB2, DB5, and DB7 respectively, thereby enhancing the confidence of prosthetic hand users.

Gesture recognition from surface electromyography signals based on the SE-DenseNet network

OBJECTIVES

In recent years, significant progress has been made in the research of gesture recognition using surface electromyography (sEMG) signals based on machine learning and deep learning techniques. The main motivation for sEMG gesture recognition research is to provide more natural, convenient, and personalized human-computer interaction, which makes research in this field have considerable application prospects in rehabilitation technology. However, the existing gesture recognition algorithms still need to be further improved in terms of global feature capture, model computational complexity, and generalizability.

METHODS

This paper proposes a fusion model of Squeeze-and-Excitation Networks (SE) and DenseNet, inserting attention mechanism between DenseBlock and Transition to focus on the most important information, improving feature representation ability, and effectively solving the problem of gradient vanishing.

RESULTS

This proposed method was tested on the electromyographic gesture datasets NinaPro DB2 and DB4, achieving accuracies of 85.93 and 82.39 % respectively. Through ablation experiments, it was found that the method based on DenseNet-101 as the backbone model produced the best results.

CONCLUSIONS

Compared with existing models, this proposed method has better robustness and generalizability in gesture recognition, providing new ideas for the development of sEMG signal gesture recognition applications in the future.

Development of a Wearable Electromyographic Sensor with Aerosol Jet Printing Technology

Electromyographic (EMG) sensors are essential tools for analyzing muscle activity, but traditional designs often face challenges such as motion artifacts, signal variability, and limited wearability. This study introduces a novel EMG sensor fabricated using Aerosol Jet Printing (AJP) technology that addresses these limitations with a focus on precision, flexibility, and stability. The innovative sensor design minimizes air interposition at the skin-electrode interface, thereby reducing variability and improving signal quality. AJP enables the precise deposition of conductive materials onto flexible substrates, achieving a thinner and more conformable sensor that enhances user comfort and wearability. Performance testing compared the novel sensor to commercially available alternatives, highlighting its superior impedance stability across frequencies, even under mechanical stress. Physiological validation on a human participant confirmed the sensor's ability to accurately capture muscle activity during rest and voluntary contractions, with clear differentiation between low and high activity states. The findings highlight the sensor's potential for diverse applications, such as clinical diagnostics, rehabilitation, and sports performance monitoring. This work establishes AJP technology as a novel approach for designing wearable EMG sensors, providing a pathway for further advancements in miniaturization, strain-insensitive designs, and real-world deployment. Future research will explore optimization for broader applications and larger populations.

A Multi-Scale CNN for Transfer Learning in sEMG-Based Hand Gesture Recognition for Prosthetic Devices

Advancements in neural network approaches have enhanced the effectiveness of surface Electromyography (sEMG)-based hand gesture recognition when measuring muscle activity. However, current deep learning architectures struggle to achieve good generalization and robustness, often demanding significant computational resources. The goal of this paper was to develop a robust model that can quickly adapt to new users using Transfer Learning. We propose a Multi-Scale Convolutional Neural Network (MSCNN), pre-trained with various strategies to improve inter-subject generalization. These strategies include domain adaptation with a gradient-reversal layer and self-supervision using triplet margin loss. We evaluated these approaches on several benchmark datasets, specifically the NinaPro databases. This study also compared two different Transfer Learning frameworks designed for user-dependent fine-tuning. The second Transfer Learning framework achieved a 97% F1 Score across 14 classes with an average of 1.40 epochs, suggesting potential for on-site model retraining in cases of performance degradation over time. The findings highlight the effectiveness of Transfer Learning in creating adaptive, user-specific models for sEMG-based prosthetic hands. Moreover, the study examined the impacts of rectification and window length, with a focus on real-time accessible normalizing techniques, suggesting significant improvements in usability and performance.

Fabric-based lamina emergent MXene-based electrode for electrophysiological monitoring

Commercial wearable biosignal sensing technologies encounter challenges associated with irritation or discomfort caused by unwanted objects in direct contact with the skin, which can discourage the widespread adoption of wearable devices. To address this issue, we propose a fabric-based lamina emergent MXene-based electrode, a lightweight and flexible shape-morphing wearable bioelectrode. This work offers an innovative approach to biosignal sensing by harnessing the high electrical conductivity and low skin-to-electrode contact impedance of MXene-based dry electrodes. Its design, inspired by Nesler's pneumatic interference actuator, ensures stable skin-to-electrode contact, enabling robust biosignal detection in diverse situations. Extensive research is conducted on key design parameters, such as the width and number of multiple semicircular legs, the radius of the anchoring frame, and pneumatic pressure, to accommodate a wide range of applications. Furthermore, a real-time wireless electrophysiological monitoring system has been developed, with a signal-to-noise ratio and accuracy comparable to those of commercial bioelectrodes. This work excels in recognizing various hand gestures through a convolutional neural network, ultimately introducing a shape-morphing electrode that provides reliable, high-performance biosignal sensing for dynamic users.

Hand gesture recognition using sEMG signals with a multi-stream time-varying feature enhancement approach

Hand gesture recognition based on sparse multichannel surface electromyography (sEMG) still poses a significant challenge to deployment as a muscle-computer interface. Many researchers have been working to develop an sEMG-based hand gesture recognition system. However, the existing system still faces challenges in achieving satisfactory performance due to ineffective feature enhancement, so the prediction is erratic and unstable. To comprehensively tackle these challenges, we introduce a novel approach: a lightweight sEMG-based hand gesture recognition system using a 4-stream deep learning architecture. Each stream strategically combines Temporal Convolutional Network (TCN)-based time-varying features with Convolutional Neural Network (CNN)-based frame-wise features. In the first stream, we harness the power of the TCN module to extract nuanced time-varying temporal features. The second stream integrates a hybrid Long short-term memory (LSTM)-TCN module. This stream extracts temporal features using LSTM and seamlessly enhances them with TCN to effectively capture intricate long-range temporal relations. The third stream adopts a spatio-temporal strategy, merging the CNN and TCN modules. This integration facilitates concurrent comprehension of both spatial and temporal features, enriching the model's understanding of the underlying dynamics of the data. The fourth stream uses a skip connection mechanism to alleviate potential problems of data loss, ensuring a robust information flow throughout the network and concatenating the 4 stream features, yielding a comprehensive and effective final feature representation. We employ a channel attention-based feature selection module to select the most effective features, aiming to reduce the computational complexity and feed them into the classification module. The proposed model achieves an average accuracy of 94.31% and 98.96% on the Ninapro DB1 and DB9 datasets, respectively. This high-performance accuracy proves the superiority of the proposed model, and its implications extend to enhancing the quality of life for individuals using prosthetic limbs and advancing control systems in the field of robotic human-machine interfaces.

Unveiling EMG semantics: a prototype-learning approach to generalizable gesture classification

Objective.Upper limb loss can profoundly impact an individual's quality of life, posing challenges to both physical capabilities and emotional well-being. To restore limb function by decoding electromyography (EMG) signals, in this paper, we present a novel deep prototype learning method for accurate and generalizable EMG-based gesture classification. Existing methods suffer from limitations in generalization across subjects due to the diverse nature of individual muscle responses, impeding seamless applicability in broader populations.Approach.By leveraging deep prototype learning, we introduce a method that goes beyond direct output prediction. Instead, it matches new EMG inputs to a set of learned prototypes and predicts the corresponding labels.Main results.This novel methodology significantly enhances the model's classification performance and generalizability by discriminating subtle differences between gestures, making it more reliable and precise in real-world applications. Our experiments on four Ninapro datasets suggest that our deep prototype learning classifier outperforms state-of-the-art methods in terms of intra-subject and inter-subject classification accuracy in gesture prediction.Significance.The results from our experiments validate the effectiveness of the proposed method and pave the way for future advancements in the field of EMG gesture classification for upper limb prosthetics.

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Objective.Recent studies have shown that integrating inertial measurement unit (IMU) signals with surface electromyographic (sEMG) can greatly improve hand gesture recognition (HGR) performance in applications such as prosthetic control and rehabilitation training. However, current deep learning models for multimodal HGR encounter difficulties in invasive modal fusion, complex feature extraction from heterogeneous signals, and limited inter-subject model generalization. To address these challenges, this study aims to develop an end-to-end and inter-subject transferable model that utilizes non-invasively fused sEMG and acceleration (ACC) data.Approach.The proposed non-invasive modal fusion-transformer (NIMFT) model utilizes 1D-convolutional neural networks-based patch embedding for local information extraction and employs a multi-head cross-attention (MCA) mechanism to non-invasively integrate sEMG and ACC signals, stabilizing the variability induced by sEMG. The proposed architecture undergoes detailed ablation studies after hyperparameter tuning. Transfer learning is employed by fine-tuning a pre-trained model on new subject and a comparative analysis is performed between the fine-tuning and subject-specific model. Additionally, the performance of NIMFT is compared to state-of-the-art fusion models.Main results.The NIMFT model achieved recognition accuracies of 93.91%, 91.02%, and 95.56% on the three action sets in the Ninapro DB2 dataset. The proposed embedding method and MCA outperformed the traditional invasive modal fusion transformer by 2.01% (embedding) and 1.23% (fusion), respectively. In comparison to subject-specific models, the fine-tuning model exhibited the highest average accuracy improvement of 2.26%, achieving a final accuracy of 96.13%. Moreover, the NIMFT model demonstrated superiority in terms of accuracy, recall, precision, and F1-score compared to the latest modal fusion models with similar model scale.Significance.The NIMFT is a novel end-to-end HGR model, utilizes a non-invasive MCA mechanism to integrate long-range intermodal information effectively. Compared to recent modal fusion models, it demonstrates superior performance in inter-subject experiments and offers higher training efficiency and accuracy levels through transfer learning than subject-specific approaches.

A Robust and Real-Time Framework of Cross-Subject Myoelectric Control Model Calibration via Multi-Source Domain Adaptation

Surface electromyogram (sEMG) has been widely used in hand gesture recognition. However, most previous studies focused on user-personalized models, which require a great amount of data from each new target user to learn the user-specific EMG patterns. In this work, we present a novel real-time gesture recognition framework based on multi-source domain adaptation, which learns extra knowledge from the data of other users, thereby reducing the data collection burdens on the target user. Additionally, compared with conventional domain adaptation methods which treat data from all users in the source domain as a whole, the proposed multi-source method treat data from different users as multiple separate source domains. Therefore, more detailed statistical information on the data distribution from each user can be learned effectively. High-density sEMG (256 channels) from 20 subjects was used to validate the proposed method. Importantly, we evaluated our method with a simulated real-time processing pipeline on continuous sEMG data stream, rather than well-segmented data. The false alarm rate during rest periods in an EMG data stream, which is typically neglected by previous studies performing offline analyses, was also considered. Our results showed that, with only 1 s sEMG data per gesture from the new user, the 10-gesture classification accuracy reached 87.66 % but the false alarm rate was reduced to 1.95 %. Our method can reduce the frustratingly heavy data collection burdens on each new user.

Classification of EMG signals with CNN features and voting ensemble classifier

Electromyography (EMG) signals are primarily used to control prosthetic hands. Classifying hand gestures efficiently with EMG signals presents numerous challenges. In addition to overcoming these challenges, a successful combination of feature extraction and classification approaches will improve classification accuracy. In the current work, convolutional neural network (CNN) features are used to reduce the redundancy problems associated with time and frequency domain features to improve classification accuracy. The features from the EMG signal are extracted using a CNN and are fed to the 'k' nearest neighbor (KNN) classifier with a different number of neighbors ( 1 N N , 3 N N , 5 N N , and   7 N N ) . It results in an ensemble of classifiers that are combined using a hard voting-based classifier. Based on the benchmark Ninapro DB4 database and CapgMyo database, the proposed framework obtained 91.3 % classification accuracy on CapgMyo and 89.5 % on Ninapro DB4.

Machine-learned wearable sensors for real-time hand-motion recognition: toward practical applications

Soft electromechanical sensors have led to a new paradigm of electronic devices for novel motion-based wearable applications in our daily lives. However, the vast amount of random and unidentified signals generated by complex body motions has hindered the precise recognition and practical application of this technology. Recent advancements in artificial-intelligence technology have enabled significant strides in extracting features from massive and intricate data sets, thereby presenting a breakthrough in utilizing wearable sensors for practical applications. Beyond traditional machine-learning techniques for classifying simple gestures, advanced machine-learning algorithms have been developed to handle more complex and nuanced motion-based tasks with restricted training data sets. Machine-learning techniques have improved the ability to perceive, and thus machine-learned wearable soft sensors have enabled accurate and rapid human-gesture recognition, providing real-time feedback to users. This forms a crucial component of future wearable electronics, contributing to a robust human-machine interface. In this review, we provide a comprehensive summary covering materials, structures and machine-learning algorithms for hand-gesture recognition and possible practical applications through machine-learned wearable electromechanical sensors.

Optimizing the performance of convolutional neural network for enhanced gesture recognition using sEMG

Deep neural networks (DNNs) have demonstrated higher performance results when compared to traditional approaches for implementing robust myoelectric control (MEC) systems. However, the delay induced by optimising a MEC remains a concern for real-time applications. As a result, an optimised DNN architecture based on fine-tuned hyperparameters is required. This study investigates the optimal configuration of convolutional neural network (CNN)-based MEC by proposing an effective data segmentation technique and a generalised set of hyperparameters. Firstly, two segmentation strategies (disjoint and overlap) and various segment and overlap sizes were studied to optimise segmentation parameters. Secondly, to address the challenge of optimising the hyperparameters of a DNN-based MEC system, the problem has been abstracted as an optimisation problem, and Bayesian optimisation has been used to solve it. From 20 healthy people, ten surface electromyography (sEMG) grasping movements abstracted from daily life were chosen as the target gesture set. With an ideal segment size of 200 ms and an overlap size of 80%, the results show that the overlap segmentation technique outperforms the disjoint segmentation technique (p-value < 0.05). In comparison to manual (12.76 ± 4.66), grid (0.10 ± 0.03), and random (0.12 ± 0.05) search hyperparameters optimisation strategies, the proposed optimisation technique resulted in a mean classification error rate (CER) of 0.08 ± 0.03 across all subjects. In addition, a generalised CNN architecture with an optimal set of hyperparameters is proposed. When tested separately on all individuals, the single generalised CNN architecture produced an overall CER of 0.09 ± 0.03. This study's significance lies in its contribution to the field of EMG signal processing by demonstrating the superiority of the overlap segmentation technique, optimizing CNN hyperparameters through Bayesian optimization, and offering practical insights for improving prosthetic control and human-computer interfaces.

Cross-User Electromyography Pattern Recognition Based on a Novel Spatial-Temporal Graph Convolutional Network

With the goal of promoting the development of myoelectric control technology, this paper focuses on exploring graph neural network (GNN) based robust electromyography (EMG) pattern recognition solutions. Given that high-density surface EMG (HD-sEMG) signal contains rich temporal and spatial information, the multi-view spatial-temporal graph convolutional network (MSTGCN)is adopted as the basic classifier, and a feature extraction convolutional neural network (CNN) module is designed and integrated into MSTGCN to generate a new model called CNN-MSTGCN. The EMG pattern recognition experiments are conducted on HD-sEMG data of 17 gestures from 11 subjects. The ablation experiments show that each functional module of the proposed CNN-MSTGCN network has played a more or less positive role in improving the performance of EMG pattern recognition. The user-independent recognition experiments and the transfer learning-based cross-user recognition experiments verify the advantages of the proposed CNN-MSTGCN network in improving recognition rate and reducing user training burden. In the user-independent recognition experiments, CNN-MSTGCN achieves the recognition rate of 68%, which is significantly better than those obtained by residual network-50 (ResNet50, 47.5%, p < 0.001) and long-short-term-memory (LSTM, 57.1%, p=0.045). In the transfer learning-based cross-user recognition experiments, TL-CMSTGCN achieves an impressive recognition rate of 92.3%, which is significantly superior to both TL-ResNet50 (84.6%, p = 0.003) and TL-LSTM (85.3%, p = 0.008). The research results of this paper indicate that GNN has certain advantages in overcoming the impact of individual differences, and can be used to provide possible solutions for achieving robust EMG pattern recognition technology.

LSTM-MSA: A Novel Deep Learning Model With Dual-Stage Attention Mechanisms Forearm EMG-Based Hand Gesture Recognition

This paper introduces the Long Short-Term Memory with Dual-Stage Attention (LSTM-MSA) model, an approach for analyzing electromyography (EMG) signals. EMG signals are crucial in applications like prosthetic control, rehabilitation, and human-computer interaction, but they come with inherent challenges such as non-stationarity and noise. The LSTM-MSA model addresses these challenges by combining LSTM layers with attention mechanisms to effectively capture relevant signal features and accurately predict intended actions. Notable features of this model include dual-stage attention, end-to-end feature extraction and classification integration, and personalized training. Extensive evaluations across diverse datasets consistently demonstrate the LSTM-MSA's superiority in terms of F1 score, accuracy, recall, and precision. This research provides a model for real-world EMG signal applications, offering improved accuracy, robustness, and adaptability.

Integrating intention-based systems in human-robot interaction: a scoping review of sensors, algorithms, and trust

The increasing adoption of robot systems in industrial settings and teaming with humans have led to a growing interest in human-robot interaction (HRI) research. While many robots use sensors to avoid harming humans, they cannot elaborate on human actions or intentions, making them passive reactors rather than interactive collaborators. Intention-based systems can determine human motives and predict future movements, but their closer interaction with humans raises concerns about trust. This scoping review provides an overview of sensors, algorithms, and examines the trust aspect of intention-based systems in HRI scenarios. We searched MEDLINE, Embase, and IEEE Xplore databases to identify studies related to the forementioned topics of intention-based systems in HRI. Results from each study were summarized and categorized according to different intention types, representing various designs. The literature shows a range of sensors and algorithms used to identify intentions, each with their own advantages and disadvantages in different scenarios. However, trust of intention-based systems is not well studied. Although some research in AI and robotics can be applied to intention-based systems, their unique characteristics warrant further study to maximize collaboration performance. This review highlights the need for more research on the trust aspects of intention-based systems to better understand and optimize their role in human-robot interactions, at the same time establishes a foundation for future research in sensor and algorithm designs for intention-based systems.

A Novel Event-Driven Spiking Convolutional Neural Network for Electromyography Pattern Recognition

Electromyography (EMG) pattern recognition is an important technology for prosthesis control and human-computer interaction etc. However, the practical application of EMG pattern recognition is hampered by poor accuracy and robustness due to electrode shift caused by repeated wearing of the signal acquisition device. Moreover, the user's acceptability is low due to the heavy training burden, which is caused by the need for a large amount of training data by traditional methods. In order to explore the advantage of spiking neural network (SNN) in solving the poor robustness and heavy training burden problems in EMG pattern recognition, a spiking convolutional neural network (SCNN) composed of cyclic convolutional neural network (CNN) and fully connected modules is proposed and implemented in this study. High density surface electromyography (HD-sEMG) signals collected from 6 gestures of 10 subjects at 6 electrode positions are taken as the research object. Compared to CNN with the same structure, CNN-Long Short Term Memory (CNN-LSTM), linear kernel linear discriminant analysis classifier (LDA) and spiking multilayer perceptron (Spiking MLP), the accuracy of SCNN is 50.69%, 33.92%, 32.94% and 9.41% higher in the small sample training experiment, 6.50%, 4.23%, 28.73%, and 2.57% higher in the electrode shifts experiment respectively. In addition, the power consumption of SCNN is about 1/93 of CNN. The advantages of the proposed framework in alleviating user training burden, mitigating the adverse effect of electrode shifts and reducing power consumption make it very meaningful for promoting the development of user-friendly real-time myoelectric control system.

Synthetic EMG Based on Adversarial Style Transfer Can Effectively Attack Biometric-Based Personal Identification Models

Biometric-based personal identification models are generally considered to be accurate and secure because biological signals are too complex and person-specific to be fabricated, and EMG signals, in particular, have been used as biological identification tokens due to their high dimension and non-linearity. We investigate the possibility of effectively attacking EMG-based identification models with adversarial biological input via a novel EMG signal individual-style transformer based on a generative adversarial network and tiny leaked data segments. Since two same EMG segments do not exist in nature; the leaked data can't be used to attack the model directly or it will be easily detected. Therefore, it is necessary to extract the style with the leaked personal signals and generate the attack signals with different contents. With our proposed method and tiny leaked personal EMG fragments, numerous EMG signals with different content can be generated in that person's style. EMG hand gesture data from eighteen subjects and three well-recognized deep EMG classifiers were used to demonstrate the effectiveness of the proposed attack methods. The proposed methods achieved an average of 99.41% success rate on confusing identification models and an average of 91.51% success rate on manipulating identification models. These results demonstrate that EMG classifiers based on deep neural networks can be vulnerable to synthetic data attacks. The proof-of-concept results reveal that synthetic EMG biological signals must be considered in biological identification system design across a vast array of relevant biometric systems to ensure personal identification security for individuals and institutions.

Automated diagnosis of cardiovascular diseases from cardiac magnetic resonance imaging using deep learning models: A review

In recent years, cardiovascular diseases (CVDs) have become one of the leading causes of mortality globally. At early stages, CVDs appear with minor symptoms and progressively get worse. The majority of people experience symptoms such as exhaustion, shortness of breath, ankle swelling, fluid retention, and other symptoms when starting CVD. Coronary artery disease (CAD), arrhythmia, cardiomyopathy, congenital heart defect (CHD), mitral regurgitation, and angina are the most common CVDs. Clinical methods such as blood tests, electrocardiography (ECG) signals, and medical imaging are the most effective methods used for the detection of CVDs. Among the diagnostic methods, cardiac magnetic resonance imaging (CMRI) is increasingly used to diagnose, monitor the disease, plan treatment and predict CVDs. Coupled with all the advantages of CMR data, CVDs diagnosis is challenging for physicians as each scan has many slices of data, and the contrast of it might be low. To address these issues, deep learning (DL) techniques have been employed in the diagnosis of CVDs using CMR data, and much research is currently being conducted in this field. This review provides an overview of the studies performed in CVDs detection using CMR images and DL techniques. The introduction section examined CVDs types, diagnostic methods, and the most important medical imaging techniques. The following presents research to detect CVDs using CMR images and the most significant DL methods. Another section discussed the challenges in diagnosing CVDs from CMRI data. Next, the discussion section discusses the results of this review, and future work in CVDs diagnosis from CMR images and DL techniques are outlined. Finally, the most important findings of this study are presented in the conclusion section.

Multi-day dataset of forearm and wrist electromyogram for hand gesture recognition and biometrics

Surface electromyography (sEMG) signals have been used for advanced prosthetics control, hand-gesture recognition (HGR), and more recently as a novel biometric trait. For these sEMG-based applications, the translation from laboratory research setting to real-life scenarios suffers from two major limitations: (1) a small subject pool, and (2) single-session data recordings, both of which prevents acceptable generalization ability. In this longitudinal database, forearm and wrist sEMG data were collected from 43 participants over three different days with long separation (Days 1, 8, and 29) while they performed static hand/wrist gestures. The objective of this dataset is to provide a comprehensive dataset for the development of robust machine learning algorithms of sEMG, for both HGR and biometric applications. We demonstrated the high quality of the current dataset by comparing with the Ninapro dataset. And we presented its usability for both HGR and biometric applications. Among other applications, the dataset can also be used for developing electrode-shift invariant generalized models, which can further bolster the development of wristband and forearm-bracelet sensors.

A deep learning method to predict ankle joint moment during walking at different speeds with ultrasound imaging: A framework for assistive devices control

Robotic assistive or rehabilitative devices are promising aids for people with neurological disorders as they help regain normative functions for both upper and lower limbs. However, it remains challenging to accurately estimate human intent or residual efforts non-invasively when using these robotic devices. In this article, we propose a deep learning approach that uses a brightness mode, that is, B-mode, of ultrasound (US) imaging from skeletal muscles to predict the ankle joint net plantarflexion moment while walking. The designed structure of customized deep convolutional neural networks (CNNs) guarantees the convergence and robustness of the deep learning approach. We investigated the influence of the US imaging's region of interest (ROI) on the net plantarflexion moment prediction performance. We also compared the CNN-based moment prediction performance utilizing B-mode US and sEMG spectrum imaging with the same ROI size. Experimental results from eight young participants walking on a treadmill at multiple speeds verified an improved accuracy by using the proposed US imaging + deep learning approach for net joint moment prediction. With the same CNN structure, compared to the prediction performance by using sEMG spectrum imaging, US imaging significantly reduced the normalized prediction root mean square error by 37.55% ( < .001) and increased the prediction coefficient of determination by 20.13% ( < .001). The findings show that the US imaging + deep learning approach personalizes the assessment of human joint voluntary effort, which can be incorporated with assistive or rehabilitative devices to improve clinical performance based on the assist-as-needed control strategy.

MNet-10: A robust shallow convolutional neural network model performing ablation study on medical images assessing the effectiveness of applying optimal data augmentation technique

Interpretation of medical images with a computer-aided diagnosis (CAD) system is arduous because of the complex structure of cancerous lesions in different imaging modalities, high degree of resemblance between inter-classes, presence of dissimilar characteristics in intra-classes, scarcity of medical data, and presence of artifacts and noises. In this study, these challenges are addressed by developing a shallow convolutional neural network (CNN) model with optimal configuration performing ablation study by altering layer structure and hyper-parameters and utilizing a suitable augmentation technique. Eight medical datasets with different modalities are investigated where the proposed model, named MNet-10, with low computational complexity is able to yield optimal performance across all datasets. The impact of photometric and geometric augmentation techniques on different datasets is also evaluated. We selected the mammogram dataset to proceed with the ablation study for being one of the most challenging imaging modalities. Before generating the model, the dataset is augmented using the two approaches. A base CNN model is constructed first and applied to both the augmented and non-augmented mammogram datasets where the highest accuracy is obtained with the photometric dataset. Therefore, the architecture and hyper-parameters of the model are determined by performing an ablation study on the base model using the mammogram photometric dataset. Afterward, the robustness of the network and the impact of different augmentation techniques are assessed by training the model with the rest of the seven datasets. We obtain a test accuracy of 97.34% on the mammogram, 98.43% on the skin cancer, 99.54% on the brain tumor magnetic resonance imaging (MRI), 97.29% on the COVID chest X-ray, 96.31% on the tympanic membrane, 99.82% on the chest computed tomography (CT) scan, and 98.75% on the breast cancer ultrasound datasets by photometric augmentation and 96.76% on the breast cancer microscopic biopsy dataset by geometric augmentation. Moreover, some elastic deformation augmentation methods are explored with the proposed model using all the datasets to evaluate their effectiveness. Finally, VGG16, InceptionV3, and ResNet50 were trained on the best-performing augmented datasets, and their performance consistency was compared with that of the MNet-10 model. The findings may aid future researchers in medical data analysis involving ablation studies and augmentation techniques.

Energy–Accuracy Aware Finger Gesture Recognition for Wearable IoT Devices

Wearable Internet of Things (IoT) devices can be used efficiently for gesture recognition applications. The nature of these applications requires high recognition accuracy with low energy consumption, which is not easy to solve at the same time. In this paper, we design a finger gesture recognition system using a wearable IoT device. The proposed recognition system uses a light-weight multi-layer perceptron (MLP) classifier which can be implemented even on a low-end micro controller unit (MCU), with a 2-axes flex sensor. To achieve high recognition accuracy with low energy consumption, we first design a framework for the finger gesture recognition system including its components, followed by system-level performance and energy models. Then, we analyze system-level accuracy and energy optimization issues, and explore the numerous design choices to finally achieve energy–accuracy aware finger gesture recognition, targeting four commonly used low-end MCUs. Our extensive simulation and measurements using prototypes demonstrate that the proposed design achieves up to 95.5% recognition accuracy with energy consumption under 2.74 mJ per gesture on a low-end embedded wearable IoT device. We also provide the Pareto-optimal designs among a total of 159 design choices to achieve energy–accuracy aware design points under given energy or accuracy constraints.

Gesture Recognition by Ensemble Extreme Learning Machine Based on Surface Electromyography Signals

In the recent years, gesture recognition based on the surface electromyography (sEMG) signals has been extensively studied. However, the accuracy and stability of gesture recognition through traditional machine learning algorithms are still insufficient to some actual application scenarios. To enhance this situation, this paper proposed a method combining feature selection and ensemble extreme learning machine (EELM) to improve the recognition performance based on sEMG signals. First, the input sEMG signals are preprocessed and 16 features are then extracted from each channel. Next, features that mostly contribute to the gesture recognition are selected from the extracted features using the recursive feature elimination (RFE) algorithm. Then, several independent ELM base classifiers are established using the selected features. Finally, the recognition results are determined by integrating the results obtained by ELM base classifiers using the majority voting method. The Ninapro DB5 dataset containing 52 different hand movements captured from 10 able-bodied subjects was used to evaluate the performance of the proposed method. The results showed that the proposed method could perform the best (overall average accuracy 77.9%) compared with decision tree (DT), ELM, and random forest (RF) methods.

Sign language identification and recognition: A comparative study

Sign Language (SL) is the main language for handicapped and disabled people. Each country has its own SL that is different from other countries. Each sign in a language is represented with variant hand gestures, body movements, and facial expressions. Researchers in this field aim to remove any obstacles that prevent the communication with deaf people by replacing all device-based techniques with vision-based techniques using Artificial Intelligence (AI) and Deep Learning. This article highlights two main SL processing tasks: Sign Language Recognition (SLR) and Sign Language Identification (SLID). The latter task is targeted to identify the signer language, while the former is aimed to translate the signer conversation into tokens (signs). The article addresses the most common datasets used in the literature for the two tasks (static and dynamic datasets that are collected from different corpora) with different contents including numerical, alphabets, words, and sentences from different SLs. It also discusses the devices required to build these datasets, as well as the different preprocessing steps applied before training and testing. The article compares the different approaches and techniques applied on these datasets. It discusses both the vision-based and the data-gloves-based approaches, aiming to analyze and focus on main methods used in vision-based approaches such as hybrid methods and deep learning algorithms. Furthermore, the article presents a graphical depiction and a tabular representation of various SLR approaches.

CSAC-Net: Fast Adaptive sEMG Recognition through Attention Convolution Network and Model-Agnostic Meta-Learning

Gesture recognition through surface electromyography (sEMG) provides a new method for the control algorithm of bionic limbs, which is a promising technology in the field of human–computer interaction. However, subject specificity of sEMG along with the offset of the electrode makes it challenging to develop a model that can quickly adapt to new subjects. In view of this, we introduce a new deep neural network called CSAC-Net. Firstly, we extract the time-frequency feature from the raw signal, which contains rich information. Secondly, we design a convolutional neural network supplemented by an attention mechanism for further feature extraction. Additionally, we propose to utilize model-agnostic meta-learning to adapt to new subjects and this learning strategy achieves better results than the state-of-the-art methods. By the basic experiment on CapgMyo and three ablation studies, we demonstrate the advancement of CSAC-Net.

A Systematic Study on Electromyography-Based Hand Gesture Recognition for Assistive Robots Using Deep Learning and Machine Learning Models

Upper limb amputation severely affects the quality of life and the activities of daily living of a person. In the last decade, many robotic hand prostheses have been developed which are controlled by using various sensing technologies such as artificial vision and tactile and surface electromyography (sEMG). If controlled properly, these prostheses can significantly improve the daily life of hand amputees by providing them with more autonomy in physical activities. However, despite the advancements in sensing technologies, as well as excellent mechanical capabilities of the prosthetic devices, their control is often limited and usually requires a long time for training and adaptation of the users. The myoelectric prostheses use signals from residual stump muscles to restore the function of the lost limbs seamlessly. However, the use of the sEMG signals in robotic as a user control signal is very complicated due to the presence of noise, and the need for heavy computational power. In this article, we developed motion intention classifiers for transradial (TR) amputees based on EMG data by implementing various machine learning and deep learning models. We benchmarked the performance of these classifiers based on overall generalization across various classes and we presented a systematic study on the impact of time domain features and pre-processing parameters on the performance of the classification models. Our results showed that Ensemble learning and deep learning algorithms outperformed other classical machine learning algorithms. Investigating the trend of varying sliding window on feature-based and non-feature-based classification model revealed interesting correlation with the level of amputation. The study also covered the analysis of performance of classifiers on amputation conditions since the history of amputation and conditions are different to each amputee. These results are vital for understanding the development of machine learning-based classifiers for assistive robotic applications.

Feature Fusion-Based Improved Capsule Network for sEMG Signal Recognition

This paper proposes a feature fusion-based improved capsule network (FFiCAPS) to improve the performance of surface electromyogram (sEMG) signal recognition with the purpose of distinguishing hand gestures. Current deep learning models, especially convolution neural networks (CNNs), only take into account the existence of certain features and ignore the correlation among features. To overcome this problem, FFiCAPS adopts the capsule network with a feature fusion method. In order to provide rich information, sEMG signal information and feature data are incorporated together to form new features as input. Improvements made on capsule network are multilayer convolution layer and e-Squash function. The former aggregates feature maps learned by different layers and kernel sizes to extract information in a multiscale and multiangle manner, while the latter grows faster at later stages to strengthen the sensitivity of this model to capsule length changes. Finally, simulation experiments show that the proposed method exceeds other eight methods in overall accuracy under the condition of electrode displacement (86.58%) and among subjects (82.12%), with a notable improvement in recognizing hand open and radial flexion, respectively.

Hand Pose Recognition Using Parallel Multi Stream CNN

Recently, several computer applications provided operating mode through pointing fingers, waving hands, and with body movement instead of a mouse, keyboard, audio, or touch input such as sign language recognition, robot control, games, appliances control, and smart surveillance. With the increase of hand-pose-based applications, new challenges in this domain have also emerged. Support vector machines and neural networks have been extensively used in this domain using conventional RGB data, which are not very effective for adequate performance. Recently, depth data have become popular due to better understating of posture attributes. In this study, a multiple parallel stream 2D CNN (two-dimensional convolution neural network) model is proposed to recognize the hand postures. The proposed model comprises multiple steps and layers to detect hand poses from image maps obtained from depth data. The hyper parameters of the proposed model are tuned through experimental analysis. Three publicly available benchmark datasets: Kaggle, First Person, and Dexter, are used independently to train and test the proposed approach. The accuracy of the proposed method is 99.99%, 99.48%, and 98% using the Kaggle hand posture dataset, First Person hand posture dataset, and Dexter dataset, respectively. Further, the results obtained for F1 and AUC scores are also near-optimal. Comparative analysis with state-of-the-art shows that the proposed model outperforms the previous methods.

Finger Gesture Recognition Using Sensing and Classification of Surface Electromyography Signals With High-Precision Wireless Surface Electromyography Sensors

Finger gesture recognition (FGR) plays a crucial role in achieving, for example, artificial limb control and human-computer interaction. Currently, the most common methods of FGR are visual-based, voice-based, and surface electromyography (EMG)-based ones. Among them, surface EMG-based FGR is very popular and successful because surface EMG is a cumulative bioelectric signal from the surface of the skin that can accurately and intuitively represent the force of the fingers. However, existing surface EMG-based methods still cannot fully satisfy the required recognition accuracy for artificial limb control as the lack of high-precision sensor and high-accurate recognition model. To address this issue, this study proposes a novel FGR model that consists of sensing and classification of surface EMG signals (SC-FGR). In the proposed SC-FGR model, wireless sensors with high-precision surface EMG are first developed for acquiring multichannel surface EMG signals from the forearm. Its resolution is 16 Bits, the sampling rate is 2 kHz, the common-mode rejection ratio (CMRR) is less than 70 dB, and the short-circuit noise (SCN) is less than 1.5 μV. In addition, a convolution neural network (CNN)-based classification algorithm is proposed to achieve FGR based on acquired surface EMG signals. The CNN is trained on a spectrum map transformed from the time-domain surface EMG by continuous wavelet transform (CWT). To evaluate the proposed SC-FGR model, we compared it with seven state-of-the-art models. The experimental results demonstrate that SC-FGR achieves 97.5% recognition accuracy on eight kinds of finger gestures with five subjects, which is much higher than that of comparable models.

Application Research on Optimization Algorithm of sEMG Gesture Recognition Based on Light CNN+LSTM Model

The deep learning gesture recognition based on surface electromyography plays an increasingly important role in human-computer interaction. In order to ensure the high accuracy of deep learning in multistate muscle action recognition and ensure that the training model can be applied in the embedded chip with small storage space, this paper presents a feature model construction and optimization method based on multichannel sEMG amplification unit. The feature model is established by using multidimensional sequential sEMG images by combining convolutional neural network and long-term memory network to solve the problem of multistate sEMG signal recognition. The experimental results show that under the same network structure, the sEMG signal with fast Fourier transform and root mean square as feature data processing has a good recognition rate, and the recognition accuracy of complex gestures is 91.40%, with the size of 1 MB. The model can still control the artificial hand accurately when the model is small and the precision is high.

Action Recognition of Lower Limbs Based on Surface Electromyography Weighted Feature Method

To improve the recognition rate of lower limb actions based on surface electromyography (sEMG), an effective weighted feature method is proposed, and an improved genetic algorithm support vector machine (IGA-SVM) is designed in this paper. First, for the problem of high feature redundancy and low discrimination in the surface electromyography feature extraction process, the weighted feature method is proposed based on the correlation between muscles and actions. Second, to solve the problem of the genetic algorithm selection operator easily falling into a local optimum solution, the improved genetic algorithm-support vector machine is designed by championship with sorting method. Finally, the proposed method is used to recognize six types of lower limb actions designed, and the average recognition rate reaches 94.75%. Experimental results indicate that the proposed method has definite potentiality in lower limb action recognition.

A Hierarchical View Pooling Network for Multichannel Surface Electromyography-Based Gesture Recognition

Hand gesture recognition based on surface electromyography (sEMG) plays an important role in the field of biomedical and rehabilitation engineering. Recently, there is a remarkable progress in gesture recognition using high-density surface electromyography (HD-sEMG) recorded by sensor arrays. On the other hand, robust gesture recognition using multichannel sEMG recorded by sparsely placed sensors remains a major challenge. In the context of multiview deep learning, this paper presents a hierarchical view pooling network (HVPN) framework, which improves multichannel sEMG-based gesture recognition by learning not only view-specific deep features but also view-shared deep features from hierarchically pooled multiview feature spaces. Extensive intrasubject and intersubject evaluations were conducted on the large-scale noninvasive adaptive prosthetics (NinaPro) database to comprehensively evaluate our proposed HVPN framework. Results showed that when using 200 ms sliding windows to segment data, the proposed HVPN framework could achieve the intrasubject gesture recognition accuracy of 88.4%, 85.8%, 68.2%, 72.9%, and 90.3% and the intersubject gesture recognition accuracy of 84.9%, 82.0%, 65.6%, 70.2%, and 88.9% on the first five subdatabases of NinaPro, respectively, which outperformed the state-of-the-art methods.

Two-Step Biometrics Using Electromyogram Signal Based on Convolutional Neural Network-Long Short-Term Memory Networks

Electromyogram (EMG) signals cannot be forged and have the advantage of being able to change the registered data as they are characterized by the waveform, which varies depending on the gesture. In this paper, a two-step biometrics method was proposed using EMG signals based on a convolutional neural network–long short-term memory (CNN-LSTM) network. After preprocessing of the EMG signals, the time domain features and LSTM network were used to examine whether the gesture matched, and single biometrics was performed if the gesture matched. In single biometrics, EMG signals were converted into a two-dimensional spectrogram, and training and classification were performed through the CNN-LSTM network. Data fusion of the gesture recognition and single biometrics was performed in the form of an AND. The experiment used Ninapro EMG signal data as the proposed two-step biometrics method, and the results showed 83.91% gesture recognition performance and 99.17% single biometrics performance. In addition, the false acceptance rate (FAR) was observed to have been reduced by 64.7% through data fusion.

FS-HGR: Few-Shot Learning for Hand Gesture Recognition via Electromyography

This work is motivated by the recent advances in Deep Neural Networks (DNNs) and their widespread applications in human-machine interfaces. DNNs have been recently used for detecting the intended hand gesture through the processing of surface electromyogram (sEMG) signals. Objective: Although DNNs have shown superior accuracy compared to conventional methods when large amounts of data are available for training, their performance substantially decreases when data are limited. Collecting large datasets for training may be feasible in research laboratories, but it is not a practical approach for real-life applications. The main objective of this work is to design a modern DNN-based gesture detection model that relies on minimal training data while providing high accuracy. Methods: We propose the novel Few-Shot learning- Hand Gesture Recognition (FS-HGR) architecture. Few-shot learning is a variant of domain adaptation with the goal of inferring the required output based on just one or a few training observations. The proposed FS-HGR generalizes after seeing very few observations from each class by combining temporal convolutions with attention mechanisms. This allows the meta-learner to aggregate contextual information from experience and to pinpoint specific pieces of information within its available set of inputs. Data Source & Summary of Results: The performance of FS-HGR was tested on the second and fifth Ninapro databases, referred to as the DB2 and DB5, respectively. The DB2 consists of 50 gestures (rest included) from 40 healthy subjects. The Ninapro DB5 contains data from 10 healthy participants performing a total of 53 different gestures (rest included). The proposed approach for the Ninapro DB2 led to 85.94% classification accuracy on new repetitions with few-shot observation (5-way 5-shot), 81.29% accuracy on new subjects with few-shot observation (5-way 5-shot), and 73.36% accuracy on new gestures with few-shot observation (5-way 5-shot). Moreover, the proposed approach for the Ninapro DB5 led to 64.65% classification accuracy on new subjects with few-shot observation (5-way 5-shot).

Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model

Using gestures can help people with certain disabilities in communicating with other people. This paper proposes a lightweight model based on YOLO (You Only Look Once) v3 and DarkNet-53 convolutional neural networks for gesture recognition without additional preprocessing, image filtering, and enhancement of images. The proposed model achieved high accuracy even in a complex environment, and it successfully detected gestures even in low-resolution picture mode. The proposed model was evaluated on a labeled dataset of hand gestures in both Pascal VOC and YOLO format. We achieved better results by extracting features from the hand and recognized hand gestures of our proposed YOLOv3 based model with accuracy, precision, recall, and an F-1 score of 97.68, 94.88, 98.66, and 96.70%, respectively. Further, we compared our model with Single Shot Detector (SSD) and Visual Geometry Group (VGG16), which achieved an accuracy between 82 and 85%. The trained model can be used for real-time detection, both for static hand images and dynamic gestures recorded on a video.

Gesture Recognition Using Surface Electromyography and Deep Learning for Prostheses Hand: State-of-the-Art, Challenges, and Future

Amputation of the upper limb brings heavy burden to amputees, reduces their quality of life, and limits their performance in activities of daily life. The realization of natural control for prosthetic hands is crucial to improving the quality of life of amputees. Surface electromyography (sEMG) signal is one of the most widely used biological signals for the prediction of upper limb motor intention, which is an essential element of the control systems of prosthetic hands. The conversion of sEMG signals into effective control signals often requires a lot of computational power and complex process. Existing commercial prosthetic hands can only provide natural control for very few active degrees of freedom. Deep learning (DL) has performed surprisingly well in the development of intelligent systems in recent years. The significant improvement of hardware equipment and the continuous emergence of large data sets of sEMG have also boosted the DL research in sEMG signal processing. DL can effectively improve the accuracy of sEMG pattern recognition and reduce the influence of interference factors. This paper analyzes the applicability and efficiency of DL in sEMG-based gesture recognition and reviews the key techniques of DL-based sEMG pattern recognition for the prosthetic hand, including signal acquisition, signal preprocessing, feature extraction, classification of patterns, post-processing, and performance evaluation. Finally, the current challenges and future prospects in clinical application of these techniques are outlined and discussed.

Surface EMG-Based Instantaneous Hand Gesture Recognition Using Convolutional Neural Network with the Transfer Learning Method

In recent years, surface electromyography (sEMG)-based human-computer interaction has been developed to improve the quality of life for people. Gesture recognition based on the instantaneous values of sEMG has the advantages of accurate prediction and low latency. However, the low generalization ability of the hand gesture recognition method limits its application to new subjects and new hand gestures, and brings a heavy training burden. For this reason, based on a convolutional neural network, a transfer learning (TL) strategy for instantaneous gesture recognition is proposed to improve the generalization performance of the target network. CapgMyo and NinaPro DB1 are used to evaluate the validity of our proposed strategy. Compared with the non-transfer learning (non-TL) strategy, our proposed strategy improves the average accuracy of new subject and new gesture recognition by 18.7% and 8.74%, respectively, when up to three repeated gestures are employed. The TL strategy reduces the training time by a factor of three. Experiments verify the transferability of spatial features and the validity of the proposed strategy in improving the recognition accuracy of new subjects and new gestures, and reducing the training burden. The proposed TL strategy provides an effective way of improving the generalization ability of the gesture recognition system.

Hand gesture classification using a novel CNN-crow search algorithm

Human–computer interaction (HCI) and related technologies focus on the implementation of interactive computational systems. The studies in HCI emphasize on system use, creation of new techniques that support user activities, access to information, and ensures seamless communication. The use of artificial intelligence and deep learning-based models has been extensive across various domains yielding state-of-the-art results. In the present study, a crow search-based convolution neural networks model has been implemented in gesture recognition pertaining to the HCI domain. The hand gesture dataset used in the study is a publicly available one, downloaded from Kaggle. In this work, a one-hot encoding technique is used to convert the categorical data values to binary form. This is followed by the implementation of a crow search algorithm (CSA) for selecting optimal hyper-parameters for training of dataset using the convolution neural networks. The irrelevant parameters are eliminated from consideration, which contributes towards enhancement of accuracy in classifying the hand gestures. The model generates 100 percent training and testing accuracy that justifies the superiority of the model against traditional state-of-the-art models.

Applying Deep Learning Techniques to Estimate Patterns of Musical Gesture

Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance.

Improving Real-Time Hand Gesture Recognition with Semantic Segmentation

Hand gesture recognition (HGR) takes a central role in human–computer interaction, covering a wide range of applications in the automotive sector, consumer electronics, home automation, and others. In recent years, accurate and efficient deep learning models have been proposed for real-time applications. However, the most accurate approaches tend to employ multiple modalities derived from RGB input frames, such as optical flow. This practice limits real-time performance due to intense extra computational cost. In this paper, we avoid the optical flow computation by proposing a real-time hand gesture recognition method based on RGB frames combined with hand segmentation masks. We employ a light-weight semantic segmentation method (FASSD-Net) to boost the accuracy of two efficient HGR methods: Temporal Segment Networks (TSN) and Temporal Shift Modules (TSM). We demonstrate the efficiency of the proposal on our IPN Hand dataset, which includes thirteen different gestures focused on interaction with touchless screens. The experimental results show that our approach significantly overcomes the accuracy of the original TSN and TSM algorithms by keeping real-time performance.

Gesture Recognition Based on Multiscale Singular Value Entropy and Deep Belief Network

As an important research direction of human–computer interaction technology, gesture recognition is the key to realizing sign language translation. To improve the accuracy of gesture recognition, a new gesture recognition method based on four channel surface electromyography (sEMG) signals is proposed. First, the S-transform is applied to four channel sEMG signals to enhance the time-frequency detail characteristics of the signals. Then, multiscale singular value decomposition is applied to the multiple time-frequency matrix output of S-transform to obtain the time-frequency joint features with better robustness. The corresponding singular value permutation entropy is calculated as the eigenvalue to effectively reduce the dimension of multiple eigenvectors. The gesture features are used as input into the deep belief network for classification, and nine kinds of gestures are recognized with an average accuracy of 93.33%. Experimental results show that the multiscale singular value permutation entropy feature is especially suitable for the pattern classification of the deep belief network.

Advanced Computational Intelligence for Object Detection, Feature Extraction and Recognition in Smart Sensor Environments

The recent years have seen a vast development in various methodologies for object detection and feature extraction and recognition, both in theory and in practice [...].

Analysis of Influence of Segmentation, Features, and Classification in sEMG Processing: A Case Study of Recognition of Brazilian Sign Language Alphabet

Sign Language recognition systems aid communication among deaf people, hearing impaired people, and speakers. One of the types of signals that has seen increased studies and that can be used as input for these systems is surface electromyography (sEMG). This work presents the recognition of a set of alphabet gestures from Brazilian Sign Language (Libras) using sEMG acquired from an armband. Only sEMG signals were used as input. Signals from 12 subjects were acquired using a MyoTM armband for the 26 signs of the Libras alphabet. Additionally, as the sEMG has several signal processing parameters, the influence of segmentation, feature extraction, and classification was considered at each step of the pattern recognition. In segmentation, window length and the presence of four levels of overlap rates were analyzed, as well as the contribution of each feature, the literature feature sets, and new feature sets proposed for different classifiers. We found that the overlap rate had a high influence on this task. Accuracies in the order of 99% were achieved for the following factors: segments of 1.75 s with a 12.5% overlap rate; the proposed set of four features; and random forest (RF) classifiers.

Performance Evaluation of Convolutional Neural Network for Hand Gesture Recognition Using EMG

Electromyography (EMG) is a measure of electrical activity generated by the contraction of muscles. Non-invasive surface EMG (sEMG)-based pattern recognition methods have shown the potential for upper limb prosthesis control. However, it is still insufficient for natural control. Recent advancements in deep learning have shown tremendous progress in biosignal processing. Multiple architectures have been proposed yielding high accuracies (>95%) for offline analysis, yet the delay caused due to optimization of the system remains a challenge for its real-time application. From this arises a need for optimized deep learning architecture based on fine-tuned hyper-parameters. Although the chance of achieving convergence is random, however, it is important to observe that the performance gain made is significant enough to justify extra computation. In this study, the convolutional neural network (CNN) was implemented to decode hand gestures from the sEMG data recorded from 18 subjects to investigate the effect of hyper-parameters on each hand gesture. Results showed that the learning rate set to either 0.0001 or 0.001 with 80-100 epochs significantly outperformed (p < 0.05) other considerations. In addition, it was observed that regardless of network configuration some motions (close hand, flex hand, extend the hand and fine grip) performed better (83.7% ± 13.5%, 71.2% ± 20.2%, 82.6% ± 13.9% and 74.6% ± 15%, respectively) throughout the course of study. So, a robust and stable myoelectric control can be designed on the basis of the best performing hand motions. With improved recognition and uniform gain in performance, the deep learning-based approach has the potential to be a more robust alternative to traditional machine learning algorithms.