Multiuser gesture recognition using sEMG signals via canonical correlation analysis and optimal transport

A Novel Instruction Gesture Set Determination Scheme for Robust Myoelectric Control Applications

OBJECTIVE

Myoelectric control technology has important application value in rehabilitation medicine, prosthesis control, human-computer interaction (HCI) and other fields. However, the user dependence of electromyography (EMG) pattern recognition is one of the key problems hindering the implementation of robust myoelectric control applications. Aimed at solving the user dependence problem, this paper proposed a novel instruction gesture set determination scheme for EMG pattern recognition in user-independent mode.

METHODS

The scheme uses T-distributed stochastic neighbor embedding (T-SNE) dimensionality reduction to analyze high-dimensional surface EMG data from multiple users and gestures. This process can identify gesture combinations with minimal individual differences and high separability.

RESULTS

The proposed scheme was validated using two large-scale EMG gesture databases with different acquisition devices, subjects, and gestures. Optimal and inferior gesture sets of varying sizes were identified. In recognition experiments conducted in both user-independent and electrode-offset modes, the optimal gesture sets demonstrated significantly higher recognition accuracies compared to the inferior sets, with improvements ranging from 12.57% to 36.92%.

CONCLUSION

The results demonstrated that the separability of the obtained optimal gesture sets was significantly superior to that of the inferior sets, confirming the effectiveness of the proposed scheme in reducing user dependence in EMG pattern recognition.

SIGNIFICANCE

The study has certain application value to promote the development of myoelectric control technology. Specifically, the scheme proposed can be used to determine instruction gesture sets with low user dependence and high separability for myoelectric control applications.

A Novel Active Learning Framework for Cross-Subject Human Activity Recognition from Surface Electromyography

Wearable sensor-based human activity recognition (HAR) methods hold considerable promise for upper-level control in exoskeleton systems. However, such methods tend to overlook the critical role of data quality and still encounter challenges in cross-subject adaptation. To address this, we propose an active learning framework that integrates the relation network architecture with data sampling techniques. Initially, target data are used to fine tune two auxiliary classifiers of the pre-trained model, thereby establishing subject-specific classification boundaries. Subsequently, we assess the significance of the target data based on classifier discrepancy and partition the data into sample and template sets. Finally, the sampled data and a category clustering algorithm are employed to tune model parameters and optimize template data distribution, respectively. This approach facilitates the adaptation of the model to the target subject, enhancing both accuracy and generalizability. To evaluate the effectiveness of the proposed adaptation framework, we conducted evaluation experiments on a public dataset and a self-constructed electromyography (EMG) dataset. Experimental results demonstrate that our method outperforms the compared methods across all three statistical metrics. Furthermore, ablation experiments highlight the necessity of data screening. Our work underscores the practical feasibility of implementing user-independent HAR methods in exoskeleton control systems.

Across Sessions and Subjects Domain Adaptation for Building Robust Myoelectric Interface

Gesture interaction via surface electromyography (sEMG) signal is a promising approach for advanced human-computer interaction systems. However, improving the performance of the myoelectric interface is challenging due to the domain shift caused by the signal's inherent variability. To enhance the interface's robustness, we propose a novel adaptive information fusion neural network (AIFNN) framework, which could effectively reduce the effects of multiple scenarios. Specifically, domain adversarial training is established to inhibit the shared network's weights from exploiting domain-specific representation, thus allowing for the extraction of domain-invariant features. Effectively, classification loss, domain diversence loss and domain discrimination loss are employed, which improve classification performance while reduce distribution mismatches between the two domains. To simulate the application of myoelectric interface, experiments were carried out involving three scenarios (intra-session, inter-session and inter-subject scenarios). Ten non-disabled subjects were recruited to perform sixteen gestures for ten consecutive days. The experimental results indicated that the performance of AIFNN was better than two other state-of-the-art transfer learning approaches, namely fine-tuning (FT) and domain adversarial network (DANN). This study demonstrates the capability of AIFNN to maintain robustness over time and generalize across users in practical myoelectric interface implementations. These findings could serve as a foundation for future deployments.

Identifying frequency-dependent imaging genetic associations via hypergraph-structured multi-task sparse canonical correlation analysis

Identifying complex associations between genetic variations and imaging phenotypes is a challenging task in the research of brain imaging genetics. The previous study has proved that neuronal oscillations within distinct frequency bands are derived from frequency-dependent genetic modulation. Thus it is meaningful to explore frequency-dependent imaging genetic associations, which may give important insights into the pathogenesis of brain disorders. In this work, the hypergraph-structured multi-task sparse canonical correlation analysis (HS-MTSCCA) was developed to explore the associations between multi-frequency imaging phenotypes and single-nucleotide polymorphisms (SNPs). Specifically, we first created a hypergraph for the imaging phenotypes of each frequency and the SNPs, respectively. Then, a new hypergraph-structured constraint was proposed to learn high-order relationships among features in each hypergraph, which can introduce biologically meaningful information into the model. The frequency-shared and frequency-specific imaging phenotypes and SNPs could be identified using the multi-task learning framework. We also proposed a useful strategy to tackle this algorithm and then demonstrated its convergence. The proposed method was evaluated on four simulation datasets and a real schizophrenia dataset. The experimental results on synthetic data showed that HS-MTSCCA outperforms the other competing methods according to canonical correlation coefficients, canonical weights, and cosine similarity. And the results on real data showed that HS-MTSCCA could obtain superior canonical coefficients and canonical weights. Furthermore, the identified frequency-shared and frequency-specific biomarkers could provide more interesting and meaningful information, demonstrating that HS-MTSCCA is a powerful method for brain imaging genetics.

Robust myoelectric pattern recognition methods for reducing users' calibration burden: challenges and future

Myoelectric pattern recognition (MPR) has evolved into a sophisticated technology widely employed in controlling myoelectric interface (MI) devices like prosthetic and orthotic robots. Current MIs not only enable multi-degree-of-freedom control of prosthetic limbs but also demonstrate substantial potential in consumer electronics. However, the non-stationary random characteristics of myoelectric signals poses challenges, leading to performance degradation in practical scenarios such as electrode shifting and switching new users. Conventional MIs often necessitate meticulous calibration, imposing a significant burden on users. To address user frustration during the calibration process, researchers have focused on identifying MPR methods that alleviate this burden. This article categorizes common scenarios that incur calibration burdens as based on data distribution shift and based on dynamic data categories. Then further investigated and summarized the popular robust MPR algorithms used to reduce the user's calibration burden. We categorize these algorithms as based on data manipulate, feature manipulation and, model structure. And describes the scenarios to which each method is applicable and the conditions required for calibration. Finally, this review is concluded with the advantages of robust MPR and the remaining challenges and future opportunities.

One-shot random forest model calibration for hand gesture decoding

Objective.Most existing machine learning models for myoelectric control require a large amount of data to learn user-specific characteristics of the electromyographic (EMG) signals, which is burdensome. Our objective is to develop an approach to enable the calibration of a pre-trained model with minimal data from a new myoelectric user.Approach.We trained a random forest (RF) model with EMG data from 20 people collected during the performance of multiple hand grips. To adapt the decision rules for a new user, first, the branches of the pre-trained decision trees were pruned using the validation data from the new user. Then new decision trees trained merely with data from the new user were appended to the pruned pre-trained model.Results.Real-time myoelectric experiments with 18 participants over two days demonstrated the improved accuracy of the proposed approach when compared to benchmark user-specific RF and the linear discriminant analysis models. Furthermore, the RF model that was calibrated on day one for a new participant yielded significantly higher accuracy on day two, when compared to the benchmark approaches, which reflects the robustness of the proposed approach.Significance.The proposed model calibration procedure is completely source-free, that is, once the base model is pre-trained, no access to the source data from the original 20 people is required. Our work promotes the use of efficient, explainable, and simple models for myoelectric control.

A novel unsupervised dynamic feature domain adaptation strategy for cross-individual myoelectric gesture recognition

Objective.Surface electromyography pattern recognition (sEMG-PR) is considered as a promising control method for human-machine interaction systems. However, the performance of a trained classifier would greatly degrade for novel users since sEMG signals are user-dependent and largely affected by a number of individual factors such as the quantity of subcutaneous fat and the skin impedance.Approach.To solve this issue, we proposed a novel unsupervised cross-individual motion recognition method that aligned sEMG features from different individuals by self-adaptive dimensional dynamic distribution adaptation (SD-DDA) in this study. In the method, both the distances of marginal and conditional distributions between source and target features were minimized through automatically selecting the optimal feature domain dimension by using a small amount of unlabeled target data.Main results.The effectiveness of the proposed method was tested on four different feature sets, and results showed that the average classification accuracy was improved by above 10% on our collected dataset with the best accuracy reached 90.4%. Compared to six kinds of classic transfer learning methods, the proposed method showed an outstanding performance with improvements of 3.2%-13.8%. Additionally, the proposed method achieved an approximate 9% improvement on a publicly available dataset.Significance.These results suggested that the proposed SD-DDA method is feasible for cross-individual motion intention recognition, which would provide help for the application of sEMG-PR based system.

Sign Language Recognition Using the Electromyographic Signal: A Systematic Literature Review

The analysis and recognition of sign languages are currently active fields of research focused on sign recognition. Various approaches differ in terms of analysis methods and the devices used for sign acquisition. Traditional methods rely on video analysis or spatial positioning data calculated using motion capture tools. In contrast to these conventional recognition and classification approaches, electromyogram (EMG) signals, which measure muscle electrical activity, offer potential technology for detecting gestures. These EMG-based approaches have recently gained attention due to their advantages. This prompted us to conduct a comprehensive study on the methods, approaches, and projects utilizing EMG sensors for sign language handshape recognition. In this paper, we provided an overview of the sign language recognition field through a literature review, with the objective of offering an in-depth review of the most significant techniques. These techniques were categorized in this article based on their respective methodologies. The survey discussed the progress and challenges in sign language recognition systems based on surface electromyography (sEMG) signals. These systems have shown promise but face issues like sEMG data variability and sensor placement. Multiple sensors enhance reliability and accuracy. Machine learning, including deep learning, is used to address these challenges. Common classifiers in sEMG-based sign language recognition include SVM, ANN, CNN, KNN, HMM, and LSTM. While SVM and ANN are widely used, random forest and KNN have shown better performance in some cases. A multilayer perceptron neural network achieved perfect accuracy in one study. CNN, often paired with LSTM, ranks as the third most popular classifier and can achieve exceptional accuracy, reaching up to 99.6% when utilizing both EMG and IMU data. LSTM is highly regarded for handling sequential dependencies in EMG signals, making it a critical component of sign language recognition systems. In summary, the survey highlights the prevalence of SVM and ANN classifiers but also suggests the effectiveness of alternative classifiers like random forests and KNNs. LSTM emerges as the most suitable algorithm for capturing sequential dependencies and improving gesture recognition in EMG-based sign language recognition systems.

Differentiation of Bolus Texture During Deglutition via High-Density Surface Electromyography: A Pilot Study

OBJECTIVE

Swallowing is a complex neuromuscular task. There is limited spatiotemporal data on normative surface electromyographic signal during swallow, particularly across standard textures. We hypothesize the pattern of electromyographic signal of the anterior neck varies cranio-caudally, that laterality can be evaluated, and categorization of bolus texture can be differentiated by high-density surface electromyography (HDsEMG) through signal analysis.

METHODS

An HDsEMG grid of 20 electrodes captured electromyographic activity in eight healthy adult subjects across 240 total swallows. Participants swallowed five standard textures: saliva, thin liquid, puree, mixed consistency, and dry solid. Data were bandpass filtered, underwent functional alignment of signal, and then placed into binary classifier receiver operating characteristic (ROC) curves. Muscular activity was visualized by creating two-dimensional EMG heat maps.

RESULTS

Signal analysis results demonstrated a positive correlation between signal amplitude and bolus texture. Greater differences of amplitude in the cranial most region of the array when compared to the caudal most region were noted in all subjects. Lateral comparison of the array revealed symmetric power levels across all subjects and textures. ROC curves demonstrated the ability to correctly classify textures within subjects in 6 of 10 texture comparisons.

CONCLUSION

This pilot study suggests that utilizing HDsEMG during deglutition can noninvasively differentiate swallows of varying texture noninvasively. This may prove useful in future diagnostic and behavioral swallow applications.

LEVEL OF EVIDENCE

4 Laryngoscope, 133:2695-2703, 2023.

Cross-user gesture recognition from sEMG signals using an optimal transport assisted student-teacher framework

The cross-user gesture recognition is a puzzle in the myoelectric control system, owing to great variability in muscle activities across different users. To address this problem, a novel optimal transport (OT) assisted student-teacher (ST) framework (termed OT-ST) was proposed in this paper to facilitate transfer across user domains in an unsupervised domain adaptation (UDA) manner. In this framework, the initial parameters of the ST models were trained with the labeled data from users in the source domain. In the model transfer stage for a new user in the target domain, the teacher model was utilized to generate pseudo labels for unlabeled testing samples, providing guidance to the adaptation of the student model. The OT algorithm was employed to optimize the pseudo labels generated from the teacher model, avoiding the model bias and further improving the effect of domain adaptation. The performance of the proposed OT-ST framework was evaluated via experiments of classifying seven hand gestures using high-density surface electromyogram (HD-sEMG) recordings from extensor digitorum muscles of eight intact-limbed subjects. The OT-ST framework yielded a high accuracy of 96.50 ± 2.88% for new users, and outperformed other common machine learning and UDA methods significantly (p < 0.01), demonstrating its effectiveness. The OT-ST framework does not require special repetitive training or any labeled data for calibration. In addition, it can incrementally learn from new testing samples and improve the recognition ability. This study provides a promising method for developing user-generic myoelectric pattern recognition, with wide applications in human-computer interaction, consumer electronics and prosthesis control.

Feasibility study on the application of a spiking neural network in myoelectric control systems

In recent years, the effectiveness of a spiking neural network (SNN) for Electromyography (EMG) pattern recognition has been validated, but there is a lack of comprehensive consideration of the problems of heavy training burden, poor robustness, and high energy consumption in the application of actual myoelectric control systems. In order to explore the feasibility of the application of SNN in actual myoelectric control systems, this paper investigated an EMG pattern recognition scheme based on SNN. To alleviate the differences in EMG distribution caused by electrode shifts and individual differences, the adaptive threshold encoding was applied to gesture sample encoding. To improve the feature extraction ability of SNN, the leaky-integrate-and-fire (LIF) neuron that combines voltage-current effect was adopted as a spike neuron model. To balance recognition accuracy and power consumption, experiments were designed to determine encoding parameter and LIF neuron release threshold. By conducting the gesture recognition experiments considering different training test ratios, electrode shifts, and user independences on the nine-gesture high-density and low-density EMG datasets respectively, the advantages of the proposed SNN-based scheme have been verified. Compared with a Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM) and Linear Discriminant Analysis (LDA), SNN can effectively reduce the number of repetitions in the training set, and its power consumption was reduced by 1-2 orders of magnitude. For the high-density and low-density EMG datasets, SNN improved the overall average accuracies by about (0.99 ~ 14.91%) under different training test ratios. For the high-density EMG dataset, the accuracy of SNN was improved by (0.94 ~ 13.76%) under electrode-shift condition and (3.81 ~ 18.95%) in user-independent case. The advantages of SNN in alleviating the user training burden, reducing power consumption, and improving robustness are of great significance for the implementation of user-friendly low-power myoelectric control systems.

A Transfer Learning Based Cross-Subject Generic Model for Continuous Estimation of Finger Joint Angles From a New User

Continuous estimation of finger joints based on surface electromyography (sEMG) has attracted much attention in the field of human-machine interface (HMI). A couple of deep learning models were proposed to estimate the finger joint angles for specific subject. When applied onto a new subject, however, the performance of the subject-specific model would degrade significantly due to the inter-subject differences. Therefore, a novel cross-subject generic (CSG) model was proposed in this study to estimate continuous kinematics of finger joints for new users. Firstly, a multi-subject model based on the LSTA-Conv network was built by using sEMG and finger joint angles data from multiple subjects. Then, the subjects adversarial knowledge (SAK) transfer learning strategy was adopted to calibrate the multi-subject model with the training data from a new user. With the updated model parameters and the testing data from the new user, multiple finger joint angles could be estimated afterwards. The overall performance of the CSG model for new users was validated on three public datasets from Ninapro. The results showed that the newly proposed CSG model significantly outperformed five subject-specific models and two transfer learning models in terms of Pearson correlation coefficient, root mean square error, and coefficient of determination. Comparison analysis showed that both the long short-term feature aggregation (LSTA) module and the SAK transfer learning strategy contributed to the CSG model. Moreover, increasing number of subjects in training set improved the generalization capability of the CSG model. The novel CSG model would facilitate the application of robotic hand control and other HMI settings.

Multi-source domain generalization and adaptation toward cross-subject myoelectric pattern recognition

Objective.Myoelectric pattern recognition (MPR) has shown satisfactory performance under ideal laboratory conditions. Nevertheless, the individual variances lead to dramatic performance degradation in cross-user MPR applications. It is crucial to enable the myoelectric interface to adapt to multiple users' surface electromyography (sEMG) distributions in practical.Approach.Domain adaptation (DA) is a promising approach to tackle cross-user challenges due to its ability to diminish the divergence between individual users' EMG distributions and escalate model generalization performance. However, existing DA methods in sEMG control are based on single-source domain adaptation (SDA). SDA solely mixes multiple training users' data as a combined source domain and attempts to align with a novel user. This simple data mixing manner ignores the sEMG distribution variations between disparate training users, leading to an insufficient variance elimination and lower performance. To this end, this paper proposes a multi-source synchronize domain adaptation framework with both DA and domain generalization (DG) capability. This multi-source framework aligns each source user and the new user in individual feature spaces, which better transfers the knowledge of existing users to the new user. Moreover, we retain the source-combined data to preserve the effectiveness of SDA. The property was further confirmed by evaluating the performance of the proposed method on data from nine subjects performing six tasks.Main results.Experiment results prove that the proposed multi-source framework achieved both positive DG and DA performance in a cross-user classification manner.Significance.This work demonstrates the usability and feasibility of the proposed multi-source framework in cross-user myoelectric control.

Reduce the User Burden of Multiuser Myoelectric Interface via Few-Shot Domain Adaptation

Due to physiological and anatomical variations across users, myoelectric interfaces trained by multiple users cannot be adapted to the unique hand movement patterns of the new user. Most current work requires the new user to provide one or more trials per gesture (dozens to hundreds of samples), applying domain adaptation methods to calibrate the model and achieve promising movement recognition performance. However, the user burden associated with time-consuming electromyography signal acquisition and annotation is a key factor hindering the practical application of myoelectric control. As shown in this work, once the number of calibration samples is reduced, the performance of previous cross-user myoelectric interfaces will degrade due to the lack of enough statistics to characterize the distributions. In this paper, a few-shot supervised domain adaptation (FSSDA) framework is proposed to address this issue. It aligns the distributions of different domains by calculating the distribution distances of point-wise surrogates. Specifically, we introduce a positive-negative pair distance loss to find a shared embedding subspace where each scarce sample from the new user will be closer to the positive samples and away from the negative samples of multiple users. Thus, FSSDA allows every target domain sample to be paired with all source domain samples and optimizes the feature distance between each target domain sample and the source domain samples within the same batch, instead of direct estimation of the data distribution of the target domain. The proposed method is validated on two high-density EMG datasets, which achieves the averaged recognition accuracies of 97.59% and 82.78% with only 5 samples per gesture. In addition, FSSDA is also effective even when only one sample per gesture is provided. The experimental results show that FSSDA greatly reduces the user burden and further facilitates the development of myoelectric pattern recognition techniques.

Time–frequency feature transform suite for deep learning-based gesture recognition using sEMG signals

Recently, deep learning methods have achieved considerable performance in gesture recognition using surface electromyography signals. However, improving the recognition accuracy in multi-subject gesture recognition remains a challenging problem. In this study, we aimed to improve recognition performance by adding subject-specific prior knowledge to provide guidance for multi-subject gesture recognition. We proposed a time–frequency feature transform suite (TFFT) that takes the maps generated by continuous wavelet transform (CWT) as input. The TFFT can be connected to a neural network to obtain an end-to-end architecture. Thus, we integrated the suite into traditional neural networks, such as convolutional neural networks and long short-term memory, to adjust the intermediate features. The results of comparative experiments showed that the deep learning models with the TFFT suite based on CWT improved the recognition performance of the original architectures without the TFFT suite in gesture recognition tasks. Our proposed TFFT suite has promising applications in multi-subject gesture recognition and prosthetic control.

Multi-user motion recognition using sEMG via discriminative canonical correlation analysis and adaptive dimensionality reduction

The inability of new users to adapt quickly to the surface electromyography (sEMG) interface has greatly hindered the development of sEMG in the field of rehabilitation. This is due mainly to the large differences in sEMG signals produced by muscles when different people perform the same motion. To address this issue, a multi-user sEMG framework is proposed, using discriminative canonical correlation analysis and adaptive dimensionality reduction (ADR). The interface projects the feature sets for training users and new users into a low-dimensional uniform style space, overcoming the problem of individual differences in sEMG. The ADR method removes the redundant information in sEMG features and improves the accuracy of system motion recognition. The presented framework was validated on eight subjects with intact limbs, with an average recognition accuracy of 92.23% in 12 categories of upper-limb movements. In rehabilitation laboratory experiments, the average recognition rate reached 90.52%. The experimental results suggest that the framework offers a good solution to enable new rehabilitation users to adapt quickly to the sEMG interface.

Detecting the universal adversarial perturbations on high-density sEMG signals

Myoelectric pattern recognition is a promising approach for upper limb neuroprosthetic control. Convolutional neural networks (CNN) are increasingly used in dealing with the electromyography (EMG) signal collected by high-density electrodes due to its capacity to take full advantage of spatial information about muscle activity. However, it has been found that CNN models are very vulnerable to well-designed and tiny perturbations, such like universal adversarial perturbation (UAP). As shown in this work, the CNN-based myoelectric pattern recognition method can achieve a classification accuracy of more than 90%, but can only achieve a classification accuracy of less than 20% after the attack. This type of attack poses a big security concern to prosthetic control. To the best of our knowledge, there is no study on the detection of adversarial attacks to the myoelectric control system. In this paper, a correlation feature based on Chebyshev distance between the adjacent channels is proposed to detect the attack for EMG signals, which will serve as early warning and defense against the adversarial attacks. The performance of the detection framework is assessed with two high-density EMG datasets. The results show that our method has a detection rate of 91.39% and 93.87% for the attacks on both datasets with a latency of no more than 2 ms, which will facilitate the security of muscle-computer interfaces.

Domain Adaptation with Self-Guided Adaptive Sampling Strategy: Feature Alignment for Cross-User Myoelectric Pattern Recognition

Gestural interfaces based on surface electromyo-graphic (sEMG) signal have been widely explored. Nevertheless, due to the individual differences in the sEMG signals, it is very challenging for a myoelectric pattern recognition control system to adapt cross-user variability. Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness, and it is a promising approach to solve the cross-user challenge. Existing UDA methods largely ignore the instantaneous data distribution during model updating, thus deteriorating the feature representation given a large domain shift. To address this issue, a novel method is proposed based on a UDA model incorporated with a self-guided adaptive sampling (SGAS) strategy. This strategy is designed to utilize the domain distance in a kernel space as an indicator to screen out reliable instantaneous samples for updating the classifier. Thus, it enables improved alignment of feature representations of myoelectric patterns across users. To evaluate the performance of the proposed method, sEMG data were recorded from forearm muscles of nine subjects performing six finger and wrist gestures. Experiment results show that the UDA method with the SGAS strategy achieved a mean accuracy of 90.41% ± 14.44% in a cross-user classification manner, outperformed the state-of-the-art methods with statistical significance. This study demonstrates the effectiveness of the proposed UDA framework and offers a novel tool for implementing cross-user myoelectric pattern recognition towards a multi-user and user-independent control.

User-Independent EMG Gesture Recognition Method Based on Adaptive Learning

In a gesture recognition system based on surface electromyogram (sEMG) signals, the recognition model established by existing users cannot directly generalize to the across-user scenarios due to the individual variability of sEMG signals. In this article, we propose an adaptive learning method to handle the problem. The muscle synergy is chosen as the feature vector because it can well-characterize the neural origin of movement. The initial train set is composed of representative samples extracted from the synergy matrix of the existing user. When the new users use the system, the label is obtained by the adaptive K nearest neighbor algorithm (KNN). The recognition process does not require the pre-experiment for new users due to the introduction of adaptive learning strategy, namely, the qualified data and the label of new user data evaluated by a risk evaluator are used to update the train set and KNN weights, so as to adapt to the new users. We have tested the algorithm in DB1 and DB5 of Ninapro databases. The average recognition accuracy is 68.04, 73.35, and 83.05% for different types of gestures, respectively, achieving the effects of the user-dependent method. Our study can avoid the re-training steps and the recognition performance will improve with the increased frequency of uses, which will further facilitate the widespread implementation of sEMG control systems using pattern recognition techniques.

Towards Robust, Adaptive and Reliable Upper-limb Motion Estimation Using Machine Learning and Deep Learning--A Survey in Myoelectric Control

To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. \textcolor{red}{Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.

Estimation of the Continuous Pronation–Supination Movement by Using Multichannel EMG Signal Features and Kalman Filter: Application to Control an Exoskeleton

The Hill muscle model can be used to estimate the human joint angles during continuous movement. However, adopting this model requires the knowledge of many parameters, such as the length and speed of contraction of muscle fibers, which are liable to change with different individuals, leading to errors in estimation. This study established the backpropagation neural network model based on surface electromyography (sEMG) features and human movement angle. First, the function of muscles in joint rotation is defined, and then, sensors are placed on muscle tissues to gain sEMG, and then, a relation model between the surface sEMG features and the joint angle is constructed. As integrated electromyography information cannot be well reflected through a single electromyography feature, a feature extraction method combining the time domain, frequency domain, and time–frequency domain was proposed. As the degree of freedom (DOF) of the pronation–supination movement was controlled by several muscles, it was difficult to make an angle prediction. A method of correcting the estimation error based on the Kalman filter was raised to cope with this problem. An exoskeleton robot with one DOF was designed and put into the tracking experiment. The results show that the proposed model was able to enhance the estimation of the joint angle during continuous pronation–supination movements.

Inter-Subject Domain Adaptation for CNN-Based Wrist Kinematics Estimation Using sEMG

Recently, convolutional neural network (CNN) has been widely investigated to decode human intentions using surface Electromyography (sEMG) signals. However, a pre-trained CNN model usually suffers from severe degradation when testing on a new individual, and this is mainly due to domain shift where characteristics of training and testing sEMG data differ substantially. To enhance inter-subject performances of CNN in the wrist kinematics estimation, we propose a novel regression scheme for supervised domain adaptation (SDA), based on which domain shift effects can be effectively reduced. Specifically, a two-stream CNN with shared weights is established to exploit source and target sEMG data simultaneously, such that domain-invariant features can be extracted. To tune CNN weights, both regression losses and a domain discrepancy loss are employed, where the former enable supervised learning and the latter minimizes distribution divergences between two domains. In this study, eight healthy subjects were recruited to perform wrist flexion-extension movements. Experiment results illustrated that the proposed regression SDA outperformed fine-tuning, a state-of-the-art transfer learning method, in both single-single and multiple-single scenarios of kinematics estimation. Unlike fine-tuning which suffers from catastrophic forgetting, regression SDA can maintain much better performances in original domains, which boosts the model reusability among multiple subjects.

Deep Cross-User Models Reduce the Training Burden in Myoelectric Control

The effort, focus, and time to collect data and train EMG pattern recognition systems is one of the largest barriers to their widespread adoption in commercial applications. In addition to multiple repetitions of motions, including exemplars of confounding factors during the training protocol has been shown to be critical for robust machine learning models. This added training burden is prohibitive for most regular use cases, so cross-user models have been proposed that could leverage inter-repetition variability supplied by other users. Existing cross-user models have not yet achieved performance levels sufficient for commercialization and require users to closely adhere to a training protocol that is impractical without expert guidance. In this work, we extend a previously reported adaptive domain adversarial neural network (ADANN) to a cross-subject framework that requires very little training data from the end-user. We compare its performance to single-repetition within-user training and the previous state-of-the-art cross-subject technique, canonical correlation analysis (CCA). ADANN significantly outperformed CCA for both intact-limb (86.8–96.2%) and amputee (64.1–84.2%) populations. Moreover, the ADANN adaptation computation time was substantially lower than the time otherwise devoted to conducting a full within-subject training protocol. This study shows that cross-user models, enabled by deep-learned adaptations, may be a viable option for improved generalized pattern recognition-based myoelectric control.