1
|
Raghu STP, MacIsaac DT, Scheme EJ. Self-supervised learning via VICReg enables training of EMG pattern recognition using continuous data with unclear labels. Comput Biol Med 2025; 185:109479. [PMID: 39637459 DOI: 10.1016/j.compbiomed.2024.109479] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2024] [Revised: 10/25/2024] [Accepted: 11/25/2024] [Indexed: 12/07/2024]
Abstract
In this study, we investigate the application of self-supervised learning via pre-trained Long Short-Term Memory (LSTM) networks for training surface electromyography pattern recognition models (sEMG-PR) using dynamic data with transitions. While labeling such data poses challenges due to the absence of ground-truth labels during transitions between classes, self-supervised pre-training offers a way to circumvent this issue. We compare the performance of LSTMs trained with either fully-supervised or self-supervised loss to a conventional non-temporal model (LDA) on two data types: segmented ramp data (lacking transition information) and continuous dynamic data inclusive of class transitions. Statistical analysis reveals that the temporal models outperform non-temporal models when trained with continuous dynamic data. Additionally, the proposed VICReg pre-trained temporal model with continuous dynamic data significantly outperformed all other models. Interestingly, when using only ramp data, the LSTM performed worse than the LDA, suggesting potential overfitting due to the absence of sufficient dynamics. This highlights the interplay between data type and model choice. Overall, this work highlights the importance of representative dynamics in training data and the potential for leveraging self-supervised approaches to enhance sEMG-PR models.
Collapse
Affiliation(s)
- Shriram Tallam Puranam Raghu
- Department of Electrical and Computer Engineering and the Institute of Biomedical Engineering, University of New Brunswick, Fredericton, E3B 5A3, NB, Canada.
| | - Dawn T MacIsaac
- Department of Electrical and Computer Engineering and the Institute of Biomedical Engineering, University of New Brunswick, Fredericton, E3B 5A3, NB, Canada
| | - Erik J Scheme
- Department of Electrical and Computer Engineering and the Institute of Biomedical Engineering, University of New Brunswick, Fredericton, E3B 5A3, NB, Canada
| |
Collapse
|
2
|
Eddy E, Campbell E, Bateman S, Scheme E. Big data in myoelectric control: large multi-user models enable robust zero-shot EMG-based discrete gesture recognition. Front Bioeng Biotechnol 2024; 12:1463377. [PMID: 39380895 PMCID: PMC11459555 DOI: 10.3389/fbioe.2024.1463377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2024] [Accepted: 08/28/2024] [Indexed: 10/10/2024] Open
Abstract
Myoelectric control, the use of electromyogram (EMG) signals generated during muscle contractions to control a system or device, is a promising input, enabling always-available control for emerging ubiquitous computing applications. However, its widespread use has historically been limited by the need for user-specific machine learning models because of behavioural and physiological differences between users. Leveraging the publicly available 612-user EMG-EPN612 dataset, this work dispels this notion, showing that true zero-shot cross-user myoelectric control is achievable without user-specific training. By taking a discrete approach to classification (i.e., recognizing the entire dynamic gesture as a single event), a classification accuracy of 93.0% for six gestures was achieved on a set of 306 unseen users, showing that big data approaches can enable robust cross-user myoelectric control. By organizing the results into a series of mini-studies, this work provides an in-depth analysis of discrete cross-user models to answer unknown questions and uncover new research directions. In particular, this work explores the number of participants required to build cross-user models, the impact of transfer learning for fine-tuning these models, and the effects of under-represented end-user demographics in the training data, among other issues. Additionally, in order to further evaluate the performance of the developed cross-user models, a completely new dataset was created (using the same recording device) that includes known covariate factors such as cross-day use and limb-position variability. The results show that the large data models can effectively generalize to new datasets and mitigate the impact of common confounding factors that have historically limited the adoption of EMG-based inputs.
Collapse
Affiliation(s)
- Ethan Eddy
- Institute of Biomedical Engineering, University of New Brunswick, Fredericton, NB, Canada
| | - Evan Campbell
- Institute of Biomedical Engineering, University of New Brunswick, Fredericton, NB, Canada
| | - Scott Bateman
- Faculty of Computer Science, University of New Brunswick, Fredericton, NB, Canada
| | - Erik Scheme
- Institute of Biomedical Engineering, University of New Brunswick, Fredericton, NB, Canada
| |
Collapse
|
3
|
Dai Q, Wong Y, Kankanhalli M, Li X, Geng W. Rejecting Unknown Gestures Based on Surface-Electromyography Using Variational Autoencoder. IEEE Trans Neural Syst Rehabil Eng 2024; 32:750-758. [PMID: 38289843 DOI: 10.1109/tnsre.2024.3360035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2024]
Abstract
The conventional surface electromyography (sEMG)-based gesture recognition systems exhibit impressive performance in controlled laboratory settings. As most systems are trained in a closed-set setting, the systems's performance may see significant deterioration when novel gestures are presented as imposter. In addition, the state-of-the-art generative and discriminative methods have achieved considerable performance on high-density sEMG signals. This can be seen as an unrealistic setting as the real-world muscle computer interface are mainly comprised of sparse multichannel sEMG signals. In this work, we propose a novel variational autoencoder based approach for open-set gesture recognition based on sparse multichannel sEMG signals. Using the predefined corresponding latent conditional distribution of known gestures, the conditional Gaussian distribution of each known gesture is learned. Those samples with low probability density are identified as unknown gestures. The sEMG signals of known gestures are classified using the Kullback-Leibler divergences between the predefined prior distributions and input samples. The proposed approach is evaluated using three benchmark sparse multichannel sEMG databases. The experimental results demonstrate that our approach outperforms the existing open-set sEMG-based gesture recognition approaches and achieves a better trade-off between classifying known gestures and rejecting unknown gestures.
Collapse
|
4
|
Yi C, Wei B, Zhu J, Chen C, Wang Y, Chen Z, Huang Y, Jiang F. Mordo2: A Personalization Framework for Silent Command Recognition. IEEE Trans Neural Syst Rehabil Eng 2024; 32:133-143. [PMID: 38090845 DOI: 10.1109/tnsre.2023.3342068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/19/2023]
Abstract
Wearable human-computer interactions in daily life are increasingly encouraged by the prevalence of intelligent wearables. It poses a demanding requirement of micro-interaction and minimizing social awkwardness. Our previous work demonstrated the feasibility of recognizing silent commands through around-ear biosensors with the limitation of user adaptation. In this work, we ease the limitation by a personalization framework that integrates spectral factorization of signals, temporal confidence rejection and commonly used transfer learning algorithms. Specifically, we first empirically formulate the user adaptation issue by presenting the accuracies of applying transfer learning algorithms to our previous method. Second, we improve the signal-to-noise ratio by proposing the supervised spectral factorization method that learns the amplitude and phase mappings between around-ear signals and the signals of articulated facial muscles. Third, we leverage the time continuity of commands and introduce the time decay into confidence rejection. Finally, extensive experiments have been conducted to evaluate the feasibility and improvements. The results indicate an average accuracy of 92.38% which is significantly larger than solely using transfer learning algorithms. And a comparable accuracy can be achieved with significantly reduced data of new users. The overall performance shows the framework can significantly improve the accuracy of user adaptations. The work would aid a further step toward commercial products for silent command recognition and inspire the solution to the user adaptation challenge of wearable human-computer interactions.
Collapse
|
5
|
Raghu STP, MacIsaac D, Scheme E. Decision-Change Informed Rejection Improves Robustness in Pattern Recognition-Based Myoelectric Control. IEEE J Biomed Health Inform 2023; 27:6051-6061. [PMID: 37721893 DOI: 10.1109/jbhi.2023.3316599] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/20/2023]
Abstract
Post-processing techniques have been shown to improve the quality of the decision stream generated by classifiers used in pattern-recognition-based myoelectric control. However, these techniques have largely been tested individually and on well-behaved, stationary data, failing to fully evaluate their trade-offs between smoothing and latency during dynamic use. Correspondingly, in this work, we survey and compare 8 different post-processing and decision stream improvement schemes in the context of continuous and dynamic class transitions: majority vote, Bayesian fusion, onset locking, outlier detection, confidence-based rejection, confidence scaling, prior adjustment, and adaptive windowing. We then propose two new temporally aware post-processing schemes that use changes in the decision and confidence streams to better reject uncertain decisions. Our decision-change informed rejection (DCIR) approach outperforms existing schemes during both steady-state and transitions based on error rates and decision stream volatility whether using conventional or deep classifiers. These results suggest that added robustness can be gained by appropriately leveraging temporal context in myoelectric control.
Collapse
|
6
|
Ren H, Liu T, Wang J. Design and Analysis of an Upper Limb Rehabilitation Robot Based on Multimodal Control. SENSORS (BASEL, SWITZERLAND) 2023; 23:8801. [PMID: 37960505 PMCID: PMC10647264 DOI: 10.3390/s23218801] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Revised: 10/24/2023] [Accepted: 10/27/2023] [Indexed: 11/15/2023]
Abstract
To address the rehabilitation needs of upper limb hemiplegic patients in various stages of recovery, streamline the workload of rehabilitation professionals, and provide data visualization, our research team designed a six-degree-of-freedom upper limb exoskeleton rehabilitation robot inspired by the human upper limb's structure. We also developed an eight-channel synchronized signal acquisition system for capturing surface electromyography (sEMG) signals and elbow joint angle data. Utilizing Solidworks, we modeled the robot with a focus on modularity, and conducted structural and kinematic analyses. To predict the elbow joint angles, we employed a back propagation neural network (BPNN). We introduced three training modes: a PID control, bilateral control, and active control, each tailored to different phases of the rehabilitation process. Our experimental results demonstrated a strong linear regression relationship between the predicted reference values and the actual elbow joint angles, with an R-squared value of 94.41% and an average error of four degrees. Furthermore, these results validated the increased stability of our model and addressed issues related to the size and single-mode limitations of upper limb rehabilitation robots. This work lays the theoretical foundation for future model enhancements and further research in the field of rehabilitation.
Collapse
Affiliation(s)
- Hang Ren
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai 200000, China;
| | - Tongyou Liu
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 201100, China;
| | - Jinwu Wang
- School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai 200000, China;
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 201100, China;
| |
Collapse
|
7
|
Zhang W, Wang Y, Zhang J, Pang G. EMG-FRNet: A feature reconstruction network for EMG irrelevant gesture recognition. Biosci Trends 2023:2023.01116. [PMID: 37394614 DOI: 10.5582/bst.2023.01116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/04/2023]
Abstract
With the development of deep learning technology, gesture recognition based on surface electromyography (EMG) signals has shown broad application prospects in various human-computer interaction fields. Most current gesture recognition technologies can achieve high recognition accuracy on a wide range of gesture actions. However, in practical applications, gesture recognition based on surface EMG signals is susceptible to interference from irrelevant gesture movements, which affects the accuracy and security of the system. Therefore, it is crucial to design an irrelevant gesture recognition method. This paper introduces the GANomaly network from the field of image anomaly detection into surface EMG-based irrelevant gesture recognition. The network has a small feature reconstruction error for target samples and a large feature reconstruction error for irrelevant samples. By comparing the relationship between the feature reconstruction error and the predefined threshold, we can determine whether the input samples are from the target category or the irrelevant category. In order to improve the performance of EMG irrelevant gesture recognition, this paper proposes a feature reconstruction network named EMG-FRNet for EMG irrelevant gesture recognition. This network is based on GANomaly and incorporates structures such as channel cropping (CC), cross-layer encoding-decoding feature fusion (CLEDFF), and SE channel attention (SE). In this paper, Ninapro DB1, Ninapro DB5 and self-collected datasets were used to verify the performance of the proposed model. The Area Under the receiver operating characteristic Curve (AUC) values of EMG-FRNet on the above three datasets were 0.940, 0.926 and 0.962, respectively. Experimental results demonstrate that the proposed model achieves the highest accuracy among related research.
Collapse
Affiliation(s)
- Wenli Zhang
- Faculty of Imformation Technology, Beijing University of Technology, Beijing, China
| | - Yufei Wang
- Faculty of Imformation Technology, Beijing University of Technology, Beijing, China
| | - Jianyi Zhang
- College of Art and Design, Beijing University of Technology, Beijing, China
| | - Gongpeng Pang
- Faculty of Imformation Technology, Beijing University of Technology, Beijing, China
| |
Collapse
|
8
|
Vu PP, Vaskov AK, Lee C, Jillala RR, Wallace DM, Davis AJ, Kung TA, Kemp SWP, Gates DH, Chestek CA, Cederna PS. Long-term upper-extremity prosthetic control using regenerative peripheral nerve interfaces and implanted EMG electrodes. J Neural Eng 2023; 20:026039. [PMID: 37023743 PMCID: PMC10126717 DOI: 10.1088/1741-2552/accb0c] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 03/23/2023] [Accepted: 04/06/2023] [Indexed: 04/08/2023]
Abstract
Objective.Extracting signals directly from the motor system poses challenges in obtaining both high amplitude and sustainable signals for upper-limb neuroprosthetic control. To translate neural interfaces into the clinical space, these interfaces must provide consistent signals and prosthetic performance.Approach.Previously, we have demonstrated that the Regenerative Peripheral Nerve Interface (RPNI) is a biologically stable, bioamplifier of efferent motor action potentials. Here, we assessed the signal reliability from electrodes surgically implanted in RPNIs and residual innervated muscles in humans for long-term prosthetic control.Main results.RPNI signal quality, measured as signal-to-noise ratio, remained greater than 15 for up to 276 and 1054 d in participant 1 (P1), and participant 2 (P2), respectively. Electromyography from both RPNIs and residual muscles was used to decode finger and grasp movements. Though signal amplitude varied between sessions, P2 maintained real-time prosthetic performance above 94% accuracy for 604 d without recalibration. Additionally, P2 completed a real-world multi-sequence coffee task with 99% accuracy for 611 d without recalibration.Significance.This study demonstrates the potential of RPNIs and implanted EMG electrodes as a long-term interface for enhanced prosthetic control.
Collapse
Affiliation(s)
- Philip P Vu
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
- Section of Plastic Surgery, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Alex K Vaskov
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
- Section of Plastic Surgery, University of Michigan, Ann Arbor, MI 48109, United States of America
- Robotics Institute, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Christina Lee
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Ritvik R Jillala
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Dylan M Wallace
- Robotics Institute, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Alicia J Davis
- University of Michigan Hospital Orthotics & Prosthetics Center Ann Arbor, Ann Arbor, MI 48109, United States of America
| | - Theodore A Kung
- Section of Plastic Surgery, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Stephen W P Kemp
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
- Section of Plastic Surgery, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Deanna H Gates
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
- Robotics Institute, University of Michigan, Ann Arbor, MI 48109, United States of America
- School of Kinesiology, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Cynthia A Chestek
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
- Robotics Institute, University of Michigan, Ann Arbor, MI 48109, United States of America
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109, United States of America
- Neuroscience Graduate Program, University of Michigan, Ann Arbor, MI 48109, United States of America
| | - Paul S Cederna
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109, United States of America
- Section of Plastic Surgery, University of Michigan, Ann Arbor, MI 48109, United States of America
| |
Collapse
|
9
|
Bu D, Guo S, Guo J, Li H, Wang H. Low-Density sEMG-Based Pattern Recognition of Unrelated Movements Rejection for Wrist Joint Rehabilitation. MICROMACHINES 2023; 14:555. [PMID: 36984962 PMCID: PMC10056026 DOI: 10.3390/mi14030555] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 02/16/2023] [Accepted: 02/25/2023] [Indexed: 06/18/2023]
Abstract
sEMG-based pattern recognition commonly assumes a limited number of target categories, and the classifiers often predict each target category depending on probability. In wrist rehabilitation training, the patients may make movements that do not belong to the target category unconsciously. However, most pattern recognition methods can only identify limited patterns and are prone to be disturbed by abnormal movement, especially for wrist joint movements. To address the above the problem, a sEMG-based rejection method for unrelated movements is proposed to identify wrist joint unrelated movements using center loss. In this paper, the sEMG signal collected by the Myo armband is used as the input of the sEMG control method. First, the sEMG signal is processed by sliding signal window and image coding. Then, the CNN with center loss and softmax loss is used to describe the spatial information from the sEMG image to extract discriminative features and target movement recognition. Finally, the deep spatial information is used to train the AE to reject unrelated movements based on the reconstruction loss. The results show that the proposed method can realize the target movements recognition and reject unrelated movements with an F-score of 93.4% and a rejection accuracy of 95% when the recall is 0.9, which reveals the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Dongdong Bu
- School of Life Science, Beijing Institute of Technology, Beijing 100081, China
| | - Shuxiang Guo
- School of Life Science, Beijing Institute of Technology, Beijing 100081, China
- Key Laboratory of Convergence Medical Engineering System and Healthcare Technology, Ministry of Industry and Information Technology, School of Life Science, Beijing Institute of Technology, Beijing 100081, China
| | - Jin Guo
- School of Life Science, Beijing Institute of Technology, Beijing 100081, China
- Key Laboratory of Convergence Medical Engineering System and Healthcare Technology, Ministry of Industry and Information Technology, School of Life Science, Beijing Institute of Technology, Beijing 100081, China
| | - He Li
- School of Life Science, Beijing Institute of Technology, Beijing 100081, China
| | - Hanze Wang
- School of Life Science, Beijing Institute of Technology, Beijing 100081, China
| |
Collapse
|
10
|
Lin Y, Palaniappan R, De Wilde P, Li L. Robust Long-Term Hand Grasp Recognition With Raw Electromyographic Signals Using Multidimensional Uncertainty-Aware Models. IEEE Trans Neural Syst Rehabil Eng 2023; 31:962-971. [PMID: 37021902 DOI: 10.1109/tnsre.2023.3236982] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
Hand grasp recognition with surface electromyography (sEMG) has been used as a possible natural strategy to control hand prosthetics. However, effectively performing activities of daily living for users relies significantly on the long-term robustness of such recognition, which is still a challenging task due to confused classes and several other variabilities. We hypothesise that this challenge can be addressed by introducing uncertainty-aware models because the rejection of uncertain movements has previously been demonstrated to improve the reliability of sEMG-based hand gesture recognition. With a particular focus on a very challenging benchmark dataset (NinaPro Database 6), we propose a novel end-to-end uncertainty-aware model, an evidential convolutional neural network (ECNN), which can generate multidimensional uncertainties, including vacuity and dissonance, for robust long-term hand grasp recognition. To avoid heuristically determining the optimal rejection threshold, we examine the performance of misclassification detection in the validation set. Extensive comparisons of accuracy under the non-rejection and rejection scheme are conducted when classifying 8 hand grasps (including rest) over 8 subjects across proposed models. The proposed ECNN is shown to improve recognition performance, achieving an accuracy of 51.44% without the rejection option and 83.51% under the rejection scheme with multidimensional uncertainties, significantly improving the current state-of-the-art (SoA) by 3.71% and 13.88%, respectively. Furthermore, its overall rejection-capable recognition accuracy remains stable with only a small accuracy degradation after the last data acquisition over 3 days. These results show the potential design of a reliable classifier that yields accurate and robust recognition performance.
Collapse
|
11
|
Dankovich LJ, Vaughn-Cooke M, Bergbreiter S. Can You Do That Again? Time Series Consolidation as a Robust Method of Tailoring Gesture Recognition to Individual Users. SENSORS (BASEL, SWITZERLAND) 2022; 22:7512. [PMID: 36236611 PMCID: PMC9572399 DOI: 10.3390/s22197512] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/03/2022] [Revised: 09/26/2022] [Accepted: 09/27/2022] [Indexed: 06/16/2023]
Abstract
Robust inter-session modeling of gestures is still an open learning challenge. A sleeve equipped with capacitive strap sensors was used to capture two gesture data sets from a convenience sample of eight subjects. Two pipelines were explored. In FILT a novel two-stage algorithm was introduced which uses an unsupervised learning algorithm to find samples representing gesture transitions and discards them prior to training and validating conventional models. In TSC a confusion matrix was used to automatically consolidate commonly confused class labels, resulting in a set of gestures tailored to an individual subject's abilities. The inter-session testing accuracy using the Time Series Consolidation (TSC) method increased from a baseline inter-session average of 42.47 ± 3.83% to 93.02% ± 4.97% while retaining an average of 5.29 ± 0.46 out of the 11 possible gesture categories. These pipelines used classic machine learning algorithms which require relatively small amounts of data and computational power compared to deep learning solutions. These methods may also offer more flexibility in interface design for users suffering from handicaps limiting their manual dexterity or ability to reliably make gestures, and be possible to implement on edge devices with low computational power.
Collapse
Affiliation(s)
- Louis J. Dankovich
- A. James Clark School of Engineering, University of Maryland, College Park, MD 20742, USA
| | - Monifa Vaughn-Cooke
- A. James Clark School of Engineering, University of Maryland, College Park, MD 20742, USA
| | - Sarah Bergbreiter
- Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA
| |
Collapse
|
12
|
Wu L, Chen X, Chen X, Zhang X. Rejecting Novel Motions in High-Density Myoelectric Pattern Recognition Using Hybrid Neural Networks. Front Neurorobot 2022; 16:862193. [PMID: 35418847 PMCID: PMC8996371 DOI: 10.3389/fnbot.2022.862193] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2022] [Accepted: 02/22/2022] [Indexed: 11/23/2022] Open
Abstract
The objective of this study is to develop a method for alleviating a novel pattern interference toward achieving a robust myoelectric pattern-recognition control system. To this end, a framework was presented for surface electromyogram (sEMG) pattern classification and novelty detection using hybrid neural networks, i.e., a convolutional neural network (CNN) and autoencoder networks. In the framework, the CNN was first used to extract spatio-temporal information conveyed in the sEMG data recorded via high-density (HD) 2-dimensional electrode arrays. Given the target motion patterns well-characterized by the CNN, autoencoder networks were applied to learn variable correlation in the spatio-temporal information, where samples from any novel pattern appeared to be significantly different from those from target patterns. Therefore, it was straightforward to discriminate and then reject the novel motion interferences identified as untargeted and unlearned patterns. The performance of the proposed method was evaluated with HD-sEMG data recorded by two 8 × 6 electrode arrays placed over the forearm extensors and flexors of 9 subjects performing seven target motion tasks and six novel motion tasks. The proposed method achieved high accuracies over 95% for identifying and rejecting novel motion tasks, and it outperformed conventional methods with statistical significance (p < 0.05). The proposed method is demonstrated to be a promising solution for rejecting novel motion interferences, which are ubiquitous in myoelectric control. This study will enhance the robustness of the myoelectric control system against novelty interference.
Collapse
|
13
|
Bao T, Xie SQ, Yang P, Zhou P, Zhang ZQ. Towards Robust, Adaptive and Reliable Upper-limb Motion Estimation Using Machine Learning and Deep Learning--A Survey in Myoelectric Control. IEEE J Biomed Health Inform 2022; 26:3822-3835. [PMID: 35294368 DOI: 10.1109/jbhi.2022.3159792] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. \textcolor{red}{Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.
Collapse
|
14
|
Lin Y, Palaniappan R, De Wilde P, Li L. Reliability Analysis For Finger Movement Recognition With Raw Electromyographic Signal by Evidential Convolutional Networks. IEEE Trans Neural Syst Rehabil Eng 2022; 30:96-107. [PMID: 34995190 DOI: 10.1109/tnsre.2022.3141593] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Hand gesture recognition with surface electromyography (sEMG) is indispensable for Muscle-Gesture-Computer Interface. The usual focus of it is upon performance evaluation involving the accuracy and robustness of hand gesture recognition. However, addressing the reliability of such classifiers has been absent, to our best knowledge. This may be due to the lack of consensus on the definition of model reliability in this field. An uncertainty-aware model has the potential to self-evaluate the quality of its inference, thereby making it more reliable. Moreover, uncertainty-based rejection has been shown to improve the performance of sEMG-based hand gesture recognition. Therefore, we first define model reliability here as the quality of its uncertainty estimation and propose an offline framework to quantify it. To promote reliability analysis, we propose a novel end-to-end uncertainty-aware finger movement classifier, i.e., evidential convolutional neural network (ECNN), and illustrate the advantages of its multidimensional uncertainties such as vacuity and dissonance. Extensive comparisons of accuracy and reliability are conducted on NinaPro Database 5, exercise A, across CNN and three variants of ECNN based on different training strategies. The results of classifying 12 finger movements over 10 subjects show that the best mean accuracy achieved by ECNN is 76.34%, which is slightly higher than the state-of-the-art performance. Furthermore, ECNN variants are more reliable than CNN in general, where the highest improvement of reliability of 19.33% is observed. This work demonstrates the potential of ECNN and recommends using the proposed reliability analysis as a supplementary measure for studying sEMG-based hand gesture recognition.
Collapse
|
15
|
Analyzing the impact of class transitions on the design of pattern recognition-based myoelectric control schemes. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103134] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
16
|
Yang Z, Guo S, Hirata H, Kawanishi M. A Mirror Bilateral Neuro-Rehabilitation Robot System with the sEMG-Based Real-Time Patient Active Participant Assessment. Life (Basel) 2021; 11:life11121290. [PMID: 34947820 PMCID: PMC8707631 DOI: 10.3390/life11121290] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2021] [Revised: 11/16/2021] [Accepted: 11/18/2021] [Indexed: 11/16/2022] Open
Abstract
In this paper, a novel mirror visual feedback-based (MVF) bilateral neurorehabilitation system with surface electromyography (sEMG)-based patient active force assessment was proposed for upper limb motor recovery and improvement of limb inter-coordination. A mirror visual feedback-based human–robot interface was designed to facilitate the bilateral isometric force output training task. To achieve patient active participant assessment, an sEMG signals-based elbow joint isometric force estimation method was implemented into the proposed system for real-time affected side force assessment and participation evaluation. To assist the affected side limb efficiently and precisely, a mirror bilateral control framework was presented for bilateral limb coordination. Preliminary experiments were conducted to evaluate the estimation accuracy of force estimation method and force tracking accuracy of system performance. The experimental results show the proposed force estimation method can efficiently calculate the elbow joint force in real-time, and the affected side limb of patients can be assisted to track output force of the non-paretic side limb for better limb coordination by the proposed bilateral rehabilitation system.
Collapse
Affiliation(s)
- Ziyi Yang
- Graduate School of Engineering, Kagawa University, Takamatsu 761-0396, Japan;
| | - Shuxiang Guo
- Key Laboratory of Convergence Medical Engineering System and Healthcare Technology, The Ministry of Industry and Information Technology, Beijing Institute of Technology, Beijing 100081, China
- Department of Intelligent Mechanical Systems Engineering, Kagawa University, Takamatsu 761-0396, Japan;
- Correspondence: ; Tel.: +81-087-864-2333
| | - Hideyuki Hirata
- Department of Intelligent Mechanical Systems Engineering, Kagawa University, Takamatsu 761-0396, Japan;
| | - Masahiko Kawanishi
- Department of Neurological Surgery, Faculty of Medicine, Kagawa University, Takamatsu 761-0793, Japan;
| |
Collapse
|
17
|
Kumar P, Phinyomark A, Scheme E. Verification-Based Design of a Robust EMG Wake Word. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:638-642. [PMID: 34891374 DOI: 10.1109/embc46164.2021.9630922] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Surface electromyography (sEMG) signals are now commonly used in continuous myoelectric control of prostheses. More recently, researchers have considered EMG-based gesture recognition systems for human computer interaction research. These systems instead focus on recognizing discrete gestures (like a finger snap). The majority of works, however, have focused on improving multi-class performance, with little consideration for false activations from "other" classes. Consequently, they lack the robustness needed for real-world applications which generally require a single motion class such as a mouse click or a wake word. Furthermore, many works have borrowed the windowed classification schemes from continuous control, and thus fail to leverage the temporal structure of the gesture. In this paper, we propose a verification-based approach to creating a robust EMG wake word using one-class classifiers (Support Vector Data Description, One Class-Support Vector Machine, Dynamic Time Warping (DTW) & Hidden Markov Models). The area under the ROC curve (AUC) is used as a feature optimization objective as it provides a better representation of the verification performance. Equal error rate (EER) and AUC are then used as evaluation metrics. The results are computed using both window-based and temporal classifiers on a dataset consisting of five different gestures, with a best EER of 0.04 and AUC of 0.98, recorded using a DTW scheme. These results demonstrate a design framework that may benefit the development of more robust solutions for EMG-based wake words or input commands for a variety of interactive applications.
Collapse
|
18
|
Wu L, Zhang X, Zhang X, Chen X, Chen X. Metric learning for novel motion rejection in high-density myoelectric pattern recognition. Knowl Based Syst 2021. [DOI: 10.1016/j.knosys.2021.107165] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
19
|
Pan L, Huang H(H. A robust model-based neural-machine interface across different loading weights applied at distal forearm. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102509] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
20
|
Recurrent Neural Network for electromyographic gesture recognition in transhumeral amputees. Appl Soft Comput 2020. [DOI: 10.1016/j.asoc.2020.106616] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
21
|
Chang J, Phinyomark A, Bateman S, Scheme E. Wearable EMG-Based Gesture Recognition Systems During Activities of Daily Living: An Exploratory Study. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:3448-3451. [PMID: 33018745 DOI: 10.1109/embc44109.2020.9176615] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Recent advancements in wearable technologies have increased the potential for practical gesture recognition systems using electromyogram (EMG) signals. However, despite the high classification accuracies reported in many studies (> 90%), there is a gap between academic results and industrial success. This is in part because state-of-the-art EMG-based gesture recognition systems are commonly evaluated in highly-controlled laboratory environments, where users are assumed to be resting and performing one of a closed set of target gestures. In real world conditions, however, a variety of non-target gestures are performed during activities of daily living (ADLs), resulting in many false positive activations. In this study, the effect of ADLs on the performance of EMG-based gesture recognition using a wearable EMG device was investigated. EMG data for 14 hand and finger gestures, as well as continuous activity during uncontrolled ADLs (>10 hours in total) were collected and analyzed. Results showed that (1) the cluster separability of 14 different gestures during ADLs was 171 times worse than during rest; (2) the probability distributions of EMG features extracted from different ADLs were significantly different (p <; 0.05). (3) of the 14 target gestures, a right angle gesture (extension of the thumb and index finger) was least often inadvertently activated during ADLs. These results suggest that ADLs and other non-trained gestures must be taken into consideration when designing EMG-based gesture recognition systems.
Collapse
|
22
|
Cene VH, Balbinot A. Resilient EMG Classification to Enable Reliable Upper-Limb Movement Intent Detection. IEEE Trans Neural Syst Rehabil Eng 2020; 28:2507-2514. [PMID: 32956063 DOI: 10.1109/tnsre.2020.3024947] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Reliable control of assistive devices using surface electromyography (sEMG) remains an unsolved task due to the signal's stochastic behavior that prevents robust pattern recognition for real-time control. Non-representative samples lead to inherent class overlaps that generate classification ripples for which the most common alternatives rely on post-processing and sample discard methods that insert additional delays and often do not offer substantial improvements. In this paper, a resilient classification pipeline based on Extreme Learning Machines (ELM) was used to classify 17 different upper-limb movements through sEMG signals from a total of 99 trials derived from three different databases. The method was compared to a baseline ELM and a sample discarding (DISC) method and proved to generate more stable and consistent classifications. The average accuracy boost of ≈ 10% in all databases lead to average weighted accuracy rates higher as 53,4% for amputees and 89,0% for non-amputee volunteers. The results match or outperform related works even without sample discards.
Collapse
|
23
|
Betthauser JL, Krall JT, Bannowsky SG, Levay G, Kaliki RR, Fifer MS, Thakor NV. Stable Responsive EMG Sequence Prediction and Adaptive Reinforcement With Temporal Convolutional Networks. IEEE Trans Biomed Eng 2020; 67:1707-1717. [PMID: 31545709 PMCID: PMC10497232 DOI: 10.1109/tbme.2019.2943309] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/22/2023]
Abstract
Prediction of movement intentions from electromyographic (EMG) signals is typically performed with a pattern recognition approach, wherein a short dataframe of raw EMG is compressed into an instantaneous feature-encoding that is meaningful for classification. However, EMG signals are time-varying, implying that a frame-wise approach may not sufficiently incorporate temporal context into predictions, leading to erratic and unstable prediction behavior. OBJECTIVE We demonstrate that sequential prediction models and, specifically, temporal convolutional networks are able to leverage useful temporal information from EMG to achieve superior predictive performance. METHODS We compare this approach to other sequential and frame-wise models predicting 3 simultaneous hand and wrist degrees-of-freedom from 2 amputee and 13 non-amputee human subjects in a minimally constrained experiment. We also compare these models on the publicly available Ninapro and CapgMyo amputee and non-amputee datasets. RESULTS Temporal convolutional networks yield predictions that are more accurate and stable than frame-wise models, especially during inter-class transitions, with an average response delay of 4.6 ms and simpler feature-encoding. Their performance can be further improved with adaptive reinforcement training. SIGNIFICANCE Sequential models that incorporate temporal information from EMG achieve superior movement prediction performance and these models allow for novel types of interactive training. CONCLUSIONS Addressing EMG decoding as a sequential modeling problem will lead to enhancements in the reliability, responsiveness, and movement complexity available from prosthesis control systems.
Collapse
|
24
|
Campbell E, Phinyomark A, Scheme E. Current Trends and Confounding Factors in Myoelectric Control: Limb Position and Contraction Intensity. SENSORS (BASEL, SWITZERLAND) 2020; 20:E1613. [PMID: 32183215 PMCID: PMC7146367 DOI: 10.3390/s20061613] [Citation(s) in RCA: 43] [Impact Index Per Article: 8.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/05/2020] [Revised: 03/08/2020] [Accepted: 03/09/2020] [Indexed: 11/17/2022]
Abstract
This manuscript presents a hybrid study of a comprehensive review and a systematic(research) analysis. Myoelectric control is the cornerstone ofmany assistive technologies used in clinicalpractice, such as prosthetics and orthoses, and human-computer interaction, such as virtual reality control.Although the classification accuracy of such devices exceeds 90% in a controlled laboratory setting,myoelectric devices still face challenges in robustness to variability of daily living conditions.The intrinsic physiological mechanisms limiting practical implementations of myoelectric deviceswere explored: the limb position effect and the contraction intensity effect. The degradationof electromyography (EMG) pattern recognition in the presence of these factors was demonstratedon six datasets, where classification performance was 13% and 20% lower than the controlledsetting for the limb position and contraction intensity effect, respectively. The experimental designsof limb position and contraction intensity literature were surveyed. Current state-of-the-art trainingstrategies and robust algorithms for both effects were compiled and presented. Recommendationsfor future limb position effect studies include: the collection protocol providing exemplars of at least 6positions (four limb positions and three forearm orientations), three-dimensional space experimentaldesigns, transfer learning approaches, and multi-modal sensor configurations. Recommendationsfor future contraction intensity effect studies include: the collection of dynamic contractions, nonlinearcomplexity features, and proportional control.
Collapse
Affiliation(s)
- Evan Campbell
- Department of Electrical and Computer Engineering, University of New Brunswick, Canada
- Institute of Biomedical Engineering, University of New Brunswick, Canada
| | - Angkoon Phinyomark
- Institute of Biomedical Engineering, University of New Brunswick, Canada
| | - Erik Scheme
- Department of Electrical and Computer Engineering, University of New Brunswick, Canada
- Institute of Biomedical Engineering, University of New Brunswick, Canada
| |
Collapse
|
25
|
Krasoulis A, Vijayakumar S, Nazarpour K. Multi-Grip Classification-Based Prosthesis Control With Two EMG-IMU Sensors. IEEE Trans Neural Syst Rehabil Eng 2020; 28:508-518. [DOI: 10.1109/tnsre.2019.2959243] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
26
|
Pan L, Crouch DL, Huang H. Comparing EMG-Based Human-Machine Interfaces for Estimating Continuous, Coordinated Movements. IEEE Trans Neural Syst Rehabil Eng 2019; 27:2145-2154. [PMID: 31478862 DOI: 10.1109/tnsre.2019.2937929] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Electromyography (EMG)-based interfaces are trending toward continuous, simultaneous control with multiple degrees of freedom. Emerging methods range from data-driven approaches to biomechanical model-based methods. However, there has been no direct comparison between these two types of continuous EMG-based interfaces. The aim of this study was to compare a musculoskeletal model (MM) with two data-driven approaches, linear regression (LR) and artificial neural network (ANN), for predicting continuous wrist and hand motions for EMG-based interfaces. Six able-bodied subjects and one transradial amputee subject performed (missing) metacarpophalangeal (MCP) and wrist flexion/extension, simultaneously or independently, while four EMG signals were recorded from forearm muscles. To add variation to the EMG signals, the subjects repeated the MCP and wrist motions at various upper extremity postures. For each subject, the EMG signals collected from the neutral posture were used to build the EMG interfaces; the EMG signals collected from all postures were used to evaluate the interfaces. The performance of the interface was quantified by Pearson's correlation coefficient (r) and the normalized root mean square error (NRMSE) between measured and estimated joint angles. The results demonstrated that the MM predicted movements more accurately, with higher r values and lower NRMSE, than either LR or ANN. Similar results were observed in the transradial amputee. Additionally, the variation in r across postures, an indicator of reliability against posture changes, was significantly lower (better) for the MM than for either LR or ANN. Our findings suggest that incorporating musculoskeletal knowledge into EMG-based human-machine interfaces could improve the estimation of continuous, coordinated motion.
Collapse
|