Interface Prostheses With Classifier-Feedback-Based User Training

HANDSON Hand: Strategies and Approaches for Competitive Success at CYBATHLON 2024

A significant number of people with disabilities rely on assistive devices, yet these technologies often face limitations, including restricted functionality, inadequate user-centered design, and a lack of standardized evaluation metrics. While upper-limb prosthetics remain a key research focus, existing commercial solutions still fall short of meeting daily reliability and usability needs, leading to high abandonment rates. CYBATHLON integrates assistive technologies into daily living tasks, driving innovation and prioritizing user needs. In CYBATHLON 2024, the HANDSON hand secured first place in the arm prosthesis race, showcasing breakthroughs in human-robot integration. This paper presents the HANDSON hand's design, core technologies, training strategies, and competition performance, offering insights for advancing multifunctional prosthetic hands to tackle real-world challenges.

DistaNet: grasp-specific distance biofeedback promotes the retention of myoelectric skills

Objective.An active myoelectric interface responds to the user's muscle signals to enable movements. Machine learning can decode user intentions from myoelectric signals. However, machine learning-based interface control lacks continuous, intuitive feedback about task performance, needed to facilitate the acquisition and retention of myoelectric control skills.Approach.We propose DistaNet as a neural network-based framework that extracts smooth, continuous, and low-dimensional signatures of the hand grasps from multi-channel myoelectric signals and provides grasp-specific biofeedback to the users.Main results.Experimental results show its effectiveness in decoding user gestures and providing biofeedback, helping users retain the acquired motor skills.Significance.We demonstrates myoelectric skill retention in a pattern recognition setting for the first time.

Using Adaptive Directed Acyclic Graph for Human In-Hand Motion Identification with Hybrid Surface Electromyography and Kinect

The multi-fingered dexterous robotic hand is increasingly used to achieve more complex and sophisticated human-like manipulation tasks on various occasions. This paper proposes a hybrid Surface Electromyography (SEMG) and Kinect-based human in-hand motion (HIM) capture system architecture for recognizing complex motions of the humans by observing the state information between an object and the human hand, then transferring the manipulation skills into bionic multi-fingered robotic hand realizing dexterous in-hand manipulation. First, an Adaptive Directed Acyclic Graph (ADAG) algorithm for recognizing HIMs is proposed and optimized based on the comparison of multi-class support vector machines; second, ten representative complex in-hand motions are demonstrated by ten subjects, and SEMG and Kinect signals are obtained based on a multi-modal data acquisition platform; then, combined with the proposed algorithm framework, a series of data preprocessing algorithms are realized. There is statistical symmetry in similar types of SEMG signals and images, and asymmetry in different types of SEMG signals and images. A detailed analysis and an in-depth discussion are given from the results of the ADAG recognizing HIMs, motion recognition rates of different perceptrons, motion recognition rates of different subjects, motion recognition rates of different multi-class SVM methods, and motion recognition rates of different machine learning methods. The results of this experiment confirm the feasibility of the proposed method, with a recognition rate of 95.10%.

Learning Non-Euclidean Representations With SPD Manifold for Myoelectric Pattern Recognition

How to learn informative representations from Electromyography (EMG) signals is of vital importance for myoelectric control systems. Traditionally, hand-crafted features are extracted from individual EMG channels and combined together for pattern recognition. The spatial topological information between different channels can also be informative, which is seldom considered. This paper presents a radically novel approach to extract spatial structural information within diverse EMG channels based on the symmetric positive definite (SPD) manifold. The object is to learn non-Euclidean representations inside EMG signals for myoelectric pattern recognition. The performance is compared with two classical feature sets using accuracy and F1-score. The algorithm is tested on eleven gestures collected from ten subjects, and the best accuracy reaches 84.85%±5.15% with an improvement of 4.04%~20.25%, which outperforms the contrast method, and reaches a significant improvement with the Wilcoxon signed-rank test. Eleven gestures from three public databases involving Ninapro DB2, DB4, and DB5 are also evaluated, and better performance is observed. Furthermore, the computational cost is less than the contrast method, making it more suitable for low-cost systems. It shows the effectiveness of the presented approach and contributes a new way for myoelectric pattern recognition.

The Influence of Training with Visual Biofeedback on the Predictability of Myoelectric Control Usability

Studies have shown that closed-loop myoelectric control schemes can lead to changes in user performance and behavior compared to open-loop systems. When users are placed within the control loop, such as during real-time use, they must correct for errors made by the controller and learn what behavior is necessary to produce desired outcomes. Augmented feedback, consequently, has been used to incorporate the user throughout the training process and to facilitate learning. This work explores the effect of visual feedback presented during user training on both the performance and predictability of a myoelectric classification-based control system. Our results suggest that properly designed feedback mechanisms and training tasks can influence the quality of the training data and the ability to predict usability using linear combinations of metrics derived from feature space. Furthermore, our results confirm that the most common in-lab training protocol, screen guided training, may yield training data that are less representative of online use than training protocols that incorporate the user in the loop. These results suggest that training protocols should be designed that better parallel the testing environment to more effectively prepare both the algorithms and users for real-time control.

A Multi-Variate Approach to Predicting Myoelectric Control Usability

Pattern recognition techniques leveraging the use of electromyography signals have become a popular approach to provide intuitive control of myoelectric devices. Performance of these control interfaces is commonly quantified using offline classification accuracy, despite studies having shown that this metric is a poor indicator of usability. Researchers have identified alternative offline metrics that better correlate with online performance; however, the relationship has yet to be fully defined in the literature. This has necessitated the continued trial-and-error-style online testing of algorithms developed using offline approaches. To bridge this information divide, we conducted an exploratory study where thirty-two different metrics from the offline training data were extracted. A correlation analysis and an ordinary least squares regression were implemented to investigate the relationship between the offline metrics and six aspects online use. The results indicate that the current offline standard, classification accuracy, is a poor indicator of usability and that other metrics may hold predictive power. The metrics identified in this work also may constitute more representative evaluation criteria when designing and reporting new control schemes. Furthermore, linear combinations of offline training metrics generate substantially more accurate predictions than using individual metrics. We found that the offline metric feature efficiency generated the best predictions for the usability metric throughput. A combination of two offline metrics (mean semi-principal axes and mean absolute value) significantly outperformed feature efficiency alone, with a 166% increase in the predicted R² value (i.e., VEcv). These findings suggest that combinations of metrics could provide a more robust framework for predicting usability.

Evaluating User and Machine Learning in Short- and Long-Term Pattern Recognition-Based Myoelectric Control

Proper training is essential to achieve reliable pattern recognition (PR) based myoelectric control. The amount of training is commonly determined by experience. The purpose of this study is to provide an offline validation method that makes the offline performance transferable to online control and find the proper amount of training that achieves good online performance. In the offline experiment, eight able-bodied subjects and three amputees participated in a ten-day training. Repeatability index (RI) and classification error (CE) were used to evaluate user learning and machine learning, respectively. The performance of cross-validation (CV) and time serial related validation (TSV) was compared. Learning curves were established with different training trials by TSV. In the online experiment, sixteen able-bodied subjects were randomly divided into two groups with one- or five-trial training, respectively, followed by participating in the test with and without classifier-output feedback. The correlation between offline and online tests was analyzed. Results indicated that five-trial training was proper to train the user and the classifier. The long-term retention of skills could not shorten the learning process. The correlation between CEs of TSV and the online test was strong ( r=0.87 ) with five-trial training, while the correlation between CEs of CV and the online test was weak ( r=0.30 ). Outcomes demonstrate that offline performance evaluated by TSV is transferable to online performance and the learning process can guide the user to achieve good online myoelectric control with minimum training.

Attribute-Driven Granular Model for EMG-Based Pinch and Fingertip Force Grand Recognition

Fine multifunctional prosthetic hand manipulation requires precise control on the pinch-type and the corresponding force, and it is a challenge to decode both aspects from myoelectric signals. This paper proposes an attribute-driven granular model (AGrM) under a machine-learning scheme to solve this problem. The model utilizes the additionally captured attribute as the latent variable for a supervised granulation procedure. It was fulfilled for EMG-based pinch-type classification and the fingertip force grand prediction. In the experiments, 16 channels of surface electromyographic signals (i.e., main attribute) and continuous fingertip force (i.e., subattribute) were simultaneously collected while subjects performing eight types of hand pinches. The use of AGrM improved the pinch-type recognition accuracy to around 97.2% by 1.8% when constructing eight granules for each grasping type and received more than 90% force grand prediction accuracy at any granular level greater than six. Further, sensitivity analysis verified its robustness with respect to different channel combination and interferences. In comparison with other clustering-based granulation methods, AGrM achieved comparable pinch recognition accuracy but was of lowest computational cost and highest force grand prediction accuracy.

Arduino-Based Myoelectric Control: Towards Longitudinal Study of Prosthesis Use

Understanding how upper-limb prostheses are used in daily life helps to improve the design and robustness of prosthesis control algorithms and prosthetic components. However, only a very small fraction of published research includes prosthesis use in community settings. The cost, limited battery life, and poor generalisation may be the main reasons limiting the implementation of home-based applications. In this work, we introduce the design of a cost-effective Arduino-based myoelectric control system with wearable electromyogram (EMG) sensors. The design considerations focused on home studies, so the robustness, user-friendly control adjustments, and user supports were the main concerns. Three control algorithms, namely, direct control, abstract control, and linear discriminant analysis (LDA) classification, were implemented in the system. In this paper, we will share our design principles and report the robustness of the system in continuous operation in the laboratory. In addition, we will show a first real-time implementation of the abstract decoder for prosthesis control with an able-bodied participant.

Shoulder muscle activation pattern recognition based on sEMG and machine learning algorithms

BACKGROUND AND OBJECTIVE

Surface electromyography (sEMG) has been used for robotic rehabilitation engineering for volitional control of hand prostheses or elbow exoskeleton, however, using sEMG for volitional control of an upper limb exoskeleton has not been perfectly developed. The long-term goal of our study is to process shoulder muscle bio-electrical signals for rehabilitative robotic assistive device motion control. The purposes of this study included: 1) to test the feasibility of machine learning algorithms in shoulder motion pattern recognition using sEMG signals from shoulder and upper limb muscles, 2) to investigate the influence of motion speed, individual variability, EMG recording device, and the amount of EMG datasets on the shoulder motion pattern recognition accuracy.

METHODS

A novel convolutional neural network (CNN) structure was constructed to process EMG signals from 12 muscles for the pattern recognition of upper arm motions including resting, drinking, backward-forward motion, and abduction motion. The accuracy of the CNN models for pattern recognition under different motion speeds, among individuals, and by EMG recording devices was statistically analyzed using ANOVA, GLM Univariate analysis, and Chi-square tests. The influence of EMG dataset number used for CNN model training on recognition accuracy was studied by gradually increasing dataset number until the highest accuracy was obtained.

RESULTS

Results showed that the accuracy of the normal speed CNN model in motion pattern recognition was 97.57% for normal speed motions and 97.07% for fast speed motions. The accuracy of the cross-subjects CNN model in motion pattern recognition was 79.64%. The accuracy of the cross-device CNN model in motion pattern recognition was 88.93% for normal speed motion and 80.87% for mixed speed. There was a statistical difference in pattern recognition accuracy between different CNN models.

CONCLUSION

The EMG signals of shoulder and upper arm muscles from the upper limb motions can be processed using CNN algorithms to recognize the identical motions of the upper limb including drinking, forward/backward, abduction, and resting. A simple CNN model trained by EMG datasets of a designated motion speed accurately detected the motion patterns of the same motion speed, yielding the highest accuracy compared with other mixed CNN models for various speeds of motion pattern recognition. Increase of the number of EMG datasets for CNN model training improved the pattern recognition accuracy.

Guiding the Training of Users With a Pattern Similarity Biofeedback to Improve the Performance of Myoelectric Pattern Recognition

Next generation prosthetics will rely massively on myoelectric "Pattern Recognition" (PR) based control approaches, to improve their users' dexterity. One major identified factor of successful functioning of these approaches lies in the training of amputees and in their understanding of how those prosthetics works. We thus propose here an intuitive pattern similarity biofeedback which can be easily used to train amputees and allow them to optimize their muscular contractions to improve their control performance. Experiments were conducted on twenty able-bodied participants and one transradial amputee. Their performance in controlling an interface through a myoelectric PR algorithm was evaluated; before and after a short automatic user training session consisting in using the proposed visual biofeedback for ten participants, and using a generic PR algorithm output feedback for the others ten. Participants who were trained with the proposed biofeedback increased their classification score for the retrained gesture (by 39.4%), without affecting the overall classification performance (which progressed by 10.2%) through over-training and increase of False Positive rate as observed in the control group. Additional analysis indicates a clear change in contraction strategy only in the group who used the proposed biofeedback. These preliminary results highlight the potential of this method which does not focus so much on over-optimizing the pattern recognition algorithm or on physically training the users, but on providing them simple and intuitive information to adapt or change their motor strategies to solve some misclassification issues.

Myoelectric Control for Upper Limb Prostheses

State-of-the-art high-end prostheses are electro-mechanically able to provide a great variety of movements. Nevertheless, in order to functionally replace a human limb, it is essential that each movement is properly controlled. This is the goal of prosthesis control, which has become a growing research field in the last decades, with the ultimate goal of reproducing biological limb control. Therefore, exploration and development of prosthesis control are crucial to improve many aspects of an amputee’s life. Nowadays, a large divergence between academia and industry has become evident in commercial systems. Although several studies propose more natural control systems with promising results, basic one degree of freedom (DoF), a control switching system is the most widely used option in industry because of simplicity, robustness and inertia. A few classification controlled prostheses have emerged in the last years but they are still a low percentage of the used ones. One of the factors that generate this situation is the lack of robustness of more advanced control algorithms in daily life activities outside of laboratory conditions. Because of this, research has shifted towards more functional prosthesis control. This work reviews the most recent literature in upper limb prosthetic control. It covers commonly used variants of possible biological inputs, its processing and translation to actual control, mostly focusing on electromyograms as well as the problems it will have to overcome in near future.

Learn the Temporal-Spatial Feature of sEMG via Dual-Flow Network

Surface electromyography (sEMG) signals have been widely used in human–machine interaction, providing more nature control expedience for external devices. However, due to the instability of sEMG, it is hard to extract consistent sEMG patterns for motion recognition. This paper proposes a dual-flow network to extract the temporal-spatial feature of sEMG for gesture recognition. The proposed network model uses convolutional neural network (CNN) and long short-term memory methods (LSTM) to, respectively, extract the spatial feature and temporal feature of sEMG, simultaneously. These features extracted by CNN and LSTM are merged into temporal-spatial feature to form an end-to-end network. A dataset was constructed for testing the performance of the network. In this database, the average recognition accuracy by using our dual-flow model reached 78.31%, which was improved by 6.69% compared to the baseline CNN (71.67%). In addition, NinaPro DB1 is also used to evaluate the proposed methods, receiving 1.86% higher recognition accuracy than the baseline CNN classifier. It is believed that the proposed dual-flow network owns the merit in extracting stable sEMG feature for gesture recognition, and can be further applied into practical applications.

An Optimized Selection Method of Channel Numbers and Electrode Layouts for Hand Motion Recognition

The channel numbers and electrode layouts are usually determined empirically that would reduce robustness when acquiring surface electromyography (EMG) signals for prosthetic hand systems. It is necessary to study how they can be exploited effectively for a more accurate extraction. In response to the problem, an experiment is designed that establishes the relationship between sEMG signals and forearm muscles based on signal-to-noise ratio (SNR). The SNR of sEMG signals in different sampling channels can be calculated and compared, and then the potential contribution of each channel during different hand motions will be evaluated comprehensively. The active muscle regions can be obtained from the established relationship that is a useful reference for feature extraction. Finally, the relations between the computational cost, channel numbers and electrode layouts are explored. The findings of this paper support the idea that the accuracy of pattern recognition will not be affected when reducing the redundant electrodes.

Facilitate sEMG-Based Human–Machine Interaction Through Channel Optimization

Electromyography (EMG) has been widely accepted to interact with prosthetic hands, but still limited to using few channels for the control of few degrees of freedom. The use of more channels can improve the controllability, but it also increases system’s complexity and reduces its wearability. It is yet clear if optimizely placing the EMG channel could provide a feasible solution to this challenge. This study customized a genetic algorithm to optimize the number of channels and its position on the forearm in inter-day hand gesture recognition scenario. Our experimental results demonstrate that optimally selected 14 channels out of 16 can reach a peak inter-day hand gesture recognition accuracy at 72.3%, and optimally selecting 9 and 11 channels would reduce the performance by 3% and 10%. The cross-validation results also demonstrate that the optimally selected EMG channels from five subjects also work on the rest of the subjects, improving the accuracies by 3.09% and 4.5% in 9- and 11-channel combination, respectively. In sum, this study demonstrates the feasibility of channel reduction through genetic algorithm, and preliminary proves the significance of EMG channel optimization for human–machine interaction.

EMG Pattern Recognition Using Convolutional Neural Network with Different Scale Signal/Spectra Input

Deep learning (DL) has made tremendous contributions to image processing. Recently, the DL has also attracted attention in the specialized field of neural decoding from raw myoelectric signals (electromyograms, EMGs). However, to our knowledge, most existing methods require some measure of preprocessing of the raw EMGs. Moreover, research to date has not accounted for the variability in the signal during time sequences. In this paper, we propose a new convolutional neural network (CNN) structure that can directly process raw EMG signals for hand gesture classification. More specifically, we assess the effects of various window sizes and of two different EMG representations (time sequence and frequency spectra) on open EMG datasets. We found that the frequency spectra derived from raw EMGs is more suitable as the model input in the task of gesture classification. Meanwhile, the combination use of long window could improve the classification accuracy (CA) and the window of 1024 ms achieved the best results on two open datasets ([Formula: see text]% and [Formula: see text]%). Further, our model requires no feature extraction procedures and is comparable with the optimal combination of features and classifier used by the traditional methods in the performance of specific tasks.

Towards Wearable A-Mode Ultrasound Sensing for Real-Time Finger Motion Recognition

It is evident that surface electromyography (sEMG) based human-machine interfaces (HMI) have inherent difficulty in predicting dexterous musculoskeletal movements such as finger motions. This paper is an attempt to investigate a plausible alternative to sEMG, ultrasound-driven HMI, for dexterous motion recognition due to its characteristic of detecting morphological changes of deep muscles and tendons. A multi-channel A-mode ultrasound lightweight device is adopted to evaluate the performance of finger motion recognition; an experiment is designed for both widely acceptable offline and online algorithms with eight able-bodied subjects employed. The experiment result presents that the offline recognition accuracy is up to 98.83% ± 0.79%. The real-time motion completion rate is 95.4% ± 8.7% and online motion selection time is 0.243 ± 0.127 s. The outcomes confirm the feasibility of A-mode ultrasound based wearable HMI and its prosperous applications in prosthetic devices, virtual reality, and remote manipulation.

Two ways to improve myoelectric control for a transhumeral amputee after targeted muscle reinnervation: a case study

BACKGROUND

Myoelectric control of multifunctional prostheses is challenging for individuals with high-level amputations due to insufficient surface electromyography (sEMG) signals. A surgical technique called targeted muscle reinnervation (TMR) has achieved impressive improvements in myoelectric control by providing more sEMG control signals. In this case, some channels of sEMG signals are coupled after TMR, which limits the performance of conventional amplitude-based control for upper-limb prostheses.

METHODS

In this paper, two different ways (training and algorithms) were attempted to solve the problem in a transhumeral amputee after TMR. Firstly, effect of rehabilitation training on generating independent sEMG signals was investigated. The results indicated that some sEMG signals recorded were still coupled over the targeted muscles after rehabilitation training for about two months. Secondly, pattern recognition (PR) algorithm was then applied to classify the sEMG signals. In the second way, to further improve the real-time performance of prosthetic control, a post-processing method named as mean absolute value-based (MAV-based) threshold switches was utilized.

RESULTS

Using the improved algorithms, substantial improvement was shown in a subset of the modified Action Research Arm Test (ARAT). Compared with common PR control without post-processing method, the total scores increased more than 18% with majority vote and more than 58% with MAV-based threshold switches. The amputee was able to finish all the tasks within the allotted time with the standard MAV-based threshold switches. Subjectively the amputee preferred the PR control with MAV-based threshold switches and reported it to be more accurate and much smoother both in experiment and practical use.

CONCLUSIONS

Although the sEMG signals were still coupled after rehabilitation training on the TMR patient, the online performance of the prosthetic operation was improved through application of PR control with combination of the MAV-based threshold switches.

TRIAL REGISTRATION

Retrospectively registered http://www.chictr.org.cn/showproj.aspx?proj=22058 .

Centered and Averaged Fuzzy Entropy to Improve Fuzzy Entropy Precision

Several entropy measures are now widely used to analyze real-world time series. Among them, we can cite approximate entropy, sample entropy and fuzzy entropy (FuzzyEn), the latter one being probably the most efficient among the three. However, FuzzyEn precision depends on the number of samples in the data under study. The longer the signal, the better it is. Nevertheless, long signals are often difficult to obtain in real applications. This is why we herein propose a new FuzzyEn that presents better precision than the standard FuzzyEn. This is performed by increasing the number of samples used in the computation of the entropy measure, without changing the length of the time series. Thus, for the comparisons of the patterns, the mean value is no longer a constraint. Moreover, translated patterns are not the only ones considered: reflected, inversed, and glide-reflected patterns are also taken into account. The new measure (so-called centered and averaged FuzzyEn) is applied to synthetic and biomedical signals. The results show that the centered and averaged FuzzyEn leads to more precise results than the standard FuzzyEn: the relative percentile range is reduced compared to the standard sample entropy and fuzzy entropy measures. The centered and averaged FuzzyEn could now be used in other applications to compare its performances to those of other already-existing entropy measures.