MyoNet: A Transfer-Learning-Based LRCN for Lower Limb Movement Recognition and Knee Joint Angle Prediction for Remote Monitoring of Rehabilitation Progress From sEMG

Meta-Transfer-Learning-Based Multimodal Human Pose Estimation for Lower Limbs

Accurate and reliable human pose estimation (HPE) is essential in interactive systems, particularly for applications requiring personalized adaptation, such as controlling cooperative robots and wearable exoskeletons, especially for healthcare monitoring equipment. However, continuously maintaining diverse datasets and frequently updating models for individual adaptation are both resource intensive and time-consuming. To address these challenges, we propose a meta-transfer learning framework that integrates multimodal inputs, including high-frequency surface electromyography (sEMG), visual-inertial odometry (VIO), and high-precision image data. This framework improves both accuracy and stability through a knowledge fusion strategy, resolving the data alignment issue, ensuring seamless integration of different modalities. To further enhance adaptability, we introduce a training and adaptation framework with few-shot learning, facilitating efficient updating of encoders and decoders for dynamic feature adjustment in real-time applications. Experimental results demonstrate that our framework provides accurate, high-frequency pose estimations, particularly for intra-subject adaptation. Our approach enables efficient adaptation to new individuals with only a few new samples, providing an effective solution for personalized motion analysis with minimal data.

Transfer learning-enhanced CNN-GRU-attention model for knee joint torque prediction

Introduction

Accurate prediction of joint torque is critical for preventing injury by providing precise insights into the forces acting on joints during activities. Traditional approaches, including inverse dynamics, EMG-driven neuromusculoskeletal (NMS) models, and standard machine learning methods, typically use surface EMG (sEMG) signals and kinematic data. However, these methods often struggle to reveal the complex, non-linear relationship between muscle activation and joint motion, particularly with complex or unfamiliar movements. The generalization of joint torque estimation models across different individuals faces a significant challenge, as feature transferability tends to decline in higher, task-specific layers, reducing model performance.

Methods

In this study, we proposed a CNN-GRU-Attention neural network model combining a neuromusculoskeletal (NMS) solver-informed (hybrid-CNN) augmented with transfer learning, designed to predict knee joint torque with higher accuracy. The neural network was trained using EMG signals, joint angles, and muscle forces as inputs to predict knee joint torque in different activities, and the predictive performance of the model was evaluated both within and between subjects. Additionally, we have developed a transfer learning method in the inter-subject model, which improved the accuracy of knee torque prediction by transferring the learning knowledge of previous participants to new participants.

Results

Our results showed that the hybrid-CNN model can predict knee joint torque within subjects with a significantly lower error (root mean square error ≤0.16 Nm/kg). A transfer learning technique was adopted in the inter-subject tests to significantly improve the generalizability with a lower error (root mean square error ≤0.14 Nm/kg).

Conclusion

The transfer learning-enhanced CNN-GRU-Attention with the NMS model shows great potential in the prediction of knee joint torque.

Real-Time Continuous Locomotion Mode Recognition and Transition Prediction for Human With Lower Limb Exoskeleton

Real-time continuous locomotion mode recognition and seamless timely transition detection is critical for the exoskeleton robot. This study aims to present a comprehensive and innovative framework for locomotion mode recognition and transition prediction, exclusively utilizing inertial measurement unit (IMU) signals from the exoskeleton. In this framework, a CNN-BiLSTM model was developed and trained to be the classifier and a novel majority filter was designed to reduce the transition misjudgment rate. Moreover, a comprehensive evaluation system encompassing eight dimensions for the classifier, incorporating evaluation metrics specifically for transition misjudgment, was proposed. We collected locomotion motion data from six subjects wearing a rigid exoskeleton robot using six IMU sensors on the exoskeleton. The proposed method achieves a high level of recognition accuracy, with an overall average of 99.58 for the five steady locomotion modes (level ground walking (LG), stair ascent/descent (SA/SD), and ramp ascent/descent (RA/RD)) across six subjects following the transition decision. All transitions are recognizable, and the majority can be predicted in advance, with an average prediction time of 353 ms. Furthermore, the implementation of majority filter resulted in an average 87.04 reduction in the transition misjudgment rate among six subjects, thereby decreasing the average transition misjudgment rate to 4.82. Finally, the model was tested on a Jetson Nano to verify its real-time performance. The results presented above were obtained under the condition where either leg could function as the first transition leg and revealed that the developed system was capable of achieving precise locomotion mode recognition and timely transition prediction, with high real-time performance.

A Fusion Dimension Reduction Method for the Features of Surface Electromyographic Signals

Surface electromyographic signals (sEMG) usually have high-dimensional properties, and direct processing of these data consumes significant computational resources. Dimensionality reduction processing can reduce the dimension of the data and improve the real-time performance and response speed. This is especially important for application scenarios such as prosthetic control and rehabilitation training where rapid feedback is required. This paper proposes a feature fusion dimension reduction method for sEMG signals. This method is constructed based on the unique correlation between the features of sEMG. To test the performance of the new dimension reduction method, the sEMG signals from five leg movements were collected from eight subjects and the classification of the feature matrix before and after dimension reduction was tested by six classifiers. The results show that the feature matrix after fusion dimension reduction has excellent classification performance in the subsequent classification tasks. It produces up to 98.3% accuracy. And the highest comprehensive evaluation index can reach 0.9958. This paper also compares the new method with three commonly used dimensionality reduction methods. The results show that the performance of the new method is not only optimal but also extremely stable. Because its classification performance will not be lower than other dimensionality reduction methods due to the change of classifiers. This confirms that the new method has a higher utility value in sEMG signals processing compared to other dimension reduction methods.

Prediction of medial knee contact force using multisource fusion recurrent neural network and transfer learning

Estimation of knee contact force (KCF) during gait provides essential information to evaluate knee joint function. Machine learning has been employed to estimate KCF because of the advantages of low computational cost and real-time. However, the existing machine learning models do not adequately consider gait-related data's temporal-dependent, multidimensional, and highly heterogeneous nature. This study is aimed at developing a multisource fusion recurrent neural network to predict the medial condyle KCF. First, a multisource fusion long short-term memory (MF-LSTM) model was established. Then, we developed a transfer learning strategy based on the MF-LSTM model for subject-specific medial KCF prediction. Four subjects with instrumented tibial prostheses were obtained from the literature. The results showed that the MF-LSTM model could predict medial KCF to a certain high level of accuracy (the mean of ρ = 0.970). The transfer learning model improved the prediction accuracy (the mean of ρ = 0.987). This study shows that the MF-LSTM model is a powerful and accurate computational tool for medial KCF prediction. Introducing transfer learning techniques could further improve the prediction performance for the target subject. This coupling strategy can help clinicians accurately estimate and track joint contact forces in real time.

3D Knee and Hip Angle Estimation With Reduced Wearable IMUs via Transfer Learning During Yoga, Golf, Swimming, Badminton, and Dance

Wearable lower-limb joint angle estimation using a reduced inertial measurement unit (IMU) sensor set could enable quick, economical sports injury risk assessment and motion capture; however the vast majority of existing research requires a full IMU set attached to every related body segment and is implemented in only a single movement, typically walking. We thus implemented 3-dimensional knee and hip angle estimation with a reduced IMU sensor set during yoga, golf, swimming (simulated lower body swimming in a seated posture), badminton, and dance movements. Additionally, current deep-learning models undergo an accuracy drop when tested with new and unseen activities, which necessitates collecting large amounts of data for the new activity. However, collecting large datasets for every new activity is time-consuming and expensive. Thus, a transfer learning (TL) approach with long short-term memory neural networks was proposed to enhance the model's generalization ability towards new activities while minimizing the need for a large new-activity dataset. This approach could transfer the generic knowledge acquired from training the model in the source-activity domain to the target-activity domain. The maximum improvement in estimation accuracy (RMSE) achieved by TL is 23.6 degrees for knee flexion/extension and 22.2 degrees for hip flexion/extension compared to without TL. These results extend the application of motion capture with reduced sensor configurations to a broader range of activities relevant to injury prevention and sports training. Moreover, they enhance the capacity of data-driven models in scenarios where acquiring a substantial amount of training data is challenging.

AI in Rehabilitation Medicine: Opportunities and Challenges

Artificial intelligence (AI) tools are increasingly able to learn from larger and more complex data, thus allowing clinicians and scientists to gain new insights from the information they collect about their patients every day. In rehabilitation medicine, AI can be used to find patterns in huge amounts of healthcare data. These patterns can then be leveraged at the individual level, to design personalized care strategies and interventions to optimize each patient's outcomes. However, building effective AI tools requires many careful considerations about how we collect and handle data, how we train the models, and how we interpret results. In this perspective, we discuss some of the current opportunities and challenges for AI in rehabilitation. We first review recent trends in AI for the screening, diagnosis, treatment, and continuous monitoring of disease or injury, with a special focus on the different types of healthcare data used for these applications. We then examine potential barriers to designing and integrating AI into the clinical workflow, and we propose an end-to-end framework to address these barriers and guide the development of effective AI for rehabilitation. Finally, we present ideas for future work to pave the way for AI implementation in real-world rehabilitation practices.

Prediction of Joint Angles Based on Human Lower Limb Surface Electromyography

Wearable exoskeletons can help people with mobility impairments by improving their rehabilitation. As electromyography (EMG) signals occur before movement, they can be used as input signals for the exoskeletons to predict the body's movement intention. In this paper, the OpenSim software is used to determine the muscle sites to be measured, i.e., rectus femoris, vastus lateralis, semitendinosus, biceps femoris, lateral gastrocnemius, and tibial anterior. The surface electromyography (sEMG) signals and inertial data are collected from the lower limbs while the human body is walking, going upstairs, and going uphill. The sEMG noise is reduced by a wavelet-threshold-based complete ensemble empirical mode decomposition with adaptive noise (CEEMDAN) reduction algorithm, and the time-domain features are extracted from the noise-reduced sEMG signals. Knee and hip angles during motion are calculated using quaternions through coordinate transformations. The random forest (RF) regression algorithm optimized by cuckoo search (CS), shortened as CS-RF, is used to establish the prediction model of lower limb joint angles by sEMG signals. Finally, root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R2) are used as evaluation metrics to compare the prediction performance of the RF, support vector machine (SVM), back propagation (BP) neural network, and CS-RF. The evaluation results of CS-RF are superior to other algorithms under the three motion scenarios, with optimal metric values of 1.9167, 1.3893, and 0.9815, respectively.

CNN-LSTM Model for Recognizing Video-Recorded Actions Performed in a Traditional Chinese Exercise

Identifying human actions from video data is an important problem in the fields of intelligent rehabilitation assessment. Motion feature extraction and pattern recognition are the two key procedures to achieve such goals. Traditional action recognition models are usually based on the geometric features manually extracted from video frames, which are however difficult to adapt to complex scenarios and cannot achieve high-precision recognition and robustness. We investigate a motion recognition model and apply it to recognize the sequence of complicated actions of a traditional Chinese exercise (ie, Baduanjin). We first developed a combined convolutional neural network (CNN) and long short-term memory (LSTM) model for recognizing the sequence of actions captured in video frames, and applied it to recognize the actions of Baduanjin. Moreover, this method has been compared with the traditional action recognition model based on geometric motion features in which Openpose is used to identify the joint positions in the skeletons. Its performance of high recognition accuracy has been verified on the testing video dataset, containing the video clips from 18 different practicers. The CNN-LSTM recognition model achieved 96.43% accuracy on the testing set; while those manually extracted features in the traditional action recognition model were only able to achieve 66.07% classification accuracy on the testing video dataset. The abstract image features extracted by the CNN module are more effective on improving the classification accuracy of the LSTM model. The proposed CNN-LSTM based method can be a useful tool in recognizing the complicated actions.

A CNN-LSTM model for six human ankle movements classification on different loads

This study aims to address three problems in current studies in decoding the ankle movement intention for robot-assisted bilateral rehabilitation using surface electromyogram (sEMG) signals: (1) only up to four ankle movements could be identified while six ankle movements should be classified to provide better training; (2) feeding the raw sEMG signals directly into the neural network leads to high computational cost; and (3) load variation has large influence on classification accuracy. To achieve this, a convolutional neural network (CNN)-long short-term memory (LSTM) model, a time-domain feature selection method of the sEMG, and a two-step method are proposed. For the first time, the Boruta algorithm is used to select time-domain features of sEMG. The selected features, rather than raw sEMG signals are fed into the CNN-LSTM model. Hence, the number of model's parameters is reduced from 331,938 to 155,042, by half. Experiments are conducted to validate the proposed method. The results show that our method could classify six ankle movements with relatively good accuracy (95.73%). The accuracy of CNN-LSTM, CNN, and LSTM models with sEMG features as input are all higher than that of corresponding models with raw sEMG as input. The overall accuracy is improved from 73.23% to 93.50% using our two-step method for identifying the ankle movements with different loads. Our proposed CNN-LSTM model have the highest accuracy for ankle movements classification compared with CNN, LSTM, and Support Vector Machine (SVM).

Low-Density sEMG-Based Pattern Recognition of Unrelated Movements Rejection for Wrist Joint Rehabilitation

sEMG-based pattern recognition commonly assumes a limited number of target categories, and the classifiers often predict each target category depending on probability. In wrist rehabilitation training, the patients may make movements that do not belong to the target category unconsciously. However, most pattern recognition methods can only identify limited patterns and are prone to be disturbed by abnormal movement, especially for wrist joint movements. To address the above the problem, a sEMG-based rejection method for unrelated movements is proposed to identify wrist joint unrelated movements using center loss. In this paper, the sEMG signal collected by the Myo armband is used as the input of the sEMG control method. First, the sEMG signal is processed by sliding signal window and image coding. Then, the CNN with center loss and softmax loss is used to describe the spatial information from the sEMG image to extract discriminative features and target movement recognition. Finally, the deep spatial information is used to train the AE to reject unrelated movements based on the reconstruction loss. The results show that the proposed method can realize the target movements recognition and reject unrelated movements with an F-score of 93.4% and a rejection accuracy of 95% when the recall is 0.9, which reveals the effectiveness of the proposed method.

Recent advancements in digital health management using multi-modal signal monitoring

Healthcare is the method of keeping or enhancing physical and mental well-being with its aid of illness and injury prevention, diagnosis, and treatment. The majority of conventional healthcare practices involve manual management and upkeep of client demographic information, case histories, diagnoses, medications, invoicing, and drug stock upkeep, which can result in human errors that have an impact on clients. By linking all the essential parameter monitoring equipment through a network with a decision-support system, digital health management based on Internet of Things (IoT) eliminates human errors and aids the doctor in making more accurate and timely diagnoses. The term "Internet of Medical Things" (IoMT) refers to medical devices that have the ability to communicate data over a network without requiring human-to-human or human-to-computer interaction. Meanwhile, more effective monitoring gadgets have been made due to the technology advancements, and these devices can typically record a few physiological signals simultaneously, including the electrocardiogram (ECG) signal, the electroglottography (EGG) signal, the electroencephalogram (EEG) signal, and the electrooculogram (EOG) signal. Yet, there has not been much research on the connection between digital health management and multi-modal signal monitoring. To bridge the gap, this article reviews the latest advancements in digital health management using multi-modal signal monitoring. Specifically, three digital health processes, namely, lower-limb data collection, statistical analysis of lower-limb data, and lower-limb rehabilitation via digital health management, are covered in this article, with the aim to fully review the current application of digital health technology in lower-limb symptom recovery.

A Hybrid Method Integrating A Musculoskeletal Model with Long Short-Term Memory (LSTM) for Human Motion Prediction

So far, it shows a growing interest in the biomechanics community in the development of wearable technologies and their clinical applications, which enables the diagnosis of movement disorders and design of the rehabilitation interventions. To provide reliable feedback in the human-machine interface for advanced rehabilitation devices, methods to predict motion intention was developed which aim to generate future human motion based on the measured motion. An inertial measurement unit (IMU) is a promising device for motion tracking, with the advantages of low cost and high convenience in sensor placement to measure motion in almost every environment. However, it reveals that few contributions have been devoted to human motion prediction with pure IMU data. Thus, we propose a hybrid method integrating a musculoskeletal (MSK) model and the long short-term memory (LSTM) artificial neural network (ANN) to predict human motion. The proposed method was capable to predict motion in the daily tasks (stand-to-sit-to-stand and walking) for healthy participants: the predicted knee joint angles had an RMSE of 2.93° when compared to measured knee joint angles from the IMU data. The proposed method outperformed the methods based on the ANN/MSK model (RMSE of 31.15°) and LSTM without the integration of the MSK model (RMSE of 31.26°) in the motion prediction. Clinical Relevance- This proposed model based on IMU data alone has the great potential to become a low-cost, easy-to-use alternative in motion prediction to interact with advanced rehabilitation devices in clinical practice.

Multi-Day EMG-Based Knee Joint Torque Estimation Using Hybrid Neuromusculoskeletal Modelling and Convolutional Neural Networks

Proportional control using surface electromyography (EMG) enables more intuitive control of a transfemoral prosthesis. However, EMG is a noisy signal which can vary over time, giving rise to the question what approach for knee torque estimation is most suitable for multi-day control. In this study we compared three different modelling frameworks to estimate knee torque in non-weight-bearing situations. The first model contained a convolutional neural network (CNN) which mapped EMG to knee torque directly. The second used a neuromusculoskeletal model (NMS) which used EMG, muscle tendon unit lengths and moment arms to compute knee torque. The third model (Hybrid) used a CNN to map EMG to specific muscle activation, which was used together with NMS components to compute knee torque. Multi-day measurements were conducted on ten able-bodied participants who performed non-weight bearing activities. CNN had the best performance in general and on each day (Normalized Root Mean Squared Error (NRMSE) 9.2 ± 4.4%). The Hybrid model (NRMSE 12.4 ± 3.4%) was able to outperform NMS (NRMSE 14.3 ± 4.2%). The NMS model showed no significant difference between measurement days. The CNN model and Hybrid models had significant performance differences between the first day and all other days. CNNs are suited for multi-day torque estimation in terms of error rate, outperforming the other two model types. NMS was the only model type which was robust over all days. This study investigated the behavior of three model types over multiple days, giving insight in the most suited modelling approach for multi-day torque estimation to be used in prosthetic control.

Human Joint Torque Estimation Based on Mechanomyography for Upper Extremity Exosuit

Human intention recognition belongs to the algorithm basis for exoskeleton robots to generate synergic movements and provide corresponding assistance. In this article, we acquire and analyze the mechanomyography (MMG) to estimate the current joint torque and apply this method to the rehabilitation training research of the upper extremity exosuit. In order to obtain relatively pure biological signals, a MMG processing method based on the Hilbert-Huang Transform (HHT) is proposed to eliminate the mixed noise and motion artifacts. After extracting features and forming the dataset, a random forest regression (RFR) model is designed to build the mapping relationship between MMG and human joint output through offline learning. In addition, an upper extremity exosuit is constructed for multi-joint assistance. Based on the above research, we develop a torque estimation-based control strategy and make it responsible for the intention understanding and motion servo of this customized system. Finally, an actual test verifies the accuracy and reliability of this recognition algorithm, and an efficiency evaluation experiment also proves the feasibility for power assistance.

Time-domain Mixup Source Data Augmentation of sEMGs for Motion Recognition towards Efficient Style Transfer Mapping

Motion recognition based on surface electromyogram (sEMG) recorded from the forearm is attracting attention for its applicability because it easily integrates with wearable devices and has a high signal-to-noise ratio. Inter-subject variability and inadequate data availability are common problems encountered in classifiers. Transfer learning (TL) techniques can reduce the inter-subject variability; however, when the amount of data recorded from each source subject is small, the TL-combined classifier is prone to overfitting problems. In this study, we tested the accuracy of motion recognition with and without TL when the source dataset was increased up to 10 times with a time-domain data augmentation method called mixup. The performance was evaluated using an 8-class sEMG dataset containing wearable sensing data from 25 subjects. We found that mixup improved the performance of TL-combined classifiers (support vector machine and 4-layered fully connected feedforward neural network). In future work, we plan to investigate the relationship between the amount of data and sEMG-based motion recognition by comparing multiple sEMG datasets and multiple data augmentation methods.

EMG and Joint Angle-Based Machine Learning to Predict Future Joint Angles at the Knee

Electromyography (EMG) is commonly used to measure electrical activity of the skeletal muscles. As exoskeleton technology advances, these signals may be used to predict human intent for control purposes. This study used an artificial neural network trained and tested with knee flexion angles and knee muscle EMG signals to predict knee flexion angles during gait at 50, 100, 150, and 200 ms into the future. The hypothesis of this study was that the algorithm's prediction accuracy would only be affected by time into the future, not subject, gender or side, and that as time into the future increased, the prediction accuracy would decrease. A secondary hypothesis was that as the number of algorithm training trials increased, the prediction accuracy of the artificial neural network (ANN) would increase. The results of this study indicate that only time into the future affected the accuracy of knee flexion angle prediction (p < 0.001), whereby greater time resulted in reduced accuracy (0.68 to 4.62 degrees root mean square error (RMSE) from 50 to 200 ms). Additionally, increased number of training trials resulted in increased angle prediction accuracy.

Locomo-Net: A Low -Complex Deep Learning Framework for sEMG-Based Hand Movement Recognition for Prosthetic Control

Background: The enhancement in the performance of the myoelectric pattern recognition techniques based on deep learning algorithm possess computationally expensive and exhibit extensive memory behavior. Therefore, in this paper we report a deep learning framework named 'Low-Complex Movement recognition-Net' (LoCoMo-Net) built with convolution neural network (CNN) for recognition of wrist and finger flexion movements; grasping and functional movements; and force pattern from single channel surface electromyography (sEMG) recording. The network consists of a two-stage pipeline: 1) input data compression; 2) data-driven weight sharing. Methods: The proposed framework was validated on two different datasets- our own dataset (DS1) and publicly available NinaPro dataset (DS2) for 16 movements and 50 movements respectively. Further, we have prototyped the proposed LoCoMo-Net on Virtex-7 Xilinx field-programmable gate array (FPGA) platform and validated for 15 movements from DS1 to demonstrate its feasibility for real-time execution. Results: The effectiveness of the proposed LoCoMo-Net was verified by a comparative analysis against the benchmarked models using the same datasets wherein our proposed model outperformed Twin- Support Vector Machine (SVM) and existing CNN based model by an average classification accuracy of 8.5 % and 16.0 % respectively. In addition, hardware complexity analysis is done to reveal the advantages of the two-stage pipeline where approximately 27 %, 49 %, 50 %, 23 %, and 43 % savings achieved in lookup tables (LUT's), registers, memory, power consumption and computational time respectively. Conclusion: The clinical significance of such sEMG based accurate and low-complex movement recognition system can be favorable for the potential improvement in quality of life of an amputated persons.