Activities of Daily Living-based Rehabilitation System for Arm and Hand Motor Function Retraining after Stroke

Dual-Modal Gesture Recognition Using Adaptive Weight Hierarchical Soft Voting Mechanism

Muscle force and morphology information offer complementary perspectives for gesture recognition and its applications. Surface Electromyography (sEMG) provides force and electrophysiological information associated with muscles, while A-mode ultrasound (AUS) reveals muscle morphological information. By leveraging these two modalities, more comprehensive muscle motor unit information relevant to gesture recognition can be obtained. In this article, we introduce the adaptive weight classification (AWC) module and its enhanced version with hierarchical classifiers, adaptive weight hierarchical soft voting (AWHSV), to integrate AUS and sEMG into a fused modality. This approach dynamically adjusts the weights of individual and fused features, compensating for lost details during fusion, leading to a richer information representation and significantly improving algorithm robustness in gesture recognition. The experimental results demonstrate that the proposed method achieves recognition rates that are 0.66%, 2.36%, and 1.30% higher than those of its counterparts using sEMG, AUS, and sEMG-AUS, respectively. Moreover, the method outperforms state-of-the-art approaches, confirming its effectiveness in gesture recognition across both single and multiple modalities. This work demonstrates the advantages of the proposed AWHSV method, providing broader application scenarios for gesture recognition.

Quantifying shoulder motion in the free-living environment using wearable inertial measurement units: Challenges and recommendations

Understanding function and dysfunction of the shoulder may be best addressed by capturing the motion of the shoulder in the unstructured, free-living environment where the magnitudes and frequencies of required daily motion can be quantified. Miniaturized wearable inertial measurement units (IMUs) enable measurement of shoulder motion in the free-living environment; however, there are challenges in using IMU-based data to estimate traditionally used measures of shoulder motion from lab-based motion capture. There are limited options for IMU placement/fixation that minimize soft tissue effects and there are significant challenges in developing the algorithms that can accurately estimate shoulder joint angles from IMU measurements of acceleration and angular velocity. In an effort to collate current knowledge and highlight solutions to addressable challenges, in this paper, we report the results of a focused search of research articles using IMUS for kinematic measurements of the shoulder in the free-living environment, discuss the basic steps required for quantifying shoulder motion in the non-laboratory field-based setting using wearable IMUs, and we discuss the challenges that must be overcome in the context of the shoulder joint and the literature review. Finally, we suggest some IMU-based measures that are less sensitive to experimental design and algorithm choices, make recommendations for the information documented in manuscripts describing studies that use IMUs to quantify shoulder motion, and propose directions for future research.

Generalized Cross-Domain Framework for Gesture Recognition via Wrist-Worn Sensing

Wearable sensing technology offers a natural and convenient means of human-computer interaction, particularly for gesture recognition, yet domain shifts in wrist-worn single-site sensing pose significant challenges for cross-domain gesture recognition. To address this, we proposed a generalized cross-domain framework for fine-grained gesture recognition using wrist-worn single-site sensing. Concretely, we presented a Multi-Branch Network, which combines feature-level multimodal fusion with enhanced inter-modal interaction to effectively capture fine-grained gestures. To this end, we constructed a multimodal dataset, which comprises fifteen static and eighteen dynamic gestures. Furthermore, we developed five fine-tuning strategies and evaluated them across the paradigms of cross-session, cross-subject, cross-gesture, and cross-modality. Through comprehensive analyses, this study provides valuable insights into the selection of optimal fine-tuning strategies and elucidates the internal mechanisms underlying multiple cross-domain paradigms. To investigate the intricate trade-off between recognition accuracy and computational cost, we applied nonlinear least squares to construct the Accuracy-Cost trade-off functions. Experimental findings indicated that the optimal transfer learning ratios for these cross-domain paradigms ranged from 6.1% to 9.0%, with most clustering around 9.0%, offering a valuable reference for determining optimal transfer learning ratios within diverse cross-domain scenarios. Additionally, we implemented a real-time online gesture recognition system, validating the feasibility of our approach through preliminary tests in real-world scenarios. In conclusion, this study serves as a preliminary investigation into the application of wrist-worn single-site sensing for fine-grained gesture recognition.

On the Benefit of FMG and EMG Sensor Fusion for Gesture Recognition Using Cross-Subject Validation

Hand gestures are a natural form of human communication, making gesture recognition a sensible approach for intuitive human-computer interaction. Wearable sensors on the forearm can be used to detect the muscle contractions that generate these gestures, but classification approaches relying on a single measured modality lack accuracy and robustness. In this work, we analyze sensor fusion of force myography (FMG) and electromyography (EMG) for gesture recognition. We employ piezoelectric FMG sensors based on ferroelectrets and a commercial EMG system in a user study with 13 participants to measure 66 distinct hand movements with 10ms labelling precision. Three classification tasks, namely flexion and extension, single finger, and all finger movement classification, are performed using common handcrafted features as input to machine learning classifiers. Subsequently, the evaluation covers the effectiveness of the sensor fusion using correlation analysis, classification performance based on leave-one-subject-out-cross-validation and 5x2cv-t-tests, and its effects of involuntary movements on classification. We find that sensor fusion leads to significant improvement (42% higher average recognition accuracy) on all three tasks and that both sensor modalities contain complementary information. Furthermore, we confirm this finding using reduced FMG and EMG sensor sets. This study reinforces the results of prior research about the effectiveness of sensor fusion by performing meticulous statistical analyses, thereby paving the way for multi-sensor gesture recognition in assistance systems.

Application of a sEMG hand motion recognition method based on variational mode decomposition and ReliefF algorithm in rehabilitation medicine

Hand motion intention recognition has been considered as one of the crucial research fields for prosthetic control and rehabilitation medicine. In recent years, surface electromyogram (sEMG) signals that directly reflect human motion information are ideal input sources for prosthetic control and rehabilitation. However, how to effectively extract components from sEMG signals containing abundant limb movement information to improve the accuracy of hand recognition still is a difficult problem. To achieve this goal, this paper proposes a novel hand motion recognition method based on variational mode decomposition (VMD) and ReliefF. First, VMD is used to decompose the sEMG signal into multiple variational mode functions (VMFs). To efficiently extract the intrinsic components of the sEMG, the recognition performance of different numbers of VMFs is evaluated. Then, four features representing hand motion intentions are extracted from the VMFs to form the initial feature space. Next, the ReliefF algorithm is used to remove redundant features from the feature space. In order to select a feature space that can effectively reflect the intention of hand movements, the hand movement recognition performance of 8 low-dimensional feature spaces is evaluated. Finally, three machine learning methods are used to recognize hand movements. The proposed method was tested on the sEMG for Basic Hand movements Data Set and achieved an average accuracy of 99.14%. Compared with existing research, the proposed method achieves better hand motion recognition performance, indicating the potential for healthcare and rehabilitation applications.

A Review on Assisted Living Using Wearable Devices

Forecasts about the aging trend of the world population agree on identifying increased life expectancy as a serious risk factor for the financial sustainability of social healthcare systems if not properly supported by innovative care management policies. Such policies should include the integration within traditional healthcare services of assistive technologies as tools for prolonging healthy and independent living at home, but also for introducing innovations in clinical practice such as long-term and remote health monitoring. For their part, solutions for active and assisted living have now reached a high degree of technological maturity, thanks to the considerable amount of research work carried out in recent years to develop highly reliable and energy-efficient wearable sensors capable of enabling the development of systems to monitor activity and physiological parameters over time, and in a minimally invasive manner. This work reviews the role of wearable sensors in the design and development of assisted living solutions, focusing on human activity recognition by joint use of onboard electromyography sensors and inertial measurement units and on the acquisition of parameters related to overall physical and psychological conditions, such as heart activity and skin conductance.

Gamified devices for stroke rehabilitation: A systematic review

BACKGROUND AND OBJECTIVE

Rehabilitation after stroke is essential to minimize permanent disability. Gamification, the integration of game elements into non-game environments, has emerged as a promising strategy for increasing motivation and rehabilitation effectiveness. This article systematically reviews the gamified devices used in stroke rehabilitation and evaluates their impact on emotional, social, and personal effects on patients, providing a comprehensive view of gamified rehabilitation.

METHODS

A comprehensive search using the PRISMA 2020 guidelines was conducted using the IEEE Xplore, PubMed, Springer Link, APA PsycInfo, and ScienceDirect databases. Empirical studies published between January 2019 and December 2023 that quantified the effects of gamification in terms of usability, motivation, engagement, and other qualitative patient responses were selected.

RESULTS

In total, 169 studies involving 6404 patients were included. Gamified devices are categorized into four types: robotic/motorized, non-motorized, virtual reality, and neuromuscular electrical stimulation. The results showed that gamified devices not only improved motor and cognitive function but also had a significant positive impact on patients' emotional, social and personal levels. Most studies have reported high levels of patient satisfaction and motivation, highlighting the effectiveness of gamification in stroke rehabilitation.

CONCLUSIONS

Gamification in stroke rehabilitation offers significant benefits beyond motor and cognitive recovery by improving patients' emotional and social well-being. This systematic review provides a comprehensive overview of the most effective gamified technologies and highlights the need for future multidisciplinary research to optimize the design and implementation of gamified solutions in stroke rehabilitation.

Post-stroke hand gesture recognition via one-shot transfer learning using prototypical networks

BACKGROUND

In-home rehabilitation systems are a promising, potential alternative to conventional therapy for stroke survivors. Unfortunately, physiological differences between participants and sensor displacement in wearable sensors pose a significant challenge to classifier performance, particularly for people with stroke who may encounter difficulties repeatedly performing trials. This makes it challenging to create reliable in-home rehabilitation systems that can accurately classify gestures.

METHODS

Twenty individuals who suffered a stroke performed seven different gestures (mass flexion, mass extension, wrist volar flexion, wrist dorsiflexion, forearm pronation, forearm supination, and rest) related to activities of daily living. They performed these gestures while wearing EMG sensors on the forearm, as well as FMG sensors and an IMU on the wrist. We developed a model based on prototypical networks for one-shot transfer learning, K-Best feature selection, and increased window size to improve model accuracy. Our model was evaluated against conventional transfer learning with neural networks, as well as subject-dependent and subject-independent classifiers: neural networks, LGBM, LDA, and SVM.

RESULTS

Our proposed model achieved 82.2% hand-gesture classification accuracy, which was better (P<0.05) than one-shot transfer learning with neural networks (63.17%), neural networks (59.72%), LGBM (65.09%), LDA (63.35%), and SVM (54.5%). In addition, our model performed similarly to subject-dependent classifiers, slightly lower than SVM (83.84%) but higher than neural networks (81.62%), LGBM (80.79%), and LDA (74.89%). Using K-Best features improved the accuracy in 3 of the 6 classifiers used for evaluation, while not affecting the accuracy in the other classifiers. Increasing the window size improved the accuracy of all the classifiers by an average of 4.28%.

CONCLUSION

Our proposed model showed significant improvements in hand-gesture recognition accuracy in individuals who have had a stroke as compared with conventional transfer learning, neural networks and traditional machine learning approaches. In addition, K-Best feature selection and increased window size can further improve the accuracy. This approach could help to alleviate the impact of physiological differences and create a subject-independent model for stroke survivors that improves the classification accuracy of wearable sensors.

TRIAL REGISTRATION NUMBER

The study was registered in Chinese Clinical Trial Registry with registration number CHiCTR1800017568 in 2018/08/04.

Deep Learning for Electromyographic Lower-Limb Motion Signal Classification Using Residual Learning

Electromyographic (EMG) signals have gained popularity for controlling prostheses and exoskeletons, particularly in the field of upper limbs for stroke patients. However, there is a lack of research in the lower limb area, and standardized open-source datasets of lower limb EMG signals, especially recording data of Asian race features, are scarce. Additionally, deep learning algorithms are rarely used for human motion intention recognition based on EMG, especially in the lower limb area. In response to these gaps, we present an open-source benchmark dataset of lower limb EMG with Asian race characteristics and large data volume, the JJ dataset, which includes approximately 13,350 clean EMG segments of 10 gait phases from 15 people. This is the first dataset of its kind to include the nine main muscles of human gait when walking. We used the processed time-domain signal as input and adjusted ResNet-18 as the classification tool. Our research explores and compares multiple key issues in this area, including the comparison of sliding time window method and other preprocessing methods, comparison of time-domain and frequency-domain signal processing effects, cross-subject motion recognition accuracy, and the possibility of using thigh and calf muscles in amputees. Our experiments demonstrate that the adjusted ResNet can achieve significant classification accuracy, with an average accuracy rate of 95.34% for human gait phases. Our research provides a valuable resource for future studies in this area and demonstrates the potential for ResNet as a robust and effective method for lower limb human motion intention pattern recognition.

A Laguerre-Volterra network model based on ant colony optimization applied to evaluate EMG-force relationship in the muscle fatigue state

With the accuracy and convenience improvement of electromyographic (EMG) acquired by wearable devices, EMG is gradually used to evaluate muscle force signal, a non-invasive evaluation method. However, the relationship between EMG and force is a complex nonlinear relationship, even which will change with different movements and different muscle states. Therefore, it is difficult to evaluate this nonlinear EMG-force relationship, especially when the muscle state gradually transits from non-fatigue to deep fatigue. For more accurate values of force in human fatigue state, this paper proposes a dual-input Laguerre-Volterra network (LVN) model based on ant colony optimization. First, the changes in 19 EMG features are discussed with increasing fatigue. We also consider two non-Gaussian features: kurtosis and negentropy in the 19 features. Later, 11 EMG fatigue features are picked out according to the fatigue test. Then, the preprocessed EMG and a composite signal of the 11 fatigue features are simultaneously input into the LVN model. Subsequently, the ant colony optimization algorithm is selected to train the model parameters. At the same time, a penalty term that we defined is introduced into the model cost function to adjust the weight of each feature adaptively. Finally, some experiments prove that the LVN model could quick fit the accurate force signal in five fatigue stages, such as non-fatigue, slight fatigue, mild fatigue, severe fatigue, and extreme fatigue. This LVN model can quickly transform EMG into strength signal in real time, which is suitable for people to observe muscle strength by a wearable device and makes it easy to detect the muscle current state. This model has good stability and can remain effective for a long time with training once, which provides convenience for the users of wearable devices.

Cross-User Electromyography Pattern Recognition Based on a Novel Spatial-Temporal Graph Convolutional Network

With the goal of promoting the development of myoelectric control technology, this paper focuses on exploring graph neural network (GNN) based robust electromyography (EMG) pattern recognition solutions. Given that high-density surface EMG (HD-sEMG) signal contains rich temporal and spatial information, the multi-view spatial-temporal graph convolutional network (MSTGCN)is adopted as the basic classifier, and a feature extraction convolutional neural network (CNN) module is designed and integrated into MSTGCN to generate a new model called CNN-MSTGCN. The EMG pattern recognition experiments are conducted on HD-sEMG data of 17 gestures from 11 subjects. The ablation experiments show that each functional module of the proposed CNN-MSTGCN network has played a more or less positive role in improving the performance of EMG pattern recognition. The user-independent recognition experiments and the transfer learning-based cross-user recognition experiments verify the advantages of the proposed CNN-MSTGCN network in improving recognition rate and reducing user training burden. In the user-independent recognition experiments, CNN-MSTGCN achieves the recognition rate of 68%, which is significantly better than those obtained by residual network-50 (ResNet50, 47.5%, p < 0.001) and long-short-term-memory (LSTM, 57.1%, p=0.045). In the transfer learning-based cross-user recognition experiments, TL-CMSTGCN achieves an impressive recognition rate of 92.3%, which is significantly superior to both TL-ResNet50 (84.6%, p = 0.003) and TL-LSTM (85.3%, p = 0.008). The research results of this paper indicate that GNN has certain advantages in overcoming the impact of individual differences, and can be used to provide possible solutions for achieving robust EMG pattern recognition technology.

Effects of acupuncture synchronized rehabilitation therapy on upper limb motor and sensory function after stroke: a study protocol for a single-center, 2 × 2 factorial design, randomized controlled trial

Background

Upper limb function reconstruction has been an important issue in the field of stroke rehabilitation. Due to the complexity of upper extremity dysfunction in stroke patients, the clinical efficacy produced by central or peripheral stimulation alone is limited. For this reason, our group has proposed acupuncture synchronized rehabilitation therapy (ASRT), i.e., simultaneous scalp acupuncture and intradermal acupuncture during rehabilitation. Pre-experiments results showed that this therapy can effectively improve the motor and sensory functions of upper limbs in post-stroke patients, but the clinical efficacy and safety of ASRT need to be further verified, and whether there is a synergistic effect between scalp acupuncture and intradermal acupuncture also needs to be studied in depth. Therefore, we designed a randomized controlled trial to compare the efficacy and safety of different therapies to explore a more scientific "synchronous treatment model."

Methods

This is a single-center, randomized controlled trial using a 2 × 2 factorial design. We will recruit 136 stroke survivors with upper extremity dysfunction and randomize them into four groups (n = 34). All subjects will undergo routine treatment, based on which the Experimental Group 1: rehabilitation training synchronized with intradermal acupuncture treatment of the affected upper limb; Experimental Group 2: rehabilitation training of the affected upper limb synchronized with focal-side scalp acupuncture treatment, and Experimental Group 3: rehabilitation training synchronized with intradermal acupuncture treatment of the affected upper limb synchronized with focal-side scalp acupuncture treatment; Control Group: rehabilitation training of the affected upper limb only. The intervention will last for 4 weeks, 5 times a week. Both acupuncture treatments will be performed according to the Revised Standards for Reporting Interventions in Clinical Trials of Acupuncture (STRICTA). The primary outcome indicators for this trial are Fugl-Meyer Assessment-Upper Extremity and Somatosensory Evoked Potential. Secondary outcome indicators include Wolf Motor Function Test, Upper Extremity Function Test, revised Nottingham Sensory Assessment Scale, Diffusion Tensor Imaging, and Modified Barthel Index. The incidence of adverse events will be used as the indicator of safety.

Discussion

The study will provide high-quality clinical evidence on whether ASRT improves upper limb motor and sensory function and activities of daily living (ADL) in stroke patients, and determine whether scalp acupuncture and intradermal acupuncture have synergistic effects.

Clinical trial registration

https://www.chictr.org.cn/, Chinese Clinical Trial Registry [ChiCTR2200066646].

A Novel Model to Generate Heterogeneous and Realistic Time-Series Data for Post-Stroke Rehabilitation Assessment

The application of machine learning-based tele-rehabilitation faces the challenge of limited availability of data. To overcome this challenge, data augmentation techniques are commonly employed to generate synthetic data that reflect the configurations of real data. One such promising data augmentation technique is the Generative Adversarial Network (GAN). However, GANs have been found to suffer from mode collapse, a common issue where the generated data fails to capture all the relevant information from the original dataset. In this paper, we aim to address the problem of mode collapse in GAN-based data augmentation techniques for post-stroke assessment. We applied the GAN to generate synthetic data for two post-stroke rehabilitation datasets and observed that the original GAN suffered from mode collapse, as expected. To address this issue, we propose a Time Series Siamese GAN (TS-SGAN) that incorporates a Siamese network and an additional discriminator. Our analysis, using the longest common sub-sequence (LCSS), demonstrates that TS-SGAN generates data uniformly for all elements of two testing datasets, in contrast to the original GAN. To further evaluate the effectiveness of TS-SGAN, we encode the generated dataset into images using Gramian Angular Field and classify them using ResNet-18. Our results show that TS-SGAN achieves a significant accuracy increase of classification accuracy (35.2%-42.07%) for both selected datasets. This represents a substantial improvement over the original GAN.

Commercial device-based hand rehabilitation systems for stroke patients: State of the art and future prospects

Various hand rehabilitation systems have recently been developed for stroke patients, particularly commercial devices. Articles from 10 electronic databases from 2010 to 2022 were extracted to conduct a systematic review to explore the existing commercial training systems (hardware and software) and evaluate their clinical effectiveness. This review divided the rehabilitation equipment into contact and non-contact types. Game-based training protocols were further classified into two types: immersion and non-immersion. The results of the review indicated that the majority of the devices included were effective in improving hand function. Users who underwent rehabilitation training with these devices reported improvements in their hand function. Game-based training protocols were particularly appealing as they helped reduce boredom during rehabilitation training sessions. However, the review also identified some common technical drawbacks in the devices, particularly in non-contact devices, such as their vulnerability to the effects of light. Additionally, it was found that currently, there is no commercially available game-based training protocol that specifically targets hand rehabilitation. Given the ongoing COVID-19 pandemic, there is a need to develop safer non-contact rehabilitation equipment and more engaging training protocols for community and home-based rehabilitation. Additionally, the review suggests the need for revisions or the development of new clinical scales for hand rehabilitation evaluation that consider the current scenario, where in-person interactions might be limited.

Deep Reinforcement Learning-Based iTrain Serious Game for Caregivers Dealing with Post-Stroke Patients

This paper describes a serious game based on a knowledge transfer model using deep reinforcement learning, with an aim to improve the caretakers’ knowledge and abilities in post-stroke care. The iTrain game was designed to improve caregiver knowledge and abilities by providing non-traditional training to formal and informal caregivers who deal with stroke survivors. The methodologies utilized professional medical experiences and real-life evidence data gathered during the duration of the iTrain project to create the scenarios for the game’s deep reinforcement caregiver behavior improvement model, as well as the design of game mechanics, game images and game characters, and gameplay implementation. Furthermore, the results of the game’s direct impact on caregivers (n = 25) and stroke survivors (n = 21) in Lithuania using the Geriatric Depression Scale (GDS) and user experience questionnaire (UEQ) are presented. Both surveys had favorable outcomes, showing the effectiveness of the approach. The GDS scale (score 10) revealed a low number of 28% of individuals depressed, and the UEQ received a very favorable grade of +0.8.

Automatic Upper-Limb Brunnstrom Recovery Stage Evaluation via Daily Activity Monitoring

Motor function assessment is crucial for post-stroke rehabilitation. Conventional evaluation methods are subjective, heavily depending on the experience of therapists. In light of the strong correlation between the stroke severity level and the performance of activities of daily living (ADLs), we explored the possibility of automatically evaluating the upper-limb Brunnstrom Recovery Stage (BRS) via three typical ADLs (tooth brushing, face washing and drinking). Multimodal data (acceleration, angular velocity, surface electromyography) were synchronously collected from 5 upper-limb-worn sensor modules. The performance of BRS evaluation system is known to be variable with different system parameters (e.g., number of sensor modules, feature types and classifiers). We systematically searched for the optimal parameters from different data segmentation strategies (five window lengths and four overlaps), 42 types of features, 12 feature optimization techniques and 9 classifiers with the leave-one-subject-out cross-validation. To achieve reliable and low-cost monitoring, we further explored whether it was possible to obtain a satisfactory result using a relatively small number of sensor modules. As a result, the proposed approach can correctly recognize the stages of all 27 participants using only three sensor modules with the optimized data segmentation parameters (window length: 7s, overlap: 50%), extracted features (simple square integral, slope sign change, modified mean absolute value 1 and modified mean absolute value 2), the feature optimization method (principal component analysis) and the logistic regression classifier. According to the literature, this is the first study to comprehensively optimize sensor configuration and parameters in each stage of the BRS classification framework. The proposed approach can serve as a factor-screening tool towards the automatic BRS classification and is promising to be further used at home.

[Methods to restore fine motor skills in stroke patients]

The review article considers the problem of nonmedical post-stroke rehabilitation, in particular the restoration of fine motor skills in patients in the early period of the disease. A review and analysis of various randomized controlled trials concerning the use of various rehabilitation methods both in monotherapy and in their combined application is carried out, and modern technical devices, with the use of computer technology and biofeedback, are reviewed. Proceeding from the presented literature data and their analysis, there are certain grounds for introducing modern apparatus complexes and robotized devices for fine motor skills restoration in post-stroke patients, especially in the early period, into the multimodal rehabilitation system. However, further research in this direction is needed to achieve a sustained positive result.