Ultrasound Sensing Can Improve Continuous Classification of Discrete Ambulation Modes Compared to Surface Electromyography

AnkleImage - An ultrafast ultrasound image dataset to understand the ankle joint muscle contractility

The role of the human ankle joint in activities of daily living, including walking, maintaining balance, and participating in sports, is of paramount importance. Ankle joint dorsiflexion and plantarflexion functionalities mainly account for ground clearance and propulsion power generation during locomotion tasks, where those functionalities are driven by the contraction of ankle joint skeleton muscles. Studies of corresponding muscle contractility during ankle dynamic functions will facilitate us to better understand the joint torque/power generation mechanism, better diagnose potential muscular disorders on the ankle joint, or better develop wearable assistive/rehabilitative robotic devices that assist in community ambulation. This data descriptor reports a new dataset that includes the ankle joint kinematics/kinetics, associated muscle surface electromyography, and ultrafast ultrasound images with various annotations, such as pennation angle, fascicle length, tissue displacements, echogenicity, and muscle thickness, of ten healthy participants when performing volitional isometric, isokinetic, and dynamic ankle joint functions (walking at multiple treadmill speeds, including 0.50 m/s, 0.75 m/s, 1.00 m/s, 1.25 m/s, and 1.50 m/s). Data were recorded by a research-use ultrasound machine, a self-designed ankle testbed, an inertia measurement unit system, a Vicon motion capture system, a surface electromyography system, and an instrumented treadmill. The descriptor in this work presents the results of a data curation or collection exercise from previous works, rather than describing a novel primary/experimental data collection.

Wearable Surface Deformation Myography (sDMG) System for Recognition of Locomotion Modes

This study designs a wearable sensing system for locomotion mode recognition using lower-limb skin surface curvature deformation caused by the morphological changes of musculotendinous complexes and soft tissues. Flexible bending sensors are embedded into stretch pants, enabling curvature deformations of specific skin segments above lower-limb muscle groups to be captured in a noncontact manner. To evaluate the performance of this system, we conducted experiments on eight able-bodied subjects completing seven common locomotive activities, including walking, running, ramp ascending/descending, stair ascending/descending, and standing. The system measured seven channels of deformation signals from two cross-sections on the shank and the thigh. The collected signals were distinguishable across different locomotion modes and exhibited consistency when monitoring steps. Using selected time-domain features and a linear discriminant analysis (LDA) classifier enabled the proposed system to continuously recognize locomotion modes with an average accuracy of 96.5%. Furthermore, the system maintains recognition performance with 95.7% accuracy even after removing and reapplying the sensors. Finally, we conducted comparison experiments to analyze how window length, feature selection, and the number of channels affect recognition performance, providing insights for optimization. We believe that this novel signal platform holds great potential as a valuable supplementary tool in wearable human motion detection, enriching the information diversity for motion analysis, and enabling new possibilities for further advancements and applications in fields including biomedical engineering, textiles, and computer graphics.

Visual Illusion Created by a Striped Pattern Through Augmented Reality for the Prevention of Tumbling on Stairs

A fall on stairs can be a dangerous accident. An important indicator of falling risk is the foot clearance, which is the height of the foot when ascending stairs or the distance of the foot from the step when descending. We developed an augmented reality system with a holographic lens using a visual illusion to improve the foot clearance on stairs. The system draws a vertical striped pattern on the stair riser as the participant ascends the stairs to create the illusion that the steps are higher than the actual steps, and draws a horizontal striped pattern on the stair tread as the participant descends the stairs to create the illusion of narrower stairs. We experimentally evaluated the accuracy of the system and fitted a model to determine the appropriate stripe thickness. Finally, participants ascended and descended stairs before, during, and after using the augmented reality system. The foot clearance significantly improved, not only while the participants used the system but also after they used the system compared with before.

Loading Recognition for Lumbar Exoskeleton Based on Multi-Channel Surface Electromyography From Low Back Muscles

Lumbar exoskeleton is an assistive robot, which can reduce the risk of injury and pain in low back muscles when lifting heavy objects. An important challenge it faces involves enhancing assistance with minimal muscle energy consumption. One of the viable solutions is to adjust the force or torque of assistance in response to changes in the load on the low back muscles. It requires accurate loading recognition, which has yet to yield satisfactory outcomes due to the limitations of available measurement tools and load classification methods. This study aimed to precisely identify muscle loading using a multi-channel surface electromyographic (sEMG) electrode array on the low back muscles, combined with a participant-specific load classification method. Ten healthy participants performed a stoop lifting task with objects of varying weights, while sEMG data was collected from the low back muscles using a 3x7 electrode array. Nineteen time segments of the lifting phase were identified, and time-domain sEMG features were extracted from each segment. Participant-specific classifiers were built using four classification algorithms to determine the object weight in each time segment, and the classification performance was evaluated using a 5-fold cross-validation method. The artificial neural network classifier achieved an impressive accuracy of up to 96%, consistently improving as the lifting phase progressed, peaking towards the end of the lifting movement. This study successfully achieves accurate recognition of load on low back muscles during the object lifting task. The obtained results hold significant potential in effectively reducing muscle energy consumption when wearing a lumbar exoskeleton.

Continuous A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Kinematics Across Different Ambulation Tasks

OBJECTIVE

Volitional control systems for powered prostheses require the detection of user intent to operate in real life scenarios. Ambulation mode classification has been proposed to address this issue. However, these approaches introduce discrete labels to the otherwise continuous task that is ambulation. An alternative approach is to provide users with direct, voluntary control of the powered prosthesis motion. Surface electromyography (EMG) sensors have been proposed for this task, but poor signal-to-noise ratios and crosstalk from neighboring muscles limit performance. B-mode ultrasound can address some of these issues at the cost of reduced clinical viability due to the substantial increase in size, weight, and cost. Thus, there is an unmet need for a lightweight, portable neural system that can effectively detect the movement intention of individuals with lower-limb amputation.

METHODS

In this study, we show that a small and lightweight A-mode ultrasound system can continuously predict prosthesis joint kinematics in seven individuals with transfemoral amputation across different ambulation tasks. Features from the A-mode ultrasound signals were mapped to the user's prosthesis kinematics via an artificial neural network.

RESULTS

Predictions on testing ambulation circuit trials resulted in a mean normalized RMSE across different ambulation modes of 8.7 ± 3.1%, 4.6 ± 2.5%, 7.2 ± 1.8%, and 4.6 ± 2.4% for knee position, knee velocity, ankle position, and ankle velocity, respectively.

CONCLUSION AND SIGNIFICANCE

This study lays the foundation for future applications of A-mode ultrasound for volitional control of powered prostheses during a variety of daily ambulation tasks.

A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Walking Kinematics via an Artificial Neural Network

Lower-limb powered prostheses can provide users with volitional control of ambulation. To accomplish this goal, they require a sensing modality that reliably interprets user intention to move. Surface electromyography (EMG) has been previously proposed to measure muscle excitation and provide volitional control to upper- and lower-limb powered prosthesis users. Unfortunately, EMG suffers from a low signal to noise ratio and crosstalk between neighboring muscles, often limiting the performance of EMG-based controllers. Ultrasound has been shown to have better resolution and specificity than surface EMG. However, this technology has yet to be integrated into lower-limb prostheses. Here we show that A-mode ultrasound sensing can reliably predict the prosthesis walking kinematics of individuals with a transfemoral amputation. Ultrasound features from the residual limb of 9 transfemoral amputee subjects were recorded with A-mode ultrasound during walking with their passive prosthesis. The ultrasound features were mapped to joint kinematics through a regression neural network. Testing of the trained model against untrained kinematics show accurate predictions of knee position, knee velocity, ankle position, and ankle velocity, with a normalized RMSE of 9.0 ± 3.1%, 7.3 ± 1.6%, 8.3 ± 2.3%, and 10.0 ± 2.5% respectively. This ultrasound-based prediction suggests that A-mode ultrasound is a viable sensing technology for recognizing user intent. This study is the first necessary step towards implementation of volitional prosthesis controller based on A-mode ultrasound for individuals with transfemoral amputation.

Ambulation Mode Classification of Individuals with Transfemoral Amputation through A-Mode Sonomyography and Convolutional Neural Networks

Many people struggle with mobility impairments due to lower limb amputations. To participate in society, they need to be able to walk on a wide variety of terrains, such as stairs, ramps, and level ground. Current lower limb powered prostheses require different control strategies for varying ambulation modes, and use data from mechanical sensors within the prosthesis to determine which ambulation mode the user is in. However, it can be challenging to distinguish between ambulation modes. Efforts have been made to improve classification accuracy by adding electromyography information, but this requires a large number of sensors, has a low signal-to-noise ratio, and cannot distinguish between superficial and deep muscle activations. An alternative sensing modality, A-mode ultrasound, can detect and distinguish between changes in superficial and deep muscles. It has also shown promising results in upper limb gesture classification. Despite these advantages, A-mode ultrasound has yet to be employed for lower limb activity classification. Here we show that A- mode ultrasound can classify ambulation mode with comparable, and in some cases, superior accuracy to mechanical sensing. In this study, seven transfemoral amputee subjects walked on an ambulation circuit while wearing A-mode ultrasound transducers, IMU sensors, and their passive prosthesis. The circuit consisted of sitting, standing, level-ground walking, ramp ascent, ramp descent, stair ascent, and stair descent, and a spatial-temporal convolutional network was trained to continuously classify these seven activities. Offline continuous classification with A-mode ultrasound alone was able to achieve an accuracy of 91.8±3.4%, compared with 93.8±3.0%, when using kinematic data alone. Combined kinematic and ultrasound produced 95.8±2.3% accuracy. This suggests that A-mode ultrasound provides additional useful information about the user's gait beyond what is provided by mechanical sensors, and that it may be able to improve ambulation mode classification. By incorporating these sensors into powered prostheses, users may enjoy higher reliability for their prostheses, and more seamless transitions between ambulation modes.

Multi-Sensing Techniques with Ultrasound for Musculoskeletal Assessment: A Review

The study of muscle contractions generated by the muscle-tendon unit (MTU) plays a critical role in medical diagnoses, monitoring, rehabilitation, and functional assessments, including the potential for movement prediction modeling used for prosthetic control. Over the last decade, the use of combined traditional techniques to quantify information about the muscle condition that is correlated to neuromuscular electrical activation and the generation of muscle force and vibration has grown. The purpose of this review is to guide the reader to relevant works in different applications of ultrasound imaging in combination with other techniques for the characterization of biological signals. Several research groups have been using multi-sensing systems to carry out specific studies in the health area. We can divide these studies into two categories: human-machine interface (HMI), in which sensors are used to capture critical information to control computerized prostheses and/or robotic actuators, and physiological study, where sensors are used to investigate a hypothesis and/or a clinical diagnosis. In addition, the relevance, challenges, and expectations for future work are discussed.

Seamless and intuitive control of a powered prosthetic leg using deep neural network for transfemoral amputees

Powered prosthetic legs are becoming a promising option for amputee patients. However, developing safe, robust, and intuitive control strategies for powered legs remains one of the greatest challenges. Although a variety of control strategies have been proposed, creating and fine-tuning the system parameters is time-intensive and complicated when more activities need to be restored. In this study, we developed a deep neural network (DNN) model that facilitates seamless and intuitive gait generation and transitions across five ambulation modes: level-ground walking, ascending/descending ramps, and ascending/descending stairs. The combination of latent and time sequence features generated the desired impedance parameters within the ambulation modes and allowed seamless transitions between ambulation modes. The model was applied to the open-source bionic leg and tested on unilateral transfemoral users. It achieved the overall coefficient of determination of 0.72 with the state machine-based impedance parameters in the offline testing session. In addition, users were able to perform in-laboratory ambulation modes with an overall success rate of 96% during the online testing session. The results indicate that the DNN model is a promising candidate for subject-independent and tuning-free prosthetic leg control for transfemoral amputees.

A deep learning method to predict ankle joint moment during walking at different speeds with ultrasound imaging: A framework for assistive devices control

Robotic assistive or rehabilitative devices are promising aids for people with neurological disorders as they help regain normative functions for both upper and lower limbs. However, it remains challenging to accurately estimate human intent or residual efforts non-invasively when using these robotic devices. In this article, we propose a deep learning approach that uses a brightness mode, that is, B-mode, of ultrasound (US) imaging from skeletal muscles to predict the ankle joint net plantarflexion moment while walking. The designed structure of customized deep convolutional neural networks (CNNs) guarantees the convergence and robustness of the deep learning approach. We investigated the influence of the US imaging's region of interest (ROI) on the net plantarflexion moment prediction performance. We also compared the CNN-based moment prediction performance utilizing B-mode US and sEMG spectrum imaging with the same ROI size. Experimental results from eight young participants walking on a treadmill at multiple speeds verified an improved accuracy by using the proposed US imaging + deep learning approach for net joint moment prediction. With the same CNN structure, compared to the prediction performance by using sEMG spectrum imaging, US imaging significantly reduced the normalized prediction root mean square error by 37.55% ( < .001) and increased the prediction coefficient of determination by 20.13% ( < .001). The findings show that the US imaging + deep learning approach personalizes the assessment of human joint voluntary effort, which can be incorporated with assistive or rehabilitative devices to improve clinical performance based on the assist-as-needed control strategy.

Evaluating Electromyography and Sonomyography Sensor Fusion to Estimate Lower-Limb Kinematics Using Gaussian Process Regression

Research on robotic lower-limb assistive devices over the past decade has generated autonomous, multiple degree-of-freedom devices to augment human performance during a variety of scenarios. However, the increase in capabilities of these devices is met with an increase in the complexity of the overall control problem and requirement for an accurate and robust sensing modality for intent recognition. Due to its ability to precede changes in motion, surface electromyography (EMG) is widely studied as a peripheral sensing modality for capturing features of muscle activity as an input for control of powered assistive devices. In order to capture features that contribute to muscle contraction and joint motion beyond muscle activity of superficial muscles, researchers have introduced sonomyography, or real-time dynamic ultrasound imaging of skeletal muscle. However, the ability of these sonomyography features to continuously predict multiple lower-limb joint kinematics during widely varying ambulation tasks, and their potential as an input for powered multiple degree-of-freedom lower-limb assistive devices is unknown. The objective of this research is to evaluate surface EMG and sonomyography, as well as the fusion of features from both sensing modalities, as inputs to Gaussian process regression models for the continuous estimation of hip, knee and ankle angle and velocity during level walking, stair ascent/descent and ramp ascent/descent ambulation. Gaussian process regression is a Bayesian nonlinear regression model that has been introduced as an alternative to musculoskeletal model-based techniques. In this study, time-intensity features of sonomyography on both the anterior and posterior thigh along with time-domain features of surface EMG from eight muscles on the lower-limb were used to train and test subject-dependent and task-invariant Gaussian process regression models for the continuous estimation of hip, knee and ankle motion. Overall, anterior sonomyography sensor fusion with surface EMG significantly improved estimation of hip, knee and ankle motion for all ambulation tasks (level ground, stair and ramp ambulation) in comparison to surface EMG alone. Additionally, anterior sonomyography alone significantly improved errors at the hip and knee for most tasks compared to surface EMG. These findings help inform the implementation and integration of volitional control strategies for robotic assistive technologies.

Performance of Sonomyographic and Electromyographic Sensing for Continuous Estimation of Joint Torque During Ambulation on Multiple Terrains

Advances in powered assistive device technology, including the ability to provide net mechanical power to multiple joints within a single device, have the potential to dramatically improve the mobility and restore independence to their users. However, these devices rely on the ability of their users to continuously control multiple powered lower-limb joints simultaneously. Success of such approaches rely on robust sensing of user intent and accurate mapping to device control parameters. Here, we compare two non-invasive sensing modalities: surface electromyography and sonomyography, (i.e., ultrasound imaging of skeletal muscle), as inputs to Gaussian process regression models trained to estimate hip, knee and ankle joint moments during varying forms of ambulation. Experiments were performed with ten non-disabled individuals instrumented with surface electromyography and sonomyography sensors while completing trials of level, incline (10°) and decline (10°) walking. Results suggest sonomyography of muscles on the anterior and posterior thigh can be used to estimate hip, knee and ankle joint moments more accurately than surface electromyography. Furthermore, these results can be achieved by training Gaussian process regression models in a task-independent manner; i.e., incorporating features of level and ramp walking within the same predictive framework. These findings support the integration of sonomyographic and electromyographic sensing within powered assistive devices to continuously control joint torque.

Muscle as a tunable material: implications for achieving muscle-like function in robotic prosthetic devices

An ideal prosthesis should perform as well as or better than the missing limb it was designed to replace. Although this ideal is currently unattainable, recent advances in design have significantly improved the function of prosthetic devices. For the lower extremity, both passive prostheses (which provide no added power) and active prostheses (which add propulsive power) aim to emulate the dynamic function of the ankle joint, whose adaptive, time-varying resistance to applied forces is essential for walking and running. Passive prostheses fail to normalize energetics because they lack variable ankle impedance that is actively controlled within each gait cycle. By contrast, robotic prostheses can normalize energetics for some users under some conditions. However, the problem of adaptive and versatile control remains a significant issue. Current prosthesis-control algorithms fail to adapt to changes in gait required for walking on level ground at different speeds or on ramps and stairs. A new paradigm of 'muscle as a tunable material' versus 'muscle as a motor' offers insights into the adaptability and versatility of biological muscles, which may provide inspiration for prosthesis design and control. In this new paradigm, neural activation tunes muscle stiffness and damping, adapting the response to applied forces rather than instructing the timing and amplitude of muscle force. A mechanistic understanding of muscle function is incomplete and would benefit from collaboration between biologists and engineers. An improved understanding of the adaptability of muscle may yield better models as well as inspiration for developing prostheses that equal or surpass the functional capabilities of biological limbs across a wide range of conditions.