Fusion of Human Gaze and Machine Vision for Predicting Intended Locomotion Mode

IPTGNet: an adaptive multi-task recognition strategy for human locomotion modes

Complexities in processing human motion are possessed by lower limb exoskeletons. In this paper, a multi-task recognition model, IPTGNet, is proposed for the human locomotion modes. Temporal convolutional network and gated recurrent unit are parallelly fused through the dynamic tuning of hyperparameters using the improved particle swarm optimization algorithm. The experimental results demonstrate that faster and more stable convergence is achieved by IPTGNet with a recognition rate of 99.47% and a standard deviation of 0.42%. Furthermore, a finite state machine is utilized for incorrection of transition states. An innovative multi-task recognition of lower limb exoskeleton is provided by this paper.

Real-Time Continuous Locomotion Mode Recognition and Transition Prediction for Human With Lower Limb Exoskeleton

Real-time continuous locomotion mode recognition and seamless timely transition detection is critical for the exoskeleton robot. This study aims to present a comprehensive and innovative framework for locomotion mode recognition and transition prediction, exclusively utilizing inertial measurement unit (IMU) signals from the exoskeleton. In this framework, a CNN-BiLSTM model was developed and trained to be the classifier and a novel majority filter was designed to reduce the transition misjudgment rate. Moreover, a comprehensive evaluation system encompassing eight dimensions for the classifier, incorporating evaluation metrics specifically for transition misjudgment, was proposed. We collected locomotion motion data from six subjects wearing a rigid exoskeleton robot using six IMU sensors on the exoskeleton. The proposed method achieves a high level of recognition accuracy, with an overall average of 99.58 for the five steady locomotion modes (level ground walking (LG), stair ascent/descent (SA/SD), and ramp ascent/descent (RA/RD)) across six subjects following the transition decision. All transitions are recognizable, and the majority can be predicted in advance, with an average prediction time of 353 ms. Furthermore, the implementation of majority filter resulted in an average 87.04 reduction in the transition misjudgment rate among six subjects, thereby decreasing the average transition misjudgment rate to 4.82. Finally, the model was tested on a Jetson Nano to verify its real-time performance. The results presented above were obtained under the condition where either leg could function as the first transition leg and revealed that the developed system was capable of achieving precise locomotion mode recognition and timely transition prediction, with high real-time performance.

Continuous Locomotion Mode and Task Identification for an Assistive Exoskeleton Based on Neuromuscular-Mechanical Fusion

Human walking parameters exhibit significant variability depending on the terrain, speed, and load. Assistive exoskeletons currently focus on the recognition of locomotion terrain, ignoring the identification of locomotion tasks, which are also essential for control strategies. The aim of this study was to develop an interface for locomotion mode and task identification based on a neuromuscular-mechanical fusion algorithm. The modes of level and incline and tasks of speed and load were explored, and seven able-bodied participants were recruited. A continuous stream of assistive decisions supporting timely exoskeleton control was achieved according to the classification of locomotion. We investigated the optimal algorithm, feature set, window increment, window length, and robustness for precise identification and synchronization between exoskeleton assistive force and human limb movements (human-machine collaboration). The best recognition results were obtained when using a support vector machine, a root mean square/waveform length/acceleration feature set, a window length of 170, and a window increment of 20. The average identification accuracy reached 98.7% ± 1.3%. These results suggest that the surface electromyography-acceleration can be effectively used for locomotion mode and task identification. This study contributes to the development of locomotion mode and task recognition as well as exoskeleton control for seamless transitions.

Continuous A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Kinematics Across Different Ambulation Tasks

OBJECTIVE

Volitional control systems for powered prostheses require the detection of user intent to operate in real life scenarios. Ambulation mode classification has been proposed to address this issue. However, these approaches introduce discrete labels to the otherwise continuous task that is ambulation. An alternative approach is to provide users with direct, voluntary control of the powered prosthesis motion. Surface electromyography (EMG) sensors have been proposed for this task, but poor signal-to-noise ratios and crosstalk from neighboring muscles limit performance. B-mode ultrasound can address some of these issues at the cost of reduced clinical viability due to the substantial increase in size, weight, and cost. Thus, there is an unmet need for a lightweight, portable neural system that can effectively detect the movement intention of individuals with lower-limb amputation.

METHODS

In this study, we show that a small and lightweight A-mode ultrasound system can continuously predict prosthesis joint kinematics in seven individuals with transfemoral amputation across different ambulation tasks. Features from the A-mode ultrasound signals were mapped to the user's prosthesis kinematics via an artificial neural network.

RESULTS

Predictions on testing ambulation circuit trials resulted in a mean normalized RMSE across different ambulation modes of 8.7 ± 3.1%, 4.6 ± 2.5%, 7.2 ± 1.8%, and 4.6 ± 2.4% for knee position, knee velocity, ankle position, and ankle velocity, respectively.

CONCLUSION AND SIGNIFICANCE

This study lays the foundation for future applications of A-mode ultrasound for volitional control of powered prostheses during a variety of daily ambulation tasks.

Wearable sensing for understanding and influencing human movement in ecological contexts

Wearable sensors offer a unique opportunity to study movement in ecological contexts - that is, outside the laboratory where movement happens in ordinary life. This article discusses the purpose, means, and impact of using wearable sensors to assess movement context, kinematics, and kinetics during locomotion, and how this information can be used to better understand and influence movement. We outline the types of information wearable sensors can gather and highlight recent developments in sensor technology, data analysis, and applications. We close with a vision for important future research and key questions the field will need to address to bring the potential benefits of wearable sensing to fruition.

A Learning-Free Method for Locomotion Mode Prediction by Terrain Reconstruction and Visual-Inertial Odometry

This research introduces a novel, highly precise, and learning-free approach to locomotion mode prediction, a technique with potential for broad applications in the field of lower-limb wearable robotics. This study represents the pioneering effort to amalgamate 3D reconstruction and Visual-Inertial Odometry (VIO) into a locomotion mode prediction method, which yields robust prediction performance across diverse subjects and terrains, and resilience against various factors including camera view, walking direction, step size, and disturbances from moving obstacles without the need of parameter adjustments. The proposed Depth-enhanced Visual-Inertial Odometry (D-VIO) has been meticulously designed to operate within computational constraints of wearable configurations while demonstrating resilience against unpredictable human movements and sparse features. Evidence of its effectiveness, both in terms of accuracy and operational time consumption, is substantiated through tests conducted using open-source dataset and closed-loop evaluations. Comprehensive experiments were undertaken to validate its prediction accuracy across various test conditions such as subjects, scenarios, sensor mounting positions, camera views, step sizes, walking directions, and disturbances from moving obstacles. A comprehensive prediction accuracy rate of 99.00% confirms the efficacy, generality, and robustness of the proposed method.

A Method of Detecting Human Movement Intentions in Real Environments

Accurate and timely movement intention detection can facilitate exoskeleton control during transitions between different locomotion modes. Detecting movement intentions in real environments remains a challenge due to unavoidable environmental uncertainties. False movement intention detection may also induce risks of falling and general danger for exoskeleton users. To this end, in this study, we developed a method for detecting human movement intentions in real environments. The proposed method is capable of online self-correcting by implementing a decision fusion layer. Gaze data from an eye tracker and inertial measurement unit (IMU) signals were fused at the feature extraction level and used to predict movement intentions using 2 different methods. Images from the scene camera embedded on the eye tracker were used to identify terrains using a convolutional neural network. The decision fusion was made based on the predicted movement intentions and identified terrains. Four able-bodied participants wearing the eye tracker and 7 IMU sensors took part in the experiments to complete the tasks of level ground walking, ramp ascending, ramp descending, stairs ascending, and stair descending. The recorded experimental data were used to test the feasibility of the proposed method. An overall accuracy of 93.4% was achieved when both feature fusion and decision fusion were used. Fusing gaze data with IMU signals improved the prediction accuracy.

Data-Driven Variable Impedance Control of a Powered Knee-Ankle Prosthesis for Adaptive Speed and Incline Walking

Most impedance-based walking controllers for powered knee-ankle prostheses use a finite state machine with dozens of user-specific parameters that require manual tuning by technical experts. These parameters are only appropriate near the task (e.g., walking speed and incline) at which they were tuned, necessitating many different parameter sets for variable-task walking. In contrast, this paper presents a data-driven, phase-based controller for variable-task walking that uses continuously-variable impedance control during stance and kinematic control during swing to enable biomimetic locomotion. After generating a data-driven model of variable joint impedance with convex optimization, we implement a novel task-invariant phase variable and real-time estimates of speed and incline to enable autonomous task adaptation. Experiments with above-knee amputee participants (N=2) show that our data-driven controller 1) features highly-linear phase estimates and accurate task estimates, 2) produces biomimetic kinematic and kinetic trends as task varies, leading to low errors relative to able-bodied references, and 3) produces biomimetic joint work and cadence trends as task varies. We show that the presented controller meets and often exceeds the performance of a benchmark finite state machine controller for our two participants, without requiring manual impedance tuning.

A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Walking Kinematics via an Artificial Neural Network

Lower-limb powered prostheses can provide users with volitional control of ambulation. To accomplish this goal, they require a sensing modality that reliably interprets user intention to move. Surface electromyography (EMG) has been previously proposed to measure muscle excitation and provide volitional control to upper- and lower-limb powered prosthesis users. Unfortunately, EMG suffers from a low signal to noise ratio and crosstalk between neighboring muscles, often limiting the performance of EMG-based controllers. Ultrasound has been shown to have better resolution and specificity than surface EMG. However, this technology has yet to be integrated into lower-limb prostheses. Here we show that A-mode ultrasound sensing can reliably predict the prosthesis walking kinematics of individuals with a transfemoral amputation. Ultrasound features from the residual limb of 9 transfemoral amputee subjects were recorded with A-mode ultrasound during walking with their passive prosthesis. The ultrasound features were mapped to joint kinematics through a regression neural network. Testing of the trained model against untrained kinematics show accurate predictions of knee position, knee velocity, ankle position, and ankle velocity, with a normalized RMSE of 9.0 ± 3.1%, 7.3 ± 1.6%, 8.3 ± 2.3%, and 10.0 ± 2.5% respectively. This ultrasound-based prediction suggests that A-mode ultrasound is a viable sensing technology for recognizing user intent. This study is the first necessary step towards implementation of volitional prosthesis controller based on A-mode ultrasound for individuals with transfemoral amputation.

Exploring the Utility of Crutch Force Sensors to Predict User Intent in Assistive Lower Limb Exoskeletons

The adoption of assistive lower limb exoskeletons in built environments is reliant on the further development of these devices to handle the varied conditions experienced in everyday life. The required development includes more varied and flexible gait patterns, but also appropriate user interfaces to enable fluid gait. This work explores the properties of an algorithm used to predict user intent based on sensors onboard a user-balanced robotic exoskeleton system. Specifically, classification algorithms built with different input data sets are compared - with varying detail of the interaction forces between the crutches and the ground, and the duration of the data sample used to make the prediction. Data were collected with one able-bodied participant using an exoskeleton, training three independent classifiers corresponding to different exoskeleton states. The results indicate the value of including information about the interaction forces between the crutches and the ground in improving prediction accuracy, with increasing prediction window also generally resulting in an increase in prediction accuracy. Whilst no categorical recommendation can be made with respect to either parameter, these results provide a baseline which can be used in conjunction deliberate consideration of the costs associated with implementation.