Prediction of gait trajectories based on the Long Short Term Memory neural networks

Forecasting motion trajectories of elbow and knee joints during infant crawling based on long-short-term memory (LSTM) networks

BACKGROUND

Hands-and-knees crawling is a promising rehabilitation intervention for infants with motor impairments, while research on assistive crawling devices for rehabilitation training was still in its early stages. In particular, precisely generating motion trajectories is a prerequisite to controlling exoskeleton assistive devices, and deep learning-based prediction algorithms, such as Long-Short-Term Memory (LSTM) networks, have proven effective in forecasting joint trajectories of gait. Despite this, no previous studies have focused on forecasting the more variable and complex trajectories of infant crawling. Therefore, this paper aims to explore the feasibility of using LSTM networks to predict crawling trajectories, thereby advancing our understanding of how to actively control crawling rehabilitation training robots.

METHODS

We collected joint trajectory data from 20 healthy infants (11 males and 9 females, aged 8-15 months) as they crawled on hands and knees. This study implemented LSTM networks to forecast bilateral elbow and knee trajectories based on corresponding joint angles. The data set comprised 58, 782 time steps, each containing 4 joint angles. We partitioned the data set into 70% for training and 30% for testing to evaluate predictive performance. We investigated a total of 24 combinations of input and output time-frames, with window sizes for input vectors ranging from 10, 15, 20, 30, 40, 50, 70, and 100 time steps, and output vectors from 5, 10, and 15 steps. Evaluation metrics included Mean Absolute Error (MAE), Mean Squared Error (MSE), and Correlation Coefficient (CC) to assess prediction accuracy.

RESULTS

The results indicate that across various input-output windows, the MAE for elbow joints ranged from 0.280 to 4.976°, MSE ranged from 0.203° to 59.186°, and CC ranged from 89.977% to 99.959%. For knee joints, MAE ranged from 0.277 to 4.262°, MSE from 0.229 to 53.272°, and CC from 89.454% to 99.944%. Results also show that smaller output window sizes lead to lower prediction errors. As expected, the LSTM predicting 5 output time steps has the lowest average error, while the LSTM predicting 15 time steps has the highest average error. In addition, variations in input window size had a minimal impact on average error when the output window size was fixed. Overall, the optimal performance for both elbow and knee joints was observed with input-output window sizes of 30 and 5 time steps, respectively, yielding an MAE of 0.295°, MSE of 0.260°, and CC of 99.938%.

CONCLUSIONS

This study demonstrates the feasibility of forecasting infant crawling trajectories using LSTM networks, which could potentially integrate with exoskeleton control systems. It experimentally explores how different input and output time-frames affect prediction accuracy and sets the stage for future research focused on optimizing models and developing effective control strategies to improve assistive crawling devices.

Leveraging graph neural networks and gate recurrent units for accurate and transparent prediction of baseball pitching speed

Long short-term memory (LSTM) networks are widely used in biomechanical data analysis but have the significant limitations in interpretability and decision transparency. Combining graph neural networks (GNN) with gate recurrent units (GRU) may offer a better solution. This study proposes and validates a hybrid GNN-GRU model for predicting baseball pitching speed and enhancing its interpretability using layer-wise relevance propagation (LRP). C3D data from 53 baseball athletes were downloaded from a public dataset. Kinematic features of 9 joints and pitching speed during the pitching process were calculated using Visual3D, resulting in a total of 208 valid pitches. The feature data were input into both LSTM and GNN-GRU hybrid models, with hyperparameters tuned using particle swarm optimization. LRP was employed to obtain the contribution rate changes of kinematic features to the prediction results throughout the pitching cycle. The prediction accuracy of the models was evaluated using mean absolute error (MAE), mean squared error (MSE), and R-squared (R2). The results showed that there were the significant statistical differences in the MAE and R2 metrics between the LSTM model and the GNN-GRU model in predicting pitching speed on the test set. The MAE (P = 0.000, Z = - 5.170, Cohen's d = 1.514) and R2 (P = 0.000, Z = - 2.981, Cohen's d = 2.314) of the LSTM model were significantly lower than those of the GNN-GRU model. Compared to LSTM, the GNN-GRU model achieved better prediction accuracy but was potentially more susceptible to the influence of data variability. Moreover, the GNN-GRU-based model demonstrated the better interpretability and decision transparency.

Estimation of Forces and Powers in Ergometer and Scull Rowing Based on Long Short-Term Memory Neural Networks

Analyzing performance in rowing, e.g., analyzing force and power output profiles produced either on ergometer or on boat, is a priority for trainers and athletes. The current state-of-the-art methodologies for rowing performance analysis involve the installation of dedicated instrumented equipment, with the most commonly employed systems being PowerLine and BioRow. This procedure can be both expensive and time-consuming, thus limiting trainers' ability to monitor athletes. In this study, we developed an easier-to-install and cheaper method for estimating rowers' forces and powers based only on cable position sensors for ergometer rowing and inertial measurement units (IMUs) and GPS for scull rowing. We used data from 12 and 11 rowers on ergometer and on boat, respectively, to train a long short-term memory (LSTM) network. The LSTM was able to reconstruct the forces and power at the gate with an overall mean absolute error of less than 5%. The reconstructed forces and power were able to reveal inter-subject differences in technique, with an accuracy of 93%. Performing leave-one-out validation showed a significant increase in error, confirming that more subjects are needed in order to develop a tool that could be generalizable to external athletes.

Time Series Classification for Predicting Biped Robot Step Viability

The prediction of the stability of future steps taken by a biped robot is a very important task, since it allows the robot controller to adopt the necessary measures in order to minimize damages if a fall is predicted. We present a classifier to predict the viability of a given planned step taken by a biped robot, i.e., if it will be stable or unstable. The features of the classifier are extracted from a feature engineering process exploiting the useful information contained in the time series generated in the trajectory planning of the step. In order to state the problem as a supervised classification one, we need the ground truth class for each planned step. This is obtained using the Predicted Step Viability (PSV) criterion. We also present a procedure to obtain a balanced and challenging training/testing dataset of planned steps that contains many steps in the border between stable and non stable regions. Following this trajectory planning strategy for the creation of the dataset we are able to improve the robustness of the classifier. Results show that the classifier is able to obtain a 95% of ROC AUC for this demanding dataset using only four time series among all the signals required by PSV to check viability. This allows to replace the PSV stability criterion, which is safe, robust but impossible to apply in real-time, by a simple, fast and embeddable classifier that can run in real time consuming much less resources than the PSV.

BlazePose-Seq2Seq: Leveraging Regular RGB Cameras for Robust Gait Assessment

Evaluation of human gait through smartphone-based pose estimation algorithms provides an attractive alternative to costly lab-bound instrumented assessment and offers a paradigm shift with real time gait capture for clinical assessment. Systems based on smart phones, such as OpenPose and BlazePose have demonstrated potential for virtual motion assessment but still lack the accuracy and repeatability standards required for clinical viability. Seq2seq architecture offers an alternative solution to conventional deep learning techniques for predicting joint kinematics during gait. This study introduces a novel enhancement to the low-powered BlazePose algorithm by incorporating a Seq2seq autoencoder deep learning model. To ensure data accuracy and reliability, synchronized motion capture involving an RGB camera and ten Vicon cameras were employed across three distinct self-selected walking speeds. This investigation presents a groundbreaking avenue for remote gait assessment, harnessing the potential of Seq2seq architectures inspired by natural language processing (NLP) to enhance pose estimation accuracy. When comparing BlazePose alone to the combination of BlazePose and 1D convolution Long Short-term Memory Network (1D-LSTM), Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM), the average mean absolute errors decreased from 13.4° to 5.3° for fast gait, from 16.3° to 7.5° for normal gait, and from 15.5° to 7.5° for slow gait at the left ankle joint angle respectively. The strategic utilization of synchronized data and rigorous testing methodologies further bolsters the robustness and credibility of these findings.

Implementing Gait Kinematic Trajectory Forecasting Models on an Embedded System

Smart algorithms for gait kinematic motion prediction in wearable assistive devices including prostheses, bionics, and exoskeletons can ensure safer and more effective device functionality. Although embedded systems can support the use of smart algorithms, there are important limitations associated with computational load. This poses a tangible barrier for models with increased complexity that demand substantial computational resources for superior performance. Forecasting through Recurrent Topology (FReT) represents a computationally lightweight time-series data forecasting algorithm with the ability to update and adapt to the input data structure that can predict complex dynamics. Here, we deployed FReT on an embedded system and evaluated its accuracy, computational time, and precision to forecast gait kinematics from lower-limb motion sensor data from fifteen subjects. FReT was compared to pretrained hyperparameter-optimized NNET and deep-NNET (D-NNET) model architectures, both with static model weight parameters and iteratively updated model weight parameters to enable adaptability to evolving data structures. We found that FReT was not only more accurate than all the network models, reducing the normalized root-mean-square error by almost half on average, but that it also provided the best balance between accuracy, computational time, and precision when considering the combination of these performance variables. The proposed FReT framework on an embedded system, with its improved performance, represents an important step towards the development of new sensor-aided technologies for assistive ambulatory devices.

Gait Trajectory Prediction on an Embedded Microcontroller Using Deep Learning

Achieving a normal gait trajectory for an amputee's active prosthesis is challenging due to its kinematic complexity. Accordingly, lower limb gait trajectory kinematics and gait phase segmentation are essential parameters in controlling an active prosthesis. Recently, the most practiced algorithm in gait trajectory generation is the neural network. Deploying such a complex Artificial Neural Network (ANN) algorithm on an embedded system requires performing the calculations on an external computational device; however, this approach lacks mobility and reliability. In this paper, more simple and reliable ANNs are investigated to be deployed on a single low-cost Microcontroller (MC) and hence provide system mobility. Two neural network configurations were studied: Multi-Layered Perceptron (MLP) and Convolutional Neural Network (CNN); the models were trained on shank and foot IMU data. The data were collected from four subjects and tested on a fifth to predict the trajectory of 200 ms ahead. The prediction was made for two cases: with and without providing the current phase of the gait. Then, the models were deployed on a low-cost microcontroller (ESP32). It was found that with fewer data (excluding the current gait phase), CNN achieved a better correlation coefficient of 0.973 when compared to 0.945 for MLP; when including the current phase, both network configurations achieved better correlation coefficients of nearly 0.98. However, when comparing the execution time required for the prediction on the intended MC, MLP was much faster than CNN, with an execution time of 2.4 ms and 142 ms, respectively. In summary, it was found that when training data are scarce, CNN is more efficient within the acceptable execution time, while MLP achieves relative accuracy with low execution time with enough data.

Deep learning with an attention mechanism for continuous biomechanical motion estimation across varied activities

Reliable estimation of desired motion trajectories plays a crucial part in the continuous control of lower extremity assistance devices such as prostheses and orthoses. Moreover, reliable estimation methods are also required to predict hard-to-measure biomechanical quantities (e.g., joint contact moment/force) for use in sports injury science. Recognising that human locomotion is an inherently time-sequential and limb-synergetic behaviour, this study investigates models and learning algorithms for predicting the motion of a subject’s leg from the motion of complementary limbs. The novel deep learning model architectures proposed are based on the Long Short-Term Memory approach with the addition of an attention mechanism. A dataset comprising Inertial Measurement Unit signals from 21 subjects traversing varied terrains was used, including stair ascent/descent, ramp ascent/descent, stopped, level-ground walking and the transitions between these conditions. Fourier Analysis is deployed to evaluate the model robustness, in addition to assessing time-based prediction errors. The experiment on three unseen test participants suggests that the branched neural network structure is preferred to tackle the multioutput problem, and the inclusion of an attention mechanism demonstrates improved performance in terms of accuracy, robustness and network size. An experimental comparison found that 57% of the model parameters were not needed after adding attention layers meanwhile the prediction error is lower than the LSTM model without attention mechanism. The attention model has errors of 9.06% and 7.64% (normalised root mean square error) for ankle and hip acceleration prediction respectively. Also, less high-frequency noise is present in the attention model predictions. We conclude that the internal structure of the proposed deep learning model is justified, principally the benefit of using an attention mechanism. Experimental results for biomechanical motion estimation are obtained, showing greater accuracy than only with LSTM. The trained attention model can be used throughout despite transitioning between terrain types. Such a model will be useful in, for example, the control of lower-limb prostheses, instead of the need to identify and switch between different trajectory generators for different walking modes.

Using Deep Learning to Predict Minimum Foot-Ground Clearance Event from Toe-Off Kinematics

Efficient, adaptive, locomotor function is critically important for maintaining our health and independence, but falls-related injuries when walking are a significant risk factor, particularly for more vulnerable populations such as older people and post-stroke individuals. Tripping is the leading cause of falls, and the swing-phase event Minimum Foot Clearance (MFC) is recognised as the key biomechanical determinant of tripping probability. MFC is defined as the minimum swing foot clearance, which is seen approximately mid-swing, and it is routinely measured in gait biomechanics laboratories using precise, high-speed, camera-based 3D motion capture systems. For practical intervention strategies designed to predict, and possibly assist, swing foot trajectory to prevent tripping, identification of the MFC event is essential; however, no technique is currently available to determine MFC timing in real-life settings outside the laboratory. One strategy has been to use wearable sensors, such as Inertial Measurement Units (IMUs), but these data are limited to primarily providing only tri-axial linear acceleration and angular velocity. The aim of this study was to develop Machine Learning (ML) algorithms to predict MFC timing based on the preceding toe-off gait event. The ML algorithms were trained using 13 young adults' foot trajectory data recorded from an Optotrak 3D motion capture system. A Deep Learning configuration was developed based on a Recurrent Neural Network with a Long Short-Term Memory (LSTM) architecture and Huber loss-functions to minimise MFC-timing prediction error. We succeeded in predicting MFC timing from toe-off characteristics with a mean absolute error of 0.07 s. Although further algorithm training using population-specific inputs are needed. The ML algorithms designed here can be used for real-time actuation of wearable active devices to increase foot clearance at critical MFC and reduce devastating tripping falls. Further developments in ML-guided actuation for active exoskeletons could prove highly effective in developing technologies to reduce tripping-related falls across a range of gait impaired populations.

Data-driven Gait-predictive Model for Anticipatory Prosthesis Control

Physiological movement is pre-planned based on current movement state, proprioception, and environmental cues. This preplanning is necessary to allow efficient use of the viscoelastic properties of musculoskeletal tissues in a 4D-environment. Similarly, efficient use of prosthetic devices needs to compensate for the time it takes to control the system. In this study, we propose a gated recurrent net-based gait predictive model to continuously predict the ankle angles and moments fifty milliseconds in advance based on the past trajectory of the input signals. It was observed that using a single input signal (the shank angle), high accuracy of prediction $(R^{2}\gt 0.91)$ was achieved for both ankle angle and moments on walking trials at a self-selected comfortable speed. The results of our study can be utilised for anticipatory lower-limb prosthesis control where embedded sensor information that reflects a prosthetic user's locomotive intent can be used to predict the required angles and moments in advance for actuating a prosthetic joint.

Performance of Deep Learning Models in Forecasting Gait Trajectories of Children with Neurological Disorders

Forecasted gait trajectories of children could be used as feedforward input to control lower limb robotic devices, such as exoskeletons and actuated orthotic devices (e.g., Powered Ankle Foot Orthosis—PAFO). Several studies have forecasted healthy gait trajectories, but, to the best of our knowledge, none have forecasted gait trajectories of children with pathological gait yet. These exhibit higher inter- and intra-subject variability compared to typically developing gait of healthy subjects. Pathological trajectories represent the typical gait patterns that rehabilitative exoskeletons and actuated orthoses would target. In this study, we implemented two deep learning models, a Long-Term Short Memory (LSTM) and a Convolutional Neural Network (CNN), to forecast hip, knee, and ankle trajectories in terms of corresponding Euler angles in the pitch, roll, and yaw form for children with neurological disorders, up to 200 ms in the future. The deep learning models implemented in our study are trained on data (available online) from children with neurological disorders collected by Gillette Children’s Speciality Healthcare over the years 1994–2017. The children’s ages range from 4 to 19 years old and the majority of them had cerebral palsy (73%), while the rest were a combination of neurological, developmental, orthopaedic, and genetic disorders (27%). Data were recorded with a motion capture system (VICON) with a sampling frequency of 120 Hz while walking for 15 m. We investigated a total of 35 combinations of input and output time-frames, with window sizes for input vectors ranging from 50–1000 ms, and output vectors from 8.33–200 ms. Results show that LSTMs outperform CNNs, and the gap in performance becomes greater the larger the input and output window sizes are. The maximum difference between the Mean Absolute Errors (MAEs) of the CNN and LSTM networks was 0.91 degrees. Results also show that the input size has no significant influence on mean prediction errors when the output window is 50 ms or smaller. For output window sizes greater than 50 ms, the larger the input window, the lower the error. Overall, we obtained MAEs ranging from 0.095–2.531 degrees for the LSTM network, and from 0.129–2.840 degrees for the CNN. This study establishes the feasibility of forecasting pathological gait trajectories of children which could be integrated with exoskeleton control systems and experimentally explores the characteristics of such intelligent systems under varying input and output window time-frames.

Application of Wearable Sensors in Actuation and Control of Powered Ankle Exoskeletons: A Comprehensive Review

Powered ankle exoskeletons (PAEs) are robotic devices developed for gait assistance, rehabilitation, and augmentation. To fulfil their purposes, PAEs vastly rely heavily on their sensor systems. Human–machine interface sensors collect the biomechanical signals from the human user to inform the higher level of the control hierarchy about the user’s locomotion intention and requirement, whereas machine–machine interface sensors monitor the output of the actuation unit to ensure precise tracking of the high-level control commands via the low-level control scheme. The current article aims to provide a comprehensive review of how wearable sensor technology has contributed to the actuation and control of the PAEs developed over the past two decades. The control schemes and actuation principles employed in the reviewed PAEs, as well as their interaction with the integrated sensor systems, are investigated in this review. Further, the role of wearable sensors in overcoming the main challenges in developing fully autonomous portable PAEs is discussed. Finally, a brief discussion on how the recent technology advancements in wearable sensors, including environment—machine interface sensors, could promote the future generation of fully autonomous portable PAEs is provided.

Machine Learning Model to Estimate Net Joint Moments during Lifting Task Using Wearable Sensors: A Preliminary Study for Design of Exoskeleton Control System

Accurately measuring the lower extremities and L5/S1 moments is important since L5/S1 moments are the principal parameters that measure the risk of musculoskeletal diseases during lifting. In this study, protocol that predicts lower extremities and L5/S1 moments with an insole sensor was proposed to replace the prior methods that have spatial constraints. The protocol is hierarchically composed of a classification model and a regression model to predict joint moments. Additionally, a single LSTM model was developed to compare with proposed protocol. To optimize hyperparameters of the machine learning model and input feature, Bayesian optimization method was adopted. As a result, the proposed protocol showed a relative root mean square error (rRMSE) of 8.06~13.88% while the single LSTM showed 9.30~18.66% rRMSE. This protocol in this research is expected to be a starting point for developing a system for estimating the lower extremity and L5/S1 moment during lifting that can replace the complex prior method and adopted to workplace environments. This novel study has the potential to precisely design a feedback iterative control system of an exoskeleton for the appropriate generation of an actuator torque.