Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics-Part 1: Robustness

Prediction of Achilles Tendon Force During Common Motor Tasks From Markerless Video

Remodeling of the Achilles tendon (AT) is partly driven by its mechanical environment. AT force can be estimated with neuromusculoskeletal (NMSK) modeling; however, the complex experimental setup required to perform the analyses confines use to the laboratory. We developed task-specific long short-term memory (LSTM) neural networks that employ markerless video data to predict the AT force during walking, running, countermovement jump, single-leg landing, and single-leg heel rise. The task-specific LSTM models were trained on pose estimation keypoints and corresponding AT force data from 16 subjects, calculated via an established NMSK modeling pipeline, and cross-validated using a leave-one-subject-out approach. As proof-of-concept, new motion data of one participant was collected with two smartphones and used to predict AT forces. The task-specific LSTM models predicted the time-series AT force using synthesized pose estimation data with root mean square error (RMSE) ≤ 526 N, normalized RMSE (nRMSE) ≤ 0.21 , R 2 ≥ 0.81 . Walking task resulted the most accurate with RMSE = 189±62 N; nRMSE = 0.11±0.03 , R 2 = 0.92±0.04 . AT force predicted with smartphones video data was physiologically plausible, agreeing in timing and magnitude with established force profiles. This study demonstrated the feasibility of using low-cost solutions to deploy complex biomechanical analyses outside the laboratory.

Agreement between a markerless and a marker-based motion capture systems for balance related quantities

Balance studies usually focus on quantities describing the global body motion. Assessing such quantities using classical marker-based approach can be tedious and modify the participant's behaviour. The recent development of markerless motion capture methods could bypass the issues related to the use of markers. This work compared dynamic balance related quantities obtained with markers and videos. Sixteen young healthy participants performed four different motor tasks: walking at self-selected speed, balance loss, walking on a narrow beam and countermovement jumps. Their movements were recorded simultaneously by marker-based and markerless motion capture systems. Videos were processed using a commercial markerless pose estimation software, Theia3D. The centre of mass position (CoM) was computed, and the associated extrapolated centre of mass position (XCoM) and whole-body angular momentum (WBAM) were derived. Bland-Altman analysis was performed and root mean square difference (RMSD) and coefficient of correlation were computed to compare the results obtained with marker-based and markerless methods. Bias remained of the magnitude of a few mm for CoM and XCoM positions, and RMSD of CoM and XCoM was around 1 cm. RMSD of the WBAM was less than 10 % of the total amplitude in any direction, and bias was less than 1 %. Results suggest that outcomes of balance studies will be similar whether marker-based or markerless motion capture system are used. Nevertheless, one should be careful when assessing dynamic movements such as jumping, as they displayed the biggest differences (both bias and RMSD), although it is unclear whether these differences are due to errors in markerless or marker-based motion capture system.

Validation of portable in-clinic video-based gait analysis for prosthesis users

Despite the common focus of gait in rehabilitation, there are few tools that allow quantitatively characterizing gait in the clinic. We recently described an algorithm, trained on a large dataset from our clinical gait analysis laboratory, which produces accurate cycle-by-cycle estimates of spatiotemporal gait parameters including step timing and walking velocity. Here, we demonstrate this system generalizes well to clinical care with a validation study on prosthetic users seen in therapy and outpatient clinics. Specifically, estimated walking velocity was similar to annotated 10-m walking velocities, and cadence and foot contact times closely mirrored our wearable sensor measurements. Additionally, we found that a 2D keypoint detector pretrained on largely able-bodied individuals struggles to localize prosthetic joints, particularly for those individuals with more proximal or bilateral amputations, but after training a prosthetic-specific joint detector video-based gait analysis also works on these individuals. Further work is required to validate the other outputs from our algorithm including sagittal plane joint angles and step length. Code for the gait transformer and the trained weights are available at https://github.com/peabody124/GaitTransformer .

Forward dynamics computational modelling of a cyclist fall with the inclusion of protective response using deep learning-based human pose estimation

Single bicycle crashes, i.e., falls and impacts not involving a collision with another road user, are a significantly underestimated road safety problem. The motions and behaviours of falling people, or fall kinematics, are often investigated in the injury biomechanics research field. Understanding the mechanics of a fall can help researchers develop better protective gear and safety measures to reduce the risk of injury. However, little is known about cyclist fall kinematics or dynamics. Therefore, in this study, a video analysis of cyclist falls is performed to investigate common kinematic forms and impact patterns. Furthermore, a pipeline involving deep learning-based human pose estimation and inverse kinematics optimisation is created for extracting human motion from real-world footage of falls to initialise forward dynamics computational human body models. A bracing active response is then optimised for using a genetic algorithm. This is then applied to a case study of a cyclist fall. The kinematic forms characterised in this study can be used to inform initial conditions for computational modelling and injury estimation in cyclist falls. Findings indicate that protective response is an important consideration in fall kinematics and dynamics, and should be included in computational modelling. Furthermore, the novel reconstruction pipeline proposed here can be applied more broadly for traumatic injury biomechanics tasks. The tool developed in this study is available at https://kevgildea.github.io/KinePose/.

Examination of the Accuracy of Movement Tracking Systems for Monitoring Exercise for Musculoskeletal Rehabilitation

When patients perform musculoskeletal rehabilitation exercises, it is of great importance to observe the correctness of their performance. The aim of this study is to increase the accuracy of recognizing human movements during exercise. The process of monitoring and evaluating musculoskeletal rehabilitation exercises was modeled using various tracking systems, and the necessary algorithms for processing information for each of the tracking systems were formalized. An approach to classifying exercises using machine learning methods is presented. Experimental studies were conducted to identify the most accurate tracking systems (virtual reality trackers, motion capture, and computer vision). A comparison of machine learning models is carried out to solve the problem of classifying musculoskeletal rehabilitation exercises, and 96% accuracy is obtained when using multilayer dense neural networks. With the use of computer vision technologies and the processing of a full set of body points, the accuracy of classification achieved is 100%. The hypotheses on the ranking of tracking systems based on the accuracy of positioning of human target points, the presence of restrictions on application in the field of musculoskeletal rehabilitation, and the potential to classify exercises are fully confirmed.

Improved Trajectory Reconstruction for Markerless Pose Estimation

Markerless pose estimation allows reconstructing human movement from multiple synchronized and calibrated views, and has the potential to make movement analysis easy and quick, including gait analysis. This could enable much more frequent and quantitative characterization of gait impairments, allowing better monitoring of outcomes and responses to interventions. However, the impact of different keypoint detectors and reconstruction algorithms on markerless pose estimation accuracy has not been thoroughly evaluated. We tested these algorithmic choices on data acquired from a multicamera system from a heterogeneous sample of 53 individuals in a rehabilitation hospital. We found that using a top-down keypoint detector and reconstructing trajectories with an implicit function enabled accurate, smooth, and anatomically plausible trajectories, with a noise in the step width estimates compared to a GaitRite walkway of only 9mm.

Towards Single Camera Human 3D-Kinematics

Markerless estimation of 3D Kinematics has the great potential to clinically diagnose and monitor movement disorders without referrals to expensive motion capture labs; however, current approaches are limited by performing multiple de-coupled steps to estimate the kinematics of a person from videos. Most current techniques work in a multi-step approach by first detecting the pose of the body and then fitting a musculoskeletal model to the data for accurate kinematic estimation. Errors in training data of the pose detection algorithms, model scaling, as well the requirement of multiple cameras limit the use of these techniques in a clinical setting. Our goal is to pave the way toward fast, easily applicable and accurate 3D kinematic estimation. To this end, we propose a novel approach for direct 3D human kinematic estimation D3KE from videos using deep neural networks. Our experiments demonstrate that the proposed end-to-end training is robust and outperforms 2D and 3D markerless motion capture based kinematic estimation pipelines in terms of joint angles error by a large margin (35% from 5.44 to 3.54 degrees). We show that D3KE is superior to the multi-step approach and can run at video framerate speeds. This technology shows the potential for clinical analysis from mobile devices in the future.

Ubiquitous Computing in Sports and Physical Activity-Recent Trends and Developments

The use of small, interconnected and intelligent tools within the broad framework of pervasive computing for analysis and assessments in sport and physical activity is not a trend in itself but defines a way for information to be handled, processed and utilised: everywhere, at any time. The demand for objective data to support decision making prompted the adoption of wearables that evolve to fulfil the aims of assessing athletes and practitioners as closely as possible with their performance environments. In the present paper, we mention and discuss the advancements in ubiquitous computing in sports and physical activity in the past 5 years. Thus, recent developments in wearable sensors, cloud computing and artificial intelligence tools have been the pillars for a major change in the ways sport-related analyses are performed. The focus of our analysis is wearable technology, computer vision solutions for markerless tracking and their major contribution to the process of acquiring more representative data from uninhibited actions in realistic ecological conditions. We selected relevant literature on the applications of such approaches in various areas of sports and physical activity while outlining some limitations of the present-day data acquisition and data processing practices and the resulting sensors' functionalities, as well as the limitations to the data-driven informed decision making in the current technological and scientific framework. Finally, we hypothesise that a continuous merger of measurement, processing and analysis will lead to the development of more reliable models utilising the advantages of open computing and unrestricted data access and allow for the development of personalised-medicine-type approaches to sport training and performance.

A New Approach for the Tribological and Mechanical Characterization of a Hip Prosthesis Trough a Numerical Model Based on Artificial Intelligence Algorithms and Humanoid Multibody Model

In recent years, thanks to the development of additive manufacturing techniques, pros-thetic surgery has reached increasingly cutting-edge levels, revolutionizing the clinical course of patients suffering from joint arthritis, rheumatoid arthritis, post-traumatic arthrosis, etc. This work aims to evaluate the best materials for prosthetic surgery in hip implants from a tribological and mechanical point of view by using a machine-learning algorithm coupling with multi-body modeling and Finite Element Method (FEM) simulations. The innovative aspect is represented by the use of machine learning for the creation of a humanoid model in a multibody software environment that aimed to evaluate the load and rotation condition at the hip joint. After the boundary conditions have been defined, a Finite Element (FE) model of the hip implant has been created. The material properties and the information on the tribological behavior of the material couplings under investigation have been obtained from literature studies. The wear process has been investigated through the implementation of the Archard’s wear law in the FE model. The results of the FE simulation show that the best wear behavior has been obtained by CoCr alloy/UHMWPE coupling with a volume loss due to a wear of 0.004 μm3 at the end of the simulation of ten sitting cycles. After the best pairs in terms of wear has been established, a topology optimization of the whole hip implant structure has been performed. The results show that, after the optimization process, it was possible to reduce implant mass making the implant 28.12% more lightweight with respect to the original one.

Identifying the Posture of Young Adults in Walking Videos by Using a Fusion Artificial Intelligent Method

Many neurological and musculoskeletal disorders are associated with problems related to postural movement. Noninvasive tracking devices are used to record, analyze, measure, and detect the postural control of the body, which may indicate health problems in real time. A total of 35 young adults without any health problems were recruited for this study to participate in a walking experiment. An iso-block postural identity method was used to quantitatively analyze posture control and walking behavior. The participants who exhibited straightforward walking and skewed walking were defined as the control and experimental groups, respectively. Fusion deep learning was applied to generate dynamic joint node plots by using OpenPose-based methods, and skewness was qualitatively analyzed using convolutional neural networks. The maximum specificity and sensitivity achieved using a combination of ResNet101 and the naïve Bayes classifier were 0.84 and 0.87, respectively. The proposed approach successfully combines cell phone camera recordings, cloud storage, and fusion deep learning for posture estimation and classification.

Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy

Two-dimensional deep-learning pose estimation algorithms can suffer from biases in joint pose localizations, which are reflected in triangulated coordinates, and then in 3D joint angle estimation. Pose2Sim, our robust markerless kinematics workflow, comes with a physically consistent OpenSim skeletal model, meant to mitigate these errors. Its accuracy was concurrently validated against a reference marker-based method. Lower-limb joint angles were estimated over three tasks (walking, running, and cycling) performed multiple times by one participant. When averaged over all joint angles, the coefficient of multiple correlation (CMC) remained above 0.9 in the sagittal plane, except for the hip in running, which suffered from a systematic 15° offset (CMC = 0.65), and for the ankle in cycling, which was partially occluded (CMC = 0.75). When averaged over all joint angles and all degrees of freedom, mean errors were 3.0°, 4.1°, and 4.0°, in walking, running, and cycling, respectively; and range of motion errors were 2.7°, 2.3°, and 4.3°, respectively. Given the magnitude of error traditionally reported in joint angles computed from a marker-based optoelectronic system, Pose2Sim is deemed accurate enough for the analysis of lower-body kinematics in walking, cycling, and running.

A SWOT Analysis of Portable and Low-Cost Markerless Motion Capture Systems to Assess Lower-Limb Musculoskeletal Kinematics in Sport

Markerless motion capture systems are promising for the assessment of movement in more real world research and clinical settings. While the technology has come a long way in the last 20 years, it is important for researchers and clinicians to understand the capacities and considerations for implementing these types of systems. The current review provides a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis related to the successful adoption of markerless motion capture technology for the assessment of lower-limb musculoskeletal kinematics in sport medicine and performance settings. 31 articles met the a priori inclusion criteria of this analysis. Findings from the analysis indicate that the improving accuracy of these systems via the refinement of machine learning algorithms, combined with their cost efficacy and the enhanced ecological validity outweighs the current weaknesses and threats. Further, the analysis makes clear that there is a need for multidisciplinary collaboration between sport scientists and computer vision scientists to develop accurate clinical and research applications that are specific to sport. While work remains to be done for broad application, markerless motion capture technology is currently on a positive trajectory and the data from this analysis provide an efficient roadmap toward widespread adoption.