Intuitive movement-based prosthesis control enables arm amputees to reach naturally in virtual reality

Impressive progress is being made in bionic limbs design and control. Yet, controlling the numerous joints of a prosthetic arm necessary to place the hand at a correct position and orientation to grasp objects remains challenging. Here, we designed an intuitive, movement-based prosthesis control that leverages natural arm coordination to predict distal joints missing in people with transhumeral limb loss based on proximal residual limb motion and knowledge of the movement goal. This control was validated on 29 participants, including seven with above-elbow limb loss, who picked and placed bottles in a wide range of locations in virtual reality, with median success rates over 99% and movement times identical to those of natural movements. This control also enabled 15 participants, including three with limb differences, to reach and grasp real objects with a robotic arm operated according to the same principle. Remarkably, this was achieved without any prior training, indicating that this control is intuitive and instantaneously usable. It could be used for phantom limb pain management in virtual reality, or to augment the reaching capabilities of invasive neural interfaces usually more focused on hand and grasp control.

Transhumeral Arm Reaching Motion Prediction through Deep Reinforcement Learning-Based Synthetic Motion Cloning

The lack of intuitive controllability remains a primary challenge in enabling transhumeral amputees to control a prosthesis for arm reaching with residual limb kinematics. Recent advancements in prosthetic arm control have focused on leveraging the predictive capabilities of artificial neural networks (ANNs) to automate elbow joint motion and wrist pronation-supination during target reaching tasks. However, large quantities of human motion data collected from different subjects for various activities of daily living (ADL) tasks are required to train these ANNs. For example, the reaching motion can be altered when the height of the desk is changed; however, it is cumbersome to conduct human experiments for all conditions. This paper proposes a framework for cloning motion datasets using deep reinforcement learning (DRL) to cater to training data requirements. DRL algorithms have been demonstrated to create human-like synergistic motion in humanoid agents to handle redundancy and optimize movements. In our study, we collected real motion data from six individuals performing multi-directional arm reaching tasks in the horizontal plane. We generated synthetic motion data that mimicked similar arm reaching tasks by utilizing a physics simulation and DRL-based arm manipulation. We then trained a CNN-LSTM network with different configurations of training motion data, including DRL, real, and hybrid datasets, to test the efficacy of the cloned motion data. The results of our evaluation showcase the effectiveness of the cloned motion data in training the ANN to predict natural elbow motion accurately across multiple subjects. Furthermore, motion data augmentation through combining real and cloned motion datasets has demonstrated the enhanced robustness of the ANN by supplementing and diversifying the limited training data. These findings have significant implications for creating synthetic dataset resources for various arm movements and fostering strategies for automatized prosthetic elbow motion.

Feasibility Evaluation of Online Classification-based Control for Gross Movement in a 2-DoF Prosthetic Arm

Regression and classification models have been extensively studied to exploit the myoelectric and kinematic input information from the residual limb for the control of multiple degree-of-freedom (DoF) powered prostheses. The gross movement control of above-elbow prostheses is mainly based on regression models which map the available inputs to continuous prosthetic poses. However, the regression output is sensitive to the variation in the input signal. The myoelectric signal variation is usually large due to unintentional muscle contractions, which can deteriorate the user-in-the-loop performance with respect to the offline analysis. Alternatively, the classification models offer the advantage of being more robust to the input signal variation, but they were predominantly used for fine motor functions such as grasping. For gross motor functions, the discrete output may cause issues. Therefore, this work attempts to investigate the feasibility of utilising the classification model to control a 2-DoF transhumeral prosthesis for gross movement. The performance of 6 able-bodied subjects was evaluated in performing reaching and orientation matching tasks with a prosthetic arm in a virtual reality environment. The results were compared with the case of using their intact arms and existing results using the regression model. Our findings indicate that the classification-based method provides comparable performance to the regression model, making it a potential alternative for gross arm movement in multi-DoF prosthetic arms.

Kinematic analysis of impairments and compensatory motor behavior during prosthetic grasping in below-elbow amputees

After a major upper limb amputation, the use of myoelectric prosthesis as assistive devices is possible. However, these prostheses remain quite difficult to control for grasping and manipulation of daily life objects. The aim of the present observational case study is to document the kinematics of grasping in a group of 10 below-elbow amputated patients fitted with a myoelectric prosthesis in order to describe and better understand their compensatory strategies. They performed a grasping to lift task toward 3 objects (a mug, a cylinder and a cone) placed at two distances within the reaching area in front of the patients. The kinematics of the trunk and upper-limb on the non-amputated and prosthetic sides were recorded with 3 electromagnetic Polhemus sensors placed on the hand, the forearm (or the corresponding site on the prosthesis) and the ipsilateral acromion. The 3D position of the elbow joint and the shoulder and elbow angles were calculated thanks to a preliminary calibration of the sensor position. We examined first the effect of side, distance and objects with non-parametric statistics. Prosthetic grasping was characterized by severe temporo-spatial impairments consistent with previous clinical or kinematic observations. The grasping phase was prolonged and the reaching and grasping components uncoupled. The 3D hand displacement was symmetrical in average, but with some differences according to the objects. Compensatory strategies involved the trunk and the proximal part of the upper-limb, as shown by a greater 3D displacement of the elbow for close target and a greater forward displacement of the acromion, particularly for far targets. The hand orientation at the time of grasping showed marked side differences with a more frontal azimuth, and a more "thumb-up" roll. The variation of hand orientation with the object on the prosthetic side, suggested that the lack of finger and wrist mobility imposed some adaptation of hand pose relative to the object. The detailed kinematic analysis allows more insight into the mechanisms of the compensatory strategies that could be due to both increased distal or proximal kinematic constraints. A better knowledge of those compensatory strategies is important for the prevention of musculoskeletal disorders and the development of innovative prosthetics.

Simultaneous control of 2DOF upper-limb prosthesis with body compensations-based control: a multiple cases study

Controlling several joints simultaneously is a common feature of natural arm movements. Robotic prostheses shall offer this possibility to their wearer. Yet, existing approaches to control a robotic upper-limb prosthesis from myoelectric interfaces do not satisfactorily respond to this need: standard methods provide sequential joint-by-joint motion control only; advanced pattern recognition-based approaches allow the control of a limited subset of synchronized multi-joint movements and remain complex to set up. In this paper, we exploit a control method of an upper-limb prosthesis based on body motion measurement called Compensations Cancellation Control (CCC). It offers a straightforward simultaneous control of the intermediate joints, namely the wrist and the elbow. Four transhumeral amputated participants performed the Refined Rolyan Clothespin Test with an experimental prosthesis alternatively running CCC and conventional joint-by-joint myoelectric control. Task performance, joint motions, body compensations and cognitive load were assessed. This experiment shows that CCC restores simultaneity between prosthetic joints while maintaining the level of performance of conventional myoelectric control (used on a daily basis by three participants), without increasing compensatory motions nor cognitive load.

Biological Plausibility of Arm Postures Influences the Controllability of Robotic Arm Teleoperation

OBJECTIVE

We investigated how participants controlling a humanoid robotic arm's 3D endpoint position by moving their own hand are influenced by the robot's postures. We hypothesized that control would be facilitated (impeded) by biologically plausible (implausible) postures of the robot.

BACKGROUND

Kinematic redundancy, whereby different arm postures achieve the same goal, is such that a robotic arm or prosthesis could theoretically be controlled with less signals than constitutive joints. However, congruency between a robot's motion and our own is known to interfere with movement production. Hence, we expect the human-likeness of a robotic arm's postures during endpoint teleoperation to influence controllability.

METHOD

Twenty-two able-bodied participants performed a target-reaching task with a robotic arm whose endpoint's 3D position was controlled by moving their own hand. They completed a two-condition experiment corresponding to the robot displaying either biologically plausible or implausible postures.

RESULTS

Upon initial practice in the experiment's first part, endpoint trajectories were faster and shorter when the robot displayed human-like postures. However, these effects did not persist in the second part, where performance with implausible postures appeared to have benefited from initial practice with plausible ones.

CONCLUSION

Humanoid robotic arm endpoint control is impaired by biologically implausible joint coordinations during initial familiarization but not afterwards, suggesting that the human-likeness of a robot's postures is more critical for control in this initial period.

APPLICATION

These findings provide insight for the design of robotic arm teleoperation and prosthesis control schemes, in order to favor better familiarization and control from their users.

Separability of Input Features and the Resulting Accuracy in Classifying Target Poses for Active Transhumeral Prosthetic Interfaces

In active prostheses, it is desired to achieve target poses for a given family of tasks, for example, in the task of forward reaching using a transhumeral prosthesis with coordinated joint movements. To do so, it is necessary to distinguish these target poses accurately using the input features (e.g. kinematic and sEMG) obtained from the human users. However, the input features have conventionally been selected through human observations and influenced heavily by the availability of sensors in this context, which may not always yield the most relevant information to differentiate the target poses in the given task. In order to better select from a pool of available input features, those most appropriate for a given set of target poses, a measure that correlates well with the resulting classification accuracy is required so that it can inform the interface design process. In this paper, a scatter-matrix based class separability measure is adopted to quantitatively evaluate the separability of the target poses from their corresponding input features. A human experiment was performed on ten able-bodied subjects. Subjects were asked to perform forward-reaching movements with their arms on nine target poses in a virtual reality (VR) platform and the corresponding kinematic information of their arm movement and muscle activities were recorded. The accuracy of the prosthetic interface in determining the intended target poses of the human user during forward reaching is evaluated for different combinations of input features, selected from the kinematic and sEMG sensors worn by the users. The results demonstrate that employing input features that yield a high separability measure between target poses results in a high accuracy in identifying the intended target poses in the execution of the task.

The effects of forearm movements on human gait during walking with various self-selected speeds

The forearms significantly contribute to the upper extremity movements and, consequently, whole-body responses during locomotion. The purpose of this study is to provide a more in-depth understanding of the mechanism controlling forearm movements during walking by comprehensively investigating the effects of the forearms on the lower and upper limb movements. Such an understanding can provide critical information for the design and control of robotic upper-limb prostheses. Twelve healthy young participants were recruited to compare their gait during (1) natural walking, (2) walking while wearing a pair of artificial passive forearms and having their actual forearms restrained by orthopedic braces, and (3) walking with only having their forearms restrained by the braces (i.e., no artificial forearms). While the passive forearms in condition 2 were to determine if the forearm movements were passively or actively controlled, condition 3 was to account for the effects of restraining the forearms in condition 2. The participants' lower-limb joint angles and spatiotemporal parameters remained unchanged across the three conditions while walking at their normal and fast self-selected gait speeds. However, significant decreases were observed in the shoulder and trunk angles, the interlimb coordination, and the shoulder-trunk correlations when walking with the artificial forearms. These observations were in tandem with the increased muscle activity of the biceps, trapeziuses, and posterior deltoids, which controlled the shoulder motion and trunk rotation during walking with the artificial forearms across both normal and fast self-selected speeds. Although not significant, the metabolic energy analysis of five participants revealed an increase during walking with artificial forearms. The results support the idea that the body actively controls the forearm movements through the shoulder and trunk rotations to mitigate the undesired disturbances induced by the passive forearm movements during locomotion.

Shoulder kinematics plus contextual target information enable control of multiple distal joints of a simulated prosthetic arm and hand

Background

Prosthetic restoration of reach and grasp function after a trans-humeral amputation requires control of multiple distal degrees of freedom in elbow, wrist and fingers. However, such a high level of amputation reduces the amount of available myoelectric and kinematic information from the residual limb.

Methods

To overcome these limits, we added contextual information about the target’s location and orientation such as can now be extracted from gaze tracking by computer vision tools. For the task of picking and placing a bottle in various positions and orientations in a 3D virtual scene, we trained artificial neural networks to predict postures of an intact subject’s elbow, forearm and wrist (4 degrees of freedom) either solely from shoulder kinematics or with additional knowledge of the movement goal. Subjects then performed the same tasks in the virtual scene with distal joints predicted from the context-aware network.

Results

Average movement times of 1.22s were only slightly longer than the naturally controlled movements (0.82 s). When using a kinematic-only network, movement times were much longer (2.31s) and compensatory movements from trunk and shoulder were much larger. Integrating contextual information also gave rise to motor synergies closer to natural joint coordination.

Conclusions

Although notable challenges remain before applying the proposed control scheme to a real-world prosthesis, our study shows that adding contextual information to command signals greatly improves prediction of distal joint angles for prosthetic control.