ProACT: An Augmented Reality Testbed for Intelligent Prosthetic Arms

Upper-limb amputees face tremendous difficulty in operating dexterous powered prostheses. Previous work has shown that aspects of prosthetic hand, wrist, or elbow control can be improved through "intelligent" control, by combining movement-based or gaze-based intent estimation with low-level robotic autonomy. However, no such solutions exist for whole-arm control. Moreover, hardware platforms for advanced prosthetic control are expensive, and existing simulation platforms are not well-designed for integration with robotics software frameworks. We present the Prosthetic Arm Control Testbed (ProACT), a platform for evaluating intelligent control methods for prosthetic arms in an immersive (Augmented Reality) simulation setting. We demonstrate the use of ProACT through preliminary studies, with non-amputee participants performing an adapted Box-and-Blocks task with and without intent estimation. We further discuss how our observations may inform the design of prosthesis control methods, as well as the design of future studies using the platform. To the best of our knowledge, this constitutes the first study of semi-autonomous control for complex whole-arm prostheses, the first study including sequential task modeling in the context of wearable prosthetic arms, and the first testbed of its kind. Towards the goal of supporting future research in intelligent prosthetics, the system is built upon on existing open-source frameworks for robotics, and is available at https://arm.stanford.edu/proact.

Movement-Based Prosthesis Control with Angular Trajectory Is Getting Closer to Natural Arm Coordination

Traditional myoelectric controls of trans-humeral prostheses fail to provide intuitive coordination of the necessary degrees of freedom. We previously showed that by using artificial neural network predictions to reconstruct distal joints, based on the shoulder posture and movement goals (i.e., position and orientation of the targeted object), participants were able to position and orient an avatar hand to grasp objects with natural arm performances. However, this control involved rapid and unintended prosthesis movements at each modification of the movement goal, impractical for real-life scenarios. Here, we eliminate this abrupt change using novel methods based on an angular trajectory, determined from the speed of stump movement and the gap between the current and the 'goal' distal configurations. These new controls are tested offline and online (i.e., involving participants-in-the-loop) and compared to performances obtained with a natural control. Despite a slight increase in movement time, the new controls allowed twelve valid participants and six participants with trans-humeral limb loss to reach objects at various positions and orientations without prior training. Furthermore, no usability or workload degradation was perceived by participants with upper limb disabilities. The good performances achieved highlight the potential acceptability and effectiveness of those controls for our target population.

Alternative muscle synergy patterns of upper limb amputees

Myoelectric hand prostheses are effective tools for upper limb amputees to regain hand functions. Much progress has been made with pattern recognition algorithms to recognize surface electromyography (sEMG) patterns, but few attentions was placed on the amputees' motor learning process. Many potential myoelectric prostheses users could not fully master the control or had declined performance over time. It is possible that learning to produce distinct and consistent muscle activation patterns with the residual limb could help amputees better control the myoelectric prosthesis. In this study, we observed longitudinal effect of motor skill learning with 2 amputees who have developed alternative muscle activation patterns in response to the same set of target prosthetic actions. During a 10-week program, amputee participants were trained to produce distinct and constant muscle activations with visual feedback of live sEMG and without interaction with prosthesis. At the end, their sEMG patterns were different from each other and from non-amputee control groups. For certain intended hand motion, gradually reducing root mean square (RMS) variance was observed. The learning effect was also assessed with a CNN-LSTM mixture classifier designed for mobile sEMG pattern recognition. The classification accuracy had a rising trend over time, implicating potential performance improvement of myoelectric prosthesis control. A follow-up session took place 6 months after the program and showed lasting effect of the motor skill learning in terms of sEMG pattern classification accuracy. The results indicated that with proper feedback training, amputees could learn unique muscle activation patterns that allow them to trigger intended prosthesis functions, and the original motor control scheme is updated. The effect of such motor skill learning could help to improve myoelectric prosthetic control performance.

Intuitive movement-based prosthesis control enables arm amputees to reach naturally in virtual reality

Impressive progress is being made in bionic limbs design and control. Yet, controlling the numerous joints of a prosthetic arm necessary to place the hand at a correct position and orientation to grasp objects remains challenging. Here, we designed an intuitive, movement-based prosthesis control that leverages natural arm coordination to predict distal joints missing in people with transhumeral limb loss based on proximal residual limb motion and knowledge of the movement goal. This control was validated on 29 participants, including seven with above-elbow limb loss, who picked and placed bottles in a wide range of locations in virtual reality, with median success rates over 99% and movement times identical to those of natural movements. This control also enabled 15 participants, including three with limb differences, to reach and grasp real objects with a robotic arm operated according to the same principle. Remarkably, this was achieved without any prior training, indicating that this control is intuitive and instantaneously usable. It could be used for phantom limb pain management in virtual reality, or to augment the reaching capabilities of invasive neural interfaces usually more focused on hand and grasp control.

IMU Sensors Measurements Towards the Development of Novel Prosthetic Arm Control Strategies

The complexity of the human upper limb makes replicating it in a prosthetic device a significant challenge. With advancements in mechatronic developments involving the addition of a large number of degrees of freedom, novel control strategies are required. To accommodate this need, this study aims at developing an IMU-based control for the HannesARM upper-limb prosthetic device, as a proof-of-concept for new control strategies integrating data-fusion approaches. The natural human control of the upper-limb is based on different inputs that allow adaptive control. To mimic this in prostheses, the implementation of IMUs provides kinematic information of both the stump and the prosthesis to enrich the EMG control. The principle of operation is to decode upper limb movements by using a custom-made system and to replicate them in prosthetic arms improving the control algorithms. To evaluate the system's effectiveness, the custom algorithm's motion extraction was compared to a motion capture system using fifteen able-bodied subjects. The results showed that this system scored 0.16 ± 0.04 and 0.81 ± 0.12 in Root Mean Squared Error and Cross-Correlation compared to the motion capture system. Experimental results demonstrate how this work can extract valuable kinematic information necessary for new and improved control strategies, such as intention detection or pattern recognition, to allow users to perform a broader range of tasks and enhancing in turn their quality of life.

A Review of Myoelectric Control for Prosthetic Hand Manipulation

Myoelectric control for prosthetic hands is an important topic in the field of rehabilitation. Intuitive and intelligent myoelectric control can help amputees to regain upper limb function. However, current research efforts are primarily focused on developing rich myoelectric classifiers and biomimetic control methods, limiting prosthetic hand manipulation to simple grasping and releasing tasks, while rarely exploring complex daily tasks. In this article, we conduct a systematic review of recent achievements in two areas, namely, intention recognition research and control strategy research. Specifically, we focus on advanced methods for motion intention types, discrete motion classification, continuous motion estimation, unidirectional control, feedback control, and shared control. In addition, based on the above review, we analyze the challenges and opportunities for research directions of functionality-augmented prosthetic hands and user burden reduction, which can help overcome the limitations of current myoelectric control research and provide development prospects for future research.

Limb loading enhances skill transfer between augmented and physical reality tasks during limb loss rehabilitation

BACKGROUND

Virtual and augmented reality (AR) have become popular modalities for training myoelectric prosthesis control with upper-limb amputees. While some systems have shown moderate success, it is unclear how well the complex motor skills learned in an AR simulation transfer to completing the same tasks in physical reality. Limb loading is a possible dimension of motor skill execution that is absent in current AR solutions that may help to increase skill transfer between the virtual and physical domains.

METHODS

We implemented an immersive AR environment where individuals could operate a myoelectric virtual prosthesis to accomplish a variety of object relocation manipulations. Intact limb participants were separated into three groups, the load control (CG_LD; [Formula: see text]), the AR control (CG_AR; [Formula: see text]), and the experimental group (EG; [Formula: see text]). Both the CG_AR and EG completed a 5-session prosthesis training protocol in AR while the CG_LD performed simple muscle training. The EG attempted manipulations in AR while undergoing limb loading. The CG_AR attempted the same manipulations without loading. All participants performed the same manipulations in physical reality while operating a real prosthesis pre- and post-training. The main outcome measure was the change in the number of manipulations completed during the physical reality assessments (i.e. completion rate). Secondary outcomes included movement kinematics and visuomotor behavior.

RESULTS

The EG experienced a greater increase in completion rate post-training than both the CG_AR and CG_LD. This performance increase was accompanied by a shorter motor learning phase, the EG's performance saturating in less sessions of AR training than the CG_AR.

CONCLUSION

The results demonstrated that limb loading plays an important role in transferring complex motor skills learned in virtual spaces to their physical reality analogs. While participants who did not receive limb loading were able to receive some functional benefit from AR training, participants who received the loading experienced a greater positive change in motor performance with their performance saturating in fewer training sessions.

Haptic shared control improves neural efficiency during myoelectric prosthesis use

Clinical myoelectric prostheses lack the sensory feedback and sufficient dexterity required to complete activities of daily living efficiently and accurately. Providing haptic feedback of relevant environmental cues to the user or imbuing the prosthesis with autonomous control authority have been separately shown to improve prosthesis utility. Few studies, however, have investigated the effect of combining these two approaches in a shared control paradigm, and none have evaluated such an approach from the perspective of neural efficiency (the relationship between task performance and mental effort measured directly from the brain). In this work, we analyzed the neural efficiency of 30 non-amputee participants in a grasp-and-lift task of a brittle object. Here, a myoelectric prosthesis featuring vibrotactile feedback of grip force and autonomous control of grasping was compared with a standard myoelectric prosthesis with and without vibrotactile feedback. As a measure of mental effort, we captured the prefrontal cortex activity changes using functional near infrared spectroscopy during the experiment. It was expected that the prosthesis with haptic shared control would improve both task performance and mental effort compared to the standard prosthesis. Results showed that only the haptic shared control system enabled users to achieve high neural efficiency, and that vibrotactile feedback was important for grasping with the appropriate grip force. These results indicate that the haptic shared control system synergistically combines the benefits of haptic feedback and autonomous controllers, and is well-poised to inform such hybrid advancements in myoelectric prosthesis technology.

Continuous Semi-autonomous Prosthesis Control Using a Depth Sensor on the Hand

Modern myoelectric prostheses can perform multiple functions (e.g., several grasp types and wrist rotation) but their intuitive control by the user is still an open challenge. It has been recently demonstrated that semi-autonomous control can allow the subjects to operate complex prostheses effectively; however, this approach often requires placing sensors on the user. The present study proposes a system for semi-autonomous control of a myoelectric prosthesis that requires a single depth sensor placed on the dorsal side of the hand. The system automatically pre-shapes the hand (grasp type, size, and wrist rotation) and allows the user to grasp objects of different shapes, sizes and orientations, placed individually or within cluttered scenes. The system “reacts” to the side from which the object is approached, and enables the user to target not only the whole object but also an object part. Another unique aspect of the system is that it relies on online interaction between the user and the prosthesis; the system reacts continuously on the targets that are in its focus, while the user interprets the movement of the prosthesis to adjust aiming. Experimental assessment was conducted in ten able-bodied participants to evaluate the feasibility and the impact of training on prosthesis-user interaction. The subjects used the system to grasp a set of objects individually (Phase I) and in cluttered scenarios (Phase II), while the time to accomplish the task (TAT) was used as the performance metric. In both phases, the TAT improved significantly across blocks. Some targets (objects and/or their parts) were more challenging, requiring thus significantly more time to handle, but all objects and scenes were successfully accomplished by all subjects. The assessment therefore demonstrated that the system is indeed robust and effective, and that the subjects could successfully learn how to aim with the system after a brief training. This is an important step toward the development of a self-contained semi-autonomous system convenient for clinical applications.

Towards Robust, Adaptive and Reliable Upper-limb Motion Estimation Using Machine Learning and Deep Learning--A Survey in Myoelectric Control

To develop multi-functional human-machine interfaces that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) and deep learning (DL) techniques have been widely implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, due to the high complexity of upper-limb movements and the inherent non-stable characteristics of sEMG, the usability of ML/DL based control schemes is still greatly limited in practical scenarios. To this end, tremendous efforts have been made to improve model robustness, adaptation, and reliability. In this article, we provide a systematic review on recent achievements, mainly from three categories: multi-modal sensing fusion to gain additional information of the user, transfer learning (TL) methods to eliminate domain shift impacts on estimation models, and post-processing approaches to obtain more reliable outcomes. Special attention is given to fusion strategies, deep TL frameworks, and confidence estimation. \textcolor{red}{Research challenges and emerging opportunities, with respect to hardware development, public resources, and decoding strategies, are also analysed to provide perspectives for future developments.

Impact of Shared Control Modalities on Performance and Usability of Semi-autonomous Prostheses

Semi-autonomous (SA) control of upper-limb prostheses can improve the performance and decrease the cognitive burden of a user. In this approach, a prosthesis is equipped with additional sensors (e.g., computer vision) that provide contextual information and enable the system to accomplish some tasks automatically. Autonomous control is fused with a volitional input of a user to compute the commands that are sent to the prosthesis. Although several promising prototypes demonstrating the potential of this approach have been presented, methods to integrate the two control streams (i.e., autonomous and volitional) have not been systematically investigated. In the present study, we implemented three shared control modalities (i.e., sequential, simultaneous, and continuous) and compared their performance, as well as the cognitive and physical burdens imposed on the user. In the sequential approach, the volitional input disabled the autonomous control. In the simultaneous approach, the volitional input to a specific degree of freedom (DoF) activated autonomous control of other DoFs, whereas in the continuous approach, autonomous control was always active except for the DoFs controlled by the user. The experiment was conducted in ten able-bodied subjects, and these subjects used an SA prosthesis to perform reach-and-grasp tasks while reacting to audio cues (dual tasking). The results demonstrated that, compared to the manual baseline (volitional control only), all three SA modalities accomplished the task in a shorter time and resulted in less volitional control input. The simultaneous SA modality performed worse than the sequential and continuous SA approaches. When systematic errors were introduced in the autonomous controller to generate a mismatch between the goals of the user and controller, the performance of SA modalities substantially decreased, even below the manual baseline. The sequential SA scheme was the least impacted one in terms of errors. The present study demonstrates that a specific approach for integrating volitional and autonomous control is indeed an important factor that significantly affects the performance and physical and cognitive load, and therefore these should be considered when designing SA prostheses.

Challenges and solutions for application and wider adoption of wearable robots

The science and technology of wearable robots are steadily advancing, and the use of such robots in our everyday life appears to be within reach. Nevertheless, widespread adoption of wearable robots should not be taken for granted, especially since many recent attempts to bring them to real-life applications resulted in mixed outcomes. The aim of this article is to address the current challenges that are limiting the application and wider adoption of wearable robots that are typically worn over the human body. We categorized the challenges into mechanical layout, actuation, sensing, body interface, control, human-robot interfacing and coadaptation, and benchmarking. For each category, we discuss specific challenges and the rationale for why solving them is important, followed by an overview of relevant recent works. We conclude with an opinion that summarizes possible solutions that could contribute to the wider adoption of wearable robots.

A User-Driven Approach to Prosthetic Upper Limb Development in Korea

Despite recent significant advances in technology and medicine, the number of patients who undergo amputation of body parts for various reasons continues to increase. Assistive devices such as prosthetic arms can enable limited activities in upper limb amputees and improve their quality of life. This study aims to help in the development of user-centered prosthetics by identifying user requirements and key considerations during selection of prosthetics. This study conducted a questionnaire survey after obtaining prior consent for persons with disabilities with upper limb amputation who visited orthosis companies, rehabilitation centers for the disabled, veteran’s hospitals, and labor welfare corporations. A modified questionnaire was conducted to upper limb prosthetic users and results were analysed using descriptive statistics and t-test. Results of the study showed that the main reasons for discontinuing the use of prosthetics were discomfort (discomfort in wear, weight, and difficulty of detachment) and complaints regarding design and function. Regardless of the prosthesis type, the color and design of the prosthesis were key considerations in prosthesis choices. Respondents indicated that they needed various prostheses designed according to the purpose and situation, such as for sports like golf and cycling as well as everyday use. Most of the respondents answered that buttoning shirts, tying knots, and using chopsticks were challenging or impossible to do on their own. Based on the results of this study, the quality of life of upper limb amputees can be improved if a prosthetic arm with various functions that can satisfy both the user’s needs and wants is developed.