Representing high-dimensional data to intelligent prostheses and other wearable assistive robots: A first comparison of tile coding and selective Kanerva coding

Machine-learned Adaptive Switching in Voluntary Lower-limb Exoskeleton Control: Preliminary Results

Lower-limb exoskeletons utilize fixed control strategies and are not adaptable to user's intention. To this end, the goal of this study was to investigate the potential of using temporal-difference learning and general value functions for predicting the next possible walking mode that will be selected by users wearing exoskeletons in order to reduce the effort and cognitive load while switching between different modes of walking. Experiments were performed with a user wearing the Indego exoskeleton and given the authority to switch between five walking modes that were different in terms of speed and turn direction. The user's switching preferences were learned and predicted from device-centric and room-centric measurements by considering similarities in the movements being performed. A switching list was updated to show the most probable future next modes to be selected by the user. In contrast to other approaches that either can only predict a single time-step or require intensive offline training, this work used a computationally inexpensive method for learning and has the potential of providing temporally extended sets of predictions in real-time. Comparing the number of required manual switches between the machine-learned switching list and the best possible static lists showed an average decrease of 42.44% in the required switches for the machine-learned adaptive strategy. These promising results will facilitate the path for real-time application of this technique.

Pavlovian control of intraspinal microstimulation to produce over-ground walking

OBJECTIVE

Neuromodulation technologies are increasingly used for improving function after neural injury. To achieve a symbiotic relationship between device and user, the device must augment remaining function, and independently adapt to day-to-day changes in function. The goal of this study was to develop predictive control strategies to produce over-ground walking in a model of hemisection spinal cord injury (SCI) using intraspinal microstimulation (ISMS).

APPROACH

Eight cats were anaesthetized and placed in a sling over a walkway. The residual function of a hemisection SCI was mimicked by manually moving one hind-limb through the walking cycle. ISMS targeted motor networks in the lumbosacral enlargement to activate muscles in the other, presumably 'paralyzed' limb, using low levels of current (<130 μA). Four people took turns to move the 'intact' limb, generating four different walking styles. Two control strategies, which used ground reaction force and angular velocity information about the manually moved 'intact' limb to control the timing of the transitions of the 'paralyzed' limb through the step cycle, were compared. The first strategy used thresholds on the raw sensor values to initiate transitions. The second strategy used reinforcement learning and Pavlovian control to learn predictions about the sensor values. Thresholds on the predictions were then used to initiate transitions.

MAIN RESULTS

Both control strategies were able to produce alternating, over-ground walking. Transitions based on raw sensor values required manual tuning of thresholds for each person to produce walking, whereas Pavlovian control did not. Learning occurred quickly during walking: predictions of the sensor signals were learned rapidly, initiating correct transitions after ≤4 steps. Pavlovian control was resilient to different walking styles and different cats, and recovered from induced mistakes during walking.

SIGNIFICANCE

This work demonstrates, for the first time, that Pavlovian control can augment remaining function and facilitate personalized walking with minimal tuning requirements.

Examining the Use of Temporal-Difference Incremental Delta-Bar-Delta for Real-World Predictive Knowledge Architectures

Predictions and predictive knowledge have seen recent success in improving not only robot control but also other applications ranging from industrial process control to rehabilitation. A property that makes these predictive approaches well-suited for robotics is that they can be learned online and incrementally through interaction with the environment. However, a remaining challenge for many prediction-learning approaches is an appropriate choice of prediction-learning parameters, especially parameters that control the magnitude of a learning machine's updates to its predictions (the learning rates or step sizes). Typically, these parameters are chosen based on an extensive parameter search—an approach that neither scales well nor is well-suited for tasks that require changing step sizes due to non-stationarity. To begin to address this challenge, we examine the use of online step-size adaptation using the Modular Prosthetic Limb: a sensor-rich robotic arm intended for use by persons with amputations. Our method of choice, Temporal-Difference Incremental Delta-Bar-Delta (TIDBD), learns and adapts step sizes on a feature level; importantly, TIDBD allows step-size tuning and representation learning to occur at the same time. As a first contribution, we show that TIDBD is a practical alternative for classic Temporal-Difference (TD) learning via an extensive parameter search. Both approaches perform comparably in terms of predicting future aspects of a robotic data stream, but TD only achieves comparable performance with a carefully hand-tuned learning rate, while TIDBD uses a robust meta-parameter and tunes its own learning rates. Secondly, our results show that for this particular application TIDBD allows the system to automatically detect patterns characteristic of sensor failures common to a number of robotic applications. As a third contribution, we investigate the sensitivity of classic TD and TIDBD with respect to the initial step-size values on our robotic data set, reaffirming the robustness of TIDBD as shown in previous papers. Together, these results promise to improve the ability of robotic devices to learn from interactions with their environments in a robust way, providing key capabilities for autonomous agents and robots.

Learning from demonstration: Teaching a myoelectric prosthesis with an intact limb via reinforcement learning

Prosthetic arms should restore and extend the capabilities of someone with an amputation. They should move naturally and be able to perform elegant, coordinated movements that approximate those of a biological arm. Despite these objectives, the control of modern-day prostheses is often nonintuitive and taxing. Existing devices and control approaches do not yet give users the ability to effect highly synergistic movements during their daily-life control of a prosthetic device. As a step towards improving the control of prosthetic arms and hands, we introduce an intuitive approach to training a prosthetic control system that helps a user achieve hard-to-engineer control behaviours. Specifically, we present an actor-critic reinforcement learning method that for the first time promises to allow someone with an amputation to use their non-amputated arm to teach their prosthetic arm how to move through a wide range of coordinated motions and grasp patterns. We evaluate our method during the myoelectric control of a multi-joint robot arm by non-amputee users, and demonstrate that by using our approach a user can train their arm to perform simultaneous gestures and movements in all three degrees of freedom in the robot's hand and wrist based only on information sampled from the robot and the user's above-elbow myoelectric signals. Our results indicate that this learning-from-demonstration paradigm may be well suited to use by both patients and clinicians with minimal technical knowledge, as it allows a user to personalize the control of his or her prosthesis without having to know the underlying mechanics of the prosthetic limb. These preliminary results also suggest that our approach may extend in a straightforward way to next-generation prostheses with precise finger and wrist control, such that these devices may someday allow users to perform fluid and intuitive movements like playing the piano, catching a ball, and comfortably shaking hands.