Vibrotactile sensory substitution for electromyographic control of object manipulation

Investigating the Effects of Intensity and Frequency on Vibrotactile Spatial Acuity

Vibrotactile devices are commonly used in applications for sensory substitution or to provide feedback in virtual reality. An important aspect of vibrotactile perception is spatial acuity, which determines the resolutions of vibrotactile displays on the skin. However, the complex vibration characteristics of vibrotactile actuators make it challenging for researchers to reference and compare previous study results. This is because the effects of typical characteristics, such as intensity and frequency, are not well understood. In this study, we investigated the effects of intensity and frequency on vibrotactile spatial acuity. Using Linear Resonant Actuators (LRAs), we conducted relative point localization experiments to measure spatial acuity under different conditions. In the first experiment, we found that intensity had a significant effect on spatial acuity, with higher intensity leading to better acuity. In the second experiment, using a carefully designed intensity calibration procedure, we did not find a significant effect of frequency on spatial acuity. These findings provide a better understanding of vibrotactile spatial acuity, allow for comparisons to previous research, and provide insights into the design of future tactile devices.

The future of sensory substitution, addition, and expansion via haptic devices

Haptic devices use the sense of touch to transmit information to the nervous system. As an example, a sound-to-touch device processes auditory information and sends it to the brain via patterns of vibration on the skin for people who have lost hearing. We here summarize the current directions of such research and draw upon examples in industry and academia. Such devices can be used for sensory substitution (replacing a lost sense, such as hearing or vision), sensory expansion (widening an existing sensory experience, such as detecting electromagnetic radiation outside the visible light spectrum), and sensory addition (providing a novel sense, such as magnetoreception). We review the relevant literature, the current status, and possible directions for the future of sensory manipulation using non-invasive haptic devices.

The Effects of Prolonged Vibrotactile EMG-Based Biofeedback on Ankle Joint Range of Motion During Gait in Children with Spastic Cerebral Palsy: A Case Series

AIMS

The objective of this case series was to examine the feasibility of vibrotactile EMG-based biofeedback (BF) as a home-based intervention tool to enhance sensory information during everyday motor activities and to explore its effectiveness to induce changes in active ankle range of motion during gait in children with spastic cerebral palsy (CP).

METHODS

Ten children ages 6 to 13 years with spastic CP were recruited. Participants wore two EMG-based vibro-tactile BF devices for at least 4 hours per day for 1-month on the ankle and knee joints muscles. The device computed the amplitude of the EMG signal of the target muscle and actuated a silent vibration motor proportional to the magnitude of the EMG.

RESULTS

Our results demonstrated the feasibility of the augmented sensory information of muscle activity to induce changes of the active ankle range of motion during gait for 6 children with an increase ranging from 8.9 to 51.6% compared to a one-month period without treatment.

CONCLUSIONS

Preliminary findings of this case series demonstrate the feasibility of vibrotactile EMG-based BF and suggest potential effectiveness to increase active ankle range of motion, therefore serving as a promising therapeutic tool to improve gait in children with spastic CP.

Perception and Performance of Electrical Stimulation for Proprioception

Proprioception, yielding awareness of the body's position and motion in space, is typically lacking in prostheses and supernumerary limbs. Electrical stimulation is one technique that may provide these devices with proprioception. This paper first investigates how the modalities of electrotactile cues, such as frequency and intensity, are perceived. Using the results, we designed and compared several comfortable and perceptible feedback mappings for spatial cues. Two experiments were conducted using a 16-electrode bracelet worn above the elbow to provide electrical stimuli. We found that subjects could localize the stimulating electrode with a precision of ±1 electrode (110 mm) in all feedback conditions. Moreover, within the range of pulse intensities perceived as comfortable, the participants' performance was more sensitive to changes in frequency than in intensity. The highest performance was obtained for the condition which increased both intensity and frequency with radial distance. These results suggest that electrical stimulation can be used for artificial proprioceptive feedback, which can ensure a comfortable and intuitive interaction and provides high spatial accuracy.

Static and dynamic proprioceptive recognition through vibrotactile stimulation

Objective.Proprioceptive information provides individuals with a sense of our limb's static position and dynamic movement. Impaired or a lack of such feedback can diminish our ability to perform dexterous motions with our biological limbs or assistive devices. Here we seek to determine whether both static and dynamic components of proprioception can be recognized using variation of the spatial and temporal components of vibrotactile feedback.Approach.An array of five vibrotactors was placed on the forearm of each subject. Each tactor was encoded to represent one of the five forearm postures. Vibratory stimulus was elicited to convey the static position and movement of the forearm. Four experimental blocks were performed to test each subject's recognition of a forearm's simulated static position, rotational amplitude, rotational amplitude and direction, and rotational speed.Main results.Our results showed that the subjects were able to perform proprioceptive recognition based on the delivered vibrotactile information. Specifically, rotational amplitude recognition resulted in the highest level of accuracy (99.0%), while the recognition accuracy of the static position and the rotational amplitude-direction was the lowest (91.7% and 90.8%, respectively). Nevertheless, all proprioceptive properties were perceived with >90% accuracy, indicating that the implemented vibrotactile encoding scheme could effectively provide proprioceptive information to the users.Significance.The outcomes suggest that information pertaining to static and dynamic aspects of proprioception can be accurately delivered using an array of vibrotactors. This feedback approach could be used to potentially evaluate the sensorimotor integration processes during human-machine interactions, and to improve sensory feedback in clinical populations with somatosensory impairments.

Sonomyography Combined with Vibrotactile Feedback Enables Precise Target Acquisition Without Visual Feedback

Upper limb prosthesis users currently lack haptic feedback from their terminal devices, which significantly limits their ability to meaningfully interact with their environment. Users therefore rely heavily on visual feedback when using terminal devices. Previously, it has been shown that force-related feedback from an end-effector or virtual environment can help the user minimize errors and improve performance. Currently, myoelectric control systems enable the user to control the velocity of terminal devices. We have developed a novel control method using ultrasound sensing, called sonomyography, that enables position control based on mechanical deformation of muscles. In this paper, we investigated whether the proprioceptive feedback from muscle deformation combined with vibrotactile haptic feedback can minimize the need for visual feedback. Able bodied subjects used sonomyography to control a virtual cursor, and performed a target acquisition task. The effect of visual and haptic feedback on performance of a target acquisition task was systematically tested. We found that subjects made large errors when they tried to reacquire a target without visual feedback, but in the presence of real-time haptic feedback, the precision of the target position improved, and were similar to when visual feedback was used for target acquisition. This result has implications for improving the performance of prosthetic control systems.

Long-Term Home-Use of Sensory-Motor-Integrated Bidirectional Bionic Prosthetic Arms Promotes Functional, Perceptual, and Cognitive Changes

Cutaneous sensation is vital to controlling our hands and upper limbs. It helps close the motor control loop by informing adjustments of grasping forces during object manipulations and provides much of the information the brain requires to perceive our limbs as a part of our bodies. This sensory information is absent to upper-limb prosthesis users. Although robotic prostheses are becoming increasingly sophisticated, the absence of feedback imposes a reliance on open-loop control and limits the functional potential as an integrated part of the body. Experimental systems to restore physiologically relevant sensory information to prosthesis users are beginning to emerge. However, the impact of their long-term use on functional abilities, body image, and neural adaptation processes remains unclear. Understanding these effects is essential to transition sensate prostheses from sophisticated assistive tools to integrated replacement limbs. We recruited three participants with high-level upper-limb amputation who previously received targeted reinnervation surgery. Each participant was fit with a neural-machine-interface prosthesis that allowed participants to operate their device by thinking about moving their missing limb. Additionally, we fit a sensory feedback system that allowed participants to experience touch to the prosthesis as touch on their missing limb. All three participants performed a long-term take-home trial. Two participants used their neural-machine-interface systems with touch feedback and one control participant used his prescribed, insensate prosthesis. A series of functional outcome metrics and psychophysical evaluations were performed using sensate neural-machine-interface prostheses before and after the take-home period to capture changes in functional abilities, limb embodiment, and neural adaptation. Our results demonstrated that the relationship between users and sensate neural-machine-interface prostheses is dynamic and changes with long-term use. The presence of touch sensation had a near-immediate impact on how the users operated their prostheses. In the multiple independent measures of users’ functional abilities employed, we observed a spectrum of performance changes following long-term use. Furthermore, after the take-home period, participants more appropriately integrated their prostheses into their body images and psychophysical tests provided strong evidence that neural and cortical adaptation occurred.

Wearable Dual-Frequency Vibrotactile System for Restoring Force and Stiffness Perception

Recently, there has been substantial progress in the mechatronic design and myoelectric control of active prostheses. However, a significant unmet need is the lack of sensory feedback in commercial prostheses for upper-limb amputees. The lack of sensory perception impacts on the control performance and embodiment, determining relatively high rejection rates. Previous research has been conducted to evaluate various non-invasive substitutional sensory channels, mainly to regenerate haptic perception. However, providing sensory channels for stiffness perception has been much less explored. In this short paper, we propose a non-invasive wearable sensory armband, named vibrotactile frequency modulation (V-FM) system. Using the V-FM, we implement a closed-loop myocontrol system with force and stiffness perception. A user study was conducted based on a repetitive two-forced alternative choice discrimination test in six able-bodied participants. The study was designed according to the method of constant stimuli. Results showed that using the V-FM armband, the participants recovered sensation comparable (in terms of difference threshold) to the natural stiffness perception. This demonstrated the potential of the proposed V-FM armband in restoring haptic and stiffness perception non-invasively.

Inter-Limb Transfer of Grasp Force Perception With Closed-Loop Hand Prosthesis

Sensory feedback of grasp forces provides important information about physical interactions between the hand and objects, enabling both reactive and anticipatory neural control mechanisms. The numerous studies have shown artificial sensory feedback of various forms improves force control during grasping tasks by prosthetic hand users through a closed-feedback loop. However, little is known about how perceptual information is transferred between an intact limb and a closed-loop prosthetic limb, and the extent to which training inter-limb transfer may improve myoelectric prosthetic control. We addressed these gaps by using a contralateral force-matching task in which able-bodied participants were asked to generate grasp forces with their native hand, and then match it using the contralateral hand or a soft-synergy prosthetic hand worn on the contralateral arm that was coupled with a mechanotactile feedback device. We found that absolute matching error and matching time were greater when using the prosthetic system than the native hand. However, with contralateral specific training, subjects were able to produce similar relative matching error with the prosthetic system and the native hand, especially at the untrained force level. These findings suggest that an association can be established between the perception produced by the prosthetic limb and the contralateral intact limb, and provide novel insights about potential applications to training and design of the closed-loop prosthesis.

Effects of vibrotactile feedback and grasp interface compliance on perception and control of a sensorized myoelectric hand

Current myoelectric prosthetic limbs are limited in their ability to provide direct sensory feedback to users, which increases attentional demands and reliance on visual cues. Vibrotactile sensory substitution (VSS), which can be used to provide sensory feedback in a non-invasive manner has demonstrated some improvement in myoelectric hand control. In this work, we developed and tested two VSS configurations: one with a single burst-rate modulated actuator and another with a spatially distributed array of five coin tactors. We performed a direct comparative assessment of these two VSS configurations with able-bodied subjects to investigate sensory perception, myoelectric control of grasp force and hand aperture with a prosthesis, and the effects of interface compliance. Six subjects completed a sensory perception experiment under a stimulation only paradigm; sixteen subjects completed experiments to compare VSS performance on perception and graded myoelectric control during grasp force and hand aperture tasks; and ten subjects completed experiments to investigate the effect of mechanical compliance of the myoelectric hand on the ability to control grasp force. Results indicated that sensory perception of vibrotactile feedback was not different for the two VSS configurations in the absence of active myoelectric control, but it was better with feedback from the coin tactor array than with the single actuator during myoelectric control of grasp force. Graded myoelectric control of grasp force and hand aperture was better with feedback from the coin tactor array than with the single actuator, and myoelectric control of grasp force was improved with a compliant grasp interface. Further investigations with VSS should focus on the use of coin tactor arrays by subjects with amputation in real-world settings and on improving control of grasp force by increasing the mechanical compliance of the hand.

Improving internal model strength and performance of prosthetic hands using augmented feedback

Background

The loss of an arm presents a substantial challenge for upper limb amputees when performing activities of daily living. Myoelectric prosthetic devices partially replace lost hand functions; however, lack of sensory feedback and strong understanding of the myoelectric control system prevent prosthesis users from interacting with their environment effectively. Although most research in augmented sensory feedback has focused on real-time regulation, sensory feedback is also essential for enabling the development and correction of internal models, which in turn are used for planning movements and reacting to control variability faster than otherwise possible in the presence of sensory delays.

Methods

Our recent work has demonstrated that audio-augmented feedback can improve both performance and internal model strength for an abstract target acquisition task. Here we use this concept in controlling a robotic hand, which has inherent dynamics and variability, and apply it to a more functional grasp-and-lift task. We assessed internal model strength using psychophysical tests and used an instrumented Virtual Egg to assess performance.

Results

Results obtained from 14 able-bodied subjects show that a classifier-based controller augmented with audio feedback enabled stronger internal model (p = 0.018) and better performance (p = 0.028) than a controller without this feedback.

Conclusions

We extended our previous work and accomplished the first steps on a path towards bridging the gap between research and clinical usability of a hand prosthesis. The main goal was to assess whether the ability to decouple internal model strength and motion variability using the continuous audio-augmented feedback extended to real-world use, where the inherent mechanical variability and dynamics in the mechanisms may contribute to a more complicated interplay between internal model formation and motion variability. We concluded that benefits of using audio-augmented feedback for improving internal model strength of myoelectric controllers extend beyond a virtual target acquisition task to include control of a prosthetic hand.

Improving Fine Control of Grasping Force during Hand-Object Interactions for a Soft Synergy-Inspired Myoelectric Prosthetic Hand

The concept of postural synergies of the human hand has been shown to potentially reduce complexity in the neuromuscular control of grasping. By merging this concept with soft robotics approaches, a multi degrees of freedom soft-synergy prosthetic hand [SoftHand-Pro (SHP)] was created. The mechanical innovation of the SHP enables adaptive and robust functional grasps with simple and intuitive myoelectric control from only two surface electromyogram (sEMG) channels. However, the current myoelectric controller has very limited capability for fine control of grasp forces. We addressed this challenge by designing a hybrid-gain myoelectric controller that switches control gains based on the sensorimotor state of the SHP. This controller was tested against a conventional single-gain (SG) controller, as well as against native hand in able-bodied subjects. We used the following tasks to evaluate the performance of grasp force control: (1) pick and place objects with different size, weight, and fragility levels using power or precision grasp and (2) squeezing objects with different stiffness. Sensory feedback of the grasp forces was provided to the user through a non-invasive, mechanotactile haptic feedback device mounted on the upper arm. We demonstrated that the novel hybrid controller enabled superior task completion speed and fine force control over SG controller in object pick-and-place tasks. We also found that the performance of the hybrid controller qualitatively agrees with the performance of native human hands.

Elucidating Sensorimotor Control Principles with Myoelectric Musculoskeletal Models

There is an old saying that you must walk a mile in someone's shoes to truly understand them. This mini-review will synthesize and discuss recent research that attempts to make humans "walk a mile" in an artificial musculoskeletal system to gain insight into the principles governing human movement control. In this approach, electromyography (EMG) is used to sample human motor commands; these commands serve as inputs to mathematical models of muscular dynamics, which in turn act on a model of skeletal dynamics to produce a simulated motor action in real-time (i.e., the model's state is updated fast enough produce smooth motion without noticeable transitions; Manal et al., 2002). In this mini-review, these are termed myoelectric musculoskeletal models (MMMs). After a brief overview of typical MMM design and operation principles, the review will highlight how MMMs have been used for understanding human sensorimotor control and learning by evoking apparent alterations in a user's biomechanics, neural control, and sensory feedback experiences.

Selectivity and Longevity of Peripheral-Nerve and Machine Interfaces: A Review

For those individuals with upper-extremity amputation, a daily normal living activity is no longer possible or it requires additional effort and time. With the aim of restoring their sensory and motor functions, theoretical and technological investigations have been carried out in the field of neuroprosthetic systems. For transmission of sensory feedback, several interfacing modalities including indirect (non-invasive), direct-to-peripheral-nerve (invasive), and cortical stimulation have been applied. Peripheral nerve interfaces demonstrate an edge over the cortical interfaces due to the sensitivity in attaining cortical brain signals. The peripheral nerve interfaces are highly dependent on interface designs and are required to be biocompatible with the nerves to achieve prolonged stability and longevity. Another criterion is the selection of nerves that allows minimal invasiveness and damages as well as high selectivity for a large number of nerve fascicles. In this paper, we review the nerve-machine interface modalities noted above with more focus on peripheral nerve interfaces, which are responsible for provision of sensory feedback. The invasive interfaces for recording and stimulation of electro-neurographic signals include intra-fascicular, regenerative-type interfaces that provide multiple contact channels to a group of axons inside the nerve and the extra-neural-cuff-type interfaces that enable interaction with many axons around the periphery of the nerve. Section Current Prosthetic Technology summarizes the advancements made to date in the field of neuroprosthetics toward the achievement of a bidirectional nerve-machine interface with more focus on sensory feedback. In the Discussion section, the authors propose a hybrid interface technique for achieving better selectivity and long-term stability using the available nerve interfacing techniques.

What Is New in Trauma-Related Amputations

Traumatic and trauma-related amputations represent unfortunate sequelae of severe injury, but should not be viewed as a treatment failure and may represent the best reconstructive option for some patients. Lessons from recent military conflicts have guided the evolution of modern surgical techniques and rehabilitation management of this challenging patient population, and treatment at a specialty center may improve patient outcomes. Despite appropriate management, however, surgical complications remain common and revision surgery is often necessary. Bridge synostosis procedures remain controversial, and clinical equipoise remains regarding their functional benefits. Based on European experience over the last 3 decades, osseointegration has evolved into a viable clinical alternative for patients unable to achieve acceptable function using conventional sockets, and several devices are being developed or tested in the United States. Targeted muscle reinnervation and advanced pattern recognition may dramatically improve the functional potential of many upper extremity amputees, and the procedure may also relieve neuroma-related pain. Furthermore, exciting new research may eventually facilitate haptic feedback and restore useful sensation for amputees. Natural disasters and global terrorism events, in addition to conventional trauma resulting in limb loss, make a working knowledge of current amputation surgical techniques essential to the practicing orthopaedic trauma surgeon.

A cross-modal feedback scheme for control of prosthetic grasp strength

Introduction

Given the lack of haptic feedback inherent in prosthetic devices, a natural and adaptable feedback scheme must be implemented. While multimodal feedback has proven successful in aiding dexterous performance, it can be mentally tasking on the individual. Conversely, cross-modal schemes relying on sensory substitution have proven to be equally effective in aiding task performance without cognitively burdening the user to the same degree.

Objectives

This experiment investigated the effectiveness of the cross-modal feedback scheme through using audio feedback to represent prosthetic grasping strength during dynamic control of a prosthetic hand.

Methods

A total of five individuals participated in two sets of experiments (four subjects in the first, one subject in the second). Participants were asked to control the grasping strength exerted by a prosthetic hand while using real-time audio feedback in order to reach up to three different levels of force within a trial set.

Results

The cross-modal feedback scheme successfully provided users with the robust ability to modulate grasping strength in real-time using only audio feedback.

Conclusion

Audio feedback effectively conveys haptic information to the user of a prosthetic hand. Retention of the training knowledge is evident and can be generalized to perform new (i.e. untrained) tasks.

Integrated and flexible multichannel interface for electrotactile stimulation

OBJECTIVE

The aim of the present work was to develop and test a flexible electrotactile stimulation system to provide real-time feedback to the prosthesis user. The system requirements were to accommodate the capabilities of advanced multi-DOF myoelectric hand prostheses and transmit the feedback variables (proprioception and force) using intuitive coding, with high resolution and after minimal training.

APPROACH

We developed a fully-programmable and integrated electrotactile interface supporting time and space distributed stimulation over custom designed flexible array electrodes. The system implements low-level access to individual stimulation channels as well as a set of high-level mapping functions translating the state of a multi-DoF prosthesis (aperture, grasping force, wrist rotation) into a set of predefined dynamic stimulation profiles. The system was evaluated using discrimination tests employing spatial and frequency coding (10 able-bodied subjects) and dynamic patterns (10 able-bodied and 6 amputee subjects). The outcome measure was the success rate (SR) in discrimination.

MAIN RESULTS

The more practical electrode with the common anode configuration performed similarly to the more usual concentric arrangement. The subjects could discriminate six spatial and four frequency levels with SR >90% after a few minutes of training, whereas the performance significantly deteriorated for more levels. The dynamic patterns were intuitive for the subjects, although amputees showed lower SR than able-bodied individuals (86% ± 10% versus 99% ± 3%).

SIGNIFICANCE

The tests demonstrated that the system was easy to setup and apply. The design and resolution of the multipad electrode was evaluated. Importantly, the novel dynamic patterns, which were successfully tested, can be superimposed to transmit multiple feedback variables intuitively and simultaneously. This is especially relevant for closing the loop in modern multifunction prostheses. Therefore, the proposed system is convenient for practical applications and can be used to implement sensory perception training and/or closed-loop control of myoelectric prostheses, providing grasping force and proprioceptive feedback.

An Empirical Evaluation of Force Feedback in Body-Powered Prostheses

Myoelectric prostheses have many advantages over body-powered prostheses, yet the absence of sensory feedback in myoelectric devices is one reason body-powered devices are often preferred by amputees. While considerable progress has been made in the mechanical design and control of myoelectric prostheses, research on haptic feedback has not had a similar impact. In this study, we seek to develop a fundamental understanding of the utility of force feedback and vision in the functional operation of a body-powered upper-limb prosthesis. Using a custom body-powered prosthesis in which force feedback can be conditionally removed, we asked N=10 non-amputee participants to identify objects based on stiffness in four separate conditions with and without visual and/or force feedback. Results indicate that the combination of visual and force feedback allows for the best accuracy, followed by force feedback only, then visual feedback only. In addition, combining force feedback with visual feedback does not significantly affect identification timing compared to visual feedback alone. These findings suggest that consideration should be given to the development of force feedback displays for myoelectric prostheses that function like a Bowden cable, coupling the amputee's control input to the resulting feedback.

Sensory feedback by peripheral nerve stimulation improves task performance in individuals with upper limb loss using a myoelectric prosthesis

OBJECTIVE

Tactile feedback is critical to grip and object manipulation. Its absence results in reliance on visual and auditory cues. Our objective was to assess the effect of sensory feedback on task performance in individuals with limb loss.

APPROACH

Stimulation of the peripheral nerves using implanted cuff electrodes provided two subjects with sensory feedback with intensity proportional to forces on the thumb, index, and middle fingers of their prosthetic hand during object manipulation. Both subjects perceived the sensation on their phantom hand at locations corresponding to the locations of the forces on the prosthetic hand. A bend sensor measured prosthetic hand span. Hand span modulated the intensity of sensory feedback perceived on the thenar eminence for subject 1 and the middle finger for subject 2. We performed three functional tests with the blindfolded subjects. First, the subject tried to determine whether or not a wooden block had been placed in his prosthetic hand. Second, the subject had to locate and remove magnetic blocks from a metal table. Third, the subject performed the Southampton Hand Assessment Procedure (SHAP). We also measured the subject's sense of embodiment with a survey and his self-confidence.

MAIN RESULTS

Blindfolded performance with sensory feedback was similar to sighted performance in the wooden block and magnetic block tasks. Performance on the SHAP, a measure of hand mechanical function and control, was similar with and without sensory feedback. An embodiment survey showed an improved sense of integration of the prosthesis in self body image with sensory feedback.

SIGNIFICANCE

Sensory feedback by peripheral nerve stimulation improved object discrimination and manipulation, embodiment, and confidence. With both forms of feedback, the blindfolded subjects tended toward results obtained with visual feedback.

Neural interfaces for somatosensory feedback: bringing life to a prosthesis

PURPOSE OF REVIEW

When an individual loses a limb, he/she loses touch with the world and with the people around him/her. Somatosensation is critical to the feeling of connection and control of one's own body. Decades of attempts to replace lost somatosensation by sensory substitutions have been ineffective outside of the laboratory. This review discusses important recent results demonstrating chronic somatosensory restoration through direct peripheral nerve stimulation.

RECENT FINDINGS

Stimulation of peripheral nerves results in somatosensory perception on the phantom limb. Sensations are localized to several independent and functionally relevant locations, such as the fingertips, thenar eminence, ulnar border and dorsal surface. Patterns in stimulation intensity change the perception experience by the user, opening new dimensions on neuromodulation.

SUMMARY

Neural interfaces with sophisticated stimulation paradigms create a user's perception of his/her hand to touch and manipulate objects. The pattern of intensity and frequency of stimulation is critical to the quality and intensity of perceived sensation. Restoring feeling has allowed the individuals to, 'feel [my] hand for the first time since the accident,' and 'feel [my] wife touch my hand'. Individuals using a prosthetic hand with sensation can pull cherries and grapes from the stem, open water bottles and move objects without destroying these objects - all while audio and visually deprived. After regaining sensation, phantom pain is eliminated in individuals that had frequent, sometimes debilitating, pain following limb loss. With over 5 subject-years of experience, this work is leading the evolution of a new era in prostheses. Somatosensory prosthetics as a standard procedure to augment and restore somatosensation are now within our reach.

An exploration of grip force regulation with a low-impedance myoelectric prosthesis featuring referred haptic feedback

Background

Haptic display technologies are well suited to relay proprioceptive, force, and contact cues from a prosthetic terminal device back to the residual limb and thereby reduce reliance on visual feedback. The ease with which an amputee interprets these haptic cues, however, likely depends on whether their dynamic signal behavior corresponds to expected behaviors—behaviors consonant with a natural limb coupled to the environment. A highly geared motor in a terminal device along with the associated high back-drive impedance influences dynamic interactions with the environment, creating effects not encountered with a natural limb. Here we explore grasp and lift performance with a backdrivable (low backdrive impedance) terminal device placed under proportional myoelectric position control that features referred haptic feedback.

Methods

We fabricated a back-drivable terminal device that could be used by amputees and non-amputees alike and drove aperture (or grip force, when a stiff object was in its grasp) in proportion to a myoelectric signal drawn from a single muscle site in the forearm. In randomly ordered trials, we assessed the performance of N=10 participants (7 non-amputee, 3 amputee) attempting to grasp and lift an object using the terminal device under three feedback conditions (no feedback, vibrotactile feedback, and joint torque feedback), and two object weights that were indiscernible by vision.

Results

Both non-amputee and amputee participants scaled their grip force according to the object weight. Our results showed only minor differences in grip force, grip/load force coordination, and slip as a function of sensory feedback condition, though the grip force at the point of lift-off for the heavier object was significantly greater for amputee participants in the presence of joint torque feedback. An examination of grip/load force phase plots revealed that our amputee participants used larger safety margins and demonstrated less coordination than our non-amputee participants.

Conclusions

Our results suggest that a backdrivable terminal device may hold advantages over non-backdrivable devices by allowing grip/load force coordination consistent with behaviors observed in the natural limb. Likewise, the inconclusive effect of referred haptic feedback on grasp and lift performance suggests the need for additional testing that includes adequate training for participants.

Object discrimination using optimized multi-frequency auditory cross-modal haptic feedback

As the field of brain-machine interfaces and neuro-prosthetics continues to grow, there is a high need for sensor and actuation mechanisms that can provide haptic feedback to the user. Current technologies employ expensive, invasive and often inefficient force feedback methods, resulting in an unrealistic solution for individuals who rely on these devices. This paper responds through the development, integration and analysis of a novel feedback architecture where haptic information during the neural control of a prosthetic hand is perceived through multi-frequency auditory signals. Through representing force magnitude with volume and force location with frequency, the feedback architecture can translate the haptic experiences of a robotic end effector into the alternative sensory modality of sound. Previous research with the proposed cross-modal feedback method confirmed its learnability, so the current work aimed to investigate which frequency map (i.e. frequency-specific locations on the hand) is optimal in helping users distinguish between hand-held objects and tasks associated with them. After short use with the cross-modal feedback during the electromyographic (EMG) control of a prosthetic hand, testing results show that users are able to use audial feedback alone to discriminate between everyday objects. While users showed adaptation to three different frequency maps, the simplest map containing only two frequencies was found to be the most useful in discriminating between objects. This outcome provides support for the feasibility and practicality of the cross-modal feedback method during the neural control of prosthetics.

Effects of Different Tactile Feedback on Myoelectric Closed-Loop Control for Grasping Based on Electrotactile Stimulation

Closed-loop control is important for amputees to manipulate myoelectric prostheses intuitively and dexterously. Tactile feedback can help amputees improve myoelectric control performance for grasping objects. To investigate the effects of different tactile feedback, we performed experiments on six amputees and six able-bodied subjects via electrotactile stimulation. Using a virtual environment, six kinds of objects with different weights and stiffnesses were used for grasping tasks. Five feedback conditions (no feedback, pressure feedback, slip feedback, pressure + slip feedback, and vision feedback) were considered. Nine evaluation indexes and three control objectives (rapidity, economy, and stability) were proposed. Under the five feedback conditions, our study investigated four issues: 1) three types of grasping-related failures; 2) four types of grasping-related time measures; 3) average grasping force; 4) standard deviation of the grasping force. Results indicate that: 1) slip feedback is better than pressure feedback; 2) pressure + slip feedback can improve grasping rapidity; 3) slip feedback significantly contributes to grasping economy and stability; and 4) pressure + slip feedback can perform as well as vision feedback.

Haptic wearables as sensory replacement, sensory augmentation and trainer - a review

Sensory impairments decrease quality of life and can slow or hinder rehabilitation. Small, computationally powerful electronics have enabled the recent development of wearable systems aimed to improve function for individuals with sensory impairments. The purpose of this review is to synthesize current haptic wearable research for clinical applications involving sensory impairments. We define haptic wearables as untethered, ungrounded body worn devices that interact with skin directly or through clothing and can be used in natural environments outside a laboratory. Results of this review are categorized by degree of sensory impairment. Total impairment, such as in an amputee, blind, or deaf individual, involves haptics acting as sensory replacement; partial impairment, as is common in rehabilitation, involves haptics as sensory augmentation; and no impairment involves haptics as trainer. This review found that wearable haptic devices improved function for a variety of clinical applications including: rehabilitation, prosthetics, vestibular loss, osteoarthritis, vision loss and hearing loss. Future haptic wearables development should focus on clinical needs, intuitive and multimodal haptic displays, low energy demands, and biomechanical compliance for long-term usage.

Somatotopical feedback versus non-somatotopical feedback for phantom digit sensation on amputees using electrotactile stimulation

Background

Transcutaneous electrical stimulation can provide amputees with tactile feedback for better manipulating an advanced prosthesis. In general, there are two ways to transfer the stimulus to the skin: somatotopical feedback (SF) that stimulates the phantom digit somatotopy on the stump and non-somatotopical feedback (NF) that stimulates other positions on the human body.

Methods

To investigate the difference between SF and NF, electrotactile experiments were conducted on seven amputees. Electrical stimulation was applied via a complete phantom map to the residual limb (SF) and to the upper arm (NF) separately. The behavior results of discrimination accuracy and response time were used to examine: 1) performance differences between SF and NF for discriminating position, type and strength of tactile feedback; 2) performance differences between SF and NF for one channel (1C), three channels (3C), and five channels (5C). NASA-TLX standardized testing was used to determine differences in mental workload between SF and NF.

Results

The grand-averaged discrimination accuracy for SF was 6% higher than NF, and the average response time for SF was 600 ms faster than NF. SF is better than NF for position, type, strength, and the overall modality regarding both accuracy and response time except for 1C modality (p<0.001). Among the six modalities of stimulation channels, performance of 1C/SF was the best, which was similar to that of 1C/NF and 3C/SF; performance of 3C/NF was similar to that of 5C/SF; performance of 5C/NF was the worst. NASA-TLX scores indicated that mental workload increased as the number of stimulation channels increased.

Conclusions

We quantified the difference between SF and NF, and the influence of different number of stimulation channels. SF was better than NF in general, but the practical issues such as the limited area of stumps could constrain the use of SF. We found that more channels increased the amount and richness of information to the amputee while fewer channels resulted in higher performance, and thus the 3C/SF modality was a good compromise. Based on this study, we provide possible solutions to the practical problems involving the implementation of tactile feedback for amputees. These results are expected to promote the application of SF and NF tactile feedback for amputees in the future.

Effects of kinematic vibrotactile feedback on learning to control a virtual prosthetic arm

Background

After a limb is lost a prosthesis can restore function. For maximum utility, prosthetic limbs should accept movement commands and provide force and motion feedback, which can be conveyed with vibrotactile feedback (VIBF). While prior studies have shown that force-based VIBF benefits control, the merits of motion-based VIBF are unclear. Our goal was to clarify the effectiveness of position- and velocity-based VIBF for prosthetic arm control.

Methods

Healthy adults with normal limb function practiced a goal-directed task with a virtual myoelectric prosthetic arm. A linear resonant actuator on the wrist provided VIBF. Two groups with nine subjects each received amplitude modulated VIBF in addition to visual feedback while practicing the task. In one group, the VIBF was proportional to the virtual arm’s position, and in the other group, velocity. A control group of nine subjects received only visual feedback. Subjects practiced for 240 trials, followed by 180 trials with feedback manipulations for the VIBF groups. Performance was characterized by end-point error, movement time, and a composite skill measure that combined these quantities. A second experiment with a new group of five subjects assessed discrimination capabilities between different position- and velocity-based VIBF profiles.

Results

With practice all groups improved their skill in controlling the virtual prosthetic arm. Subjects who received additional position- and velocity-based VIBF learned at the same rate as the control group, who received only visual feedback (learning rate time constant: about 40 trials). When visual feedback was subsequently removed leaving only VIBF, performance was no better than with no feedback at all. When VIBF was removed leaving only visual feedback, about half of the participants performed better, instead of worse. The VIBF discrimination tests showed that subjects could detect virtual arm angular position and velocity differences of about 5 deg and 20 deg/s, respectively.

Conclusions

Kinematic VIBF did not increase the rate of skill acquisition or improve performance when controlling a virtual myoelectric prosthetic arm, whether provided in isolation or coupled with visual feedback. VIBF had a deleterious effect on performance for some individuals, who may have had difficulty integrating kinematic VIBF information into their control strategies.

Assessing vibrotactile feedback strategies by controlling a cursor with unstable dynamics

Brain computer interface (BCI) control predominately uses visual feedback. Real arm movements, however, are controlled under a diversity of feedback mechanisms. The lack of additional BCI feedback modalities forces users to maintain visual contact while performing tasks. Such stringent requirements result in poor BCI control during tasks that inherently lack visual feedback, such as grasping, or when visual attention is diverted. Using a modified version of the Critical Tracking Task which we call the Critical Stability Task (CST), we tested the ability of 9 human subjects to control an unstable system using either free arm movements or pinch force. The subjects were provided either visual feedback, 'proportional' vibrotactile feedback, or 'on-off' vibrotactile feedback about the state of the unstable system. We increased the difficulty of the control task by making the virtual system more unstable. We judged the effectiveness of a particular form of feedback as the maximal instability the system could reach before the subject lost control of it. We found three main results. First, subjects can use solely vibrotactile feedback to control an unstable system, although control was better using visual feedback. Second, 'proportional' vibrotactile feedback provided slightly better control than 'on-off' vibrotactile feedback. Third, there was large intra-subject variability in terms of the most effective input and feedback methods. This highlights the need to tailor the input and feedback methods to the subject when a high degree of control is desired. Our new task can provide a complement to traditional center-out paradigms to help boost the real-world relevance of BCI research in the lab.

Internal models of upper limb prosthesis users when grasping and lifting a fragile object with their prosthetic limb

Internal models allow unimpaired individuals to appropriately scale grip force when grasping and lifting familiar objects. In prosthesis users, the internal model must adapt to the characteristics of the prosthetic devices and reduced sensory feedback. We studied the internal models of 11 amputees and eight unimpaired controls when grasping and lifting a fragile object. When the object was modified from a rigid to fragile state, both subject groups adapted appropriately by significantly reducing grasp force on the first trial with the fragile object compared to the rigid object (p < 0.020). There was a wide range of performance skill illustrated by amputee subjects when lifting the fragile object in 10 repeated trials. One subject, using a voluntary close device, never broke the object, four subjects broke the fragile device on every attempt and seven others failed on their initial attempts, but improved over the repeated trials. Amputees decreased their grip forces 51 ± 7 % from the first to the last trial (p < 0.001), indicating a practice effect. However, amputees used much higher levels of force than controls throughout the testing (p < 0.015). Amputees with better performance on the Box and Blocks test used lower grip force levels (p = 0.006) and had more successful lifts of the fragile object (p = 0.002). In summary, amputees do employ internal models when picking up objects; however, the accuracy of these models is poor and grip force modulation is significantly impaired. Further studies could examine the alternative sensory modalities and training parameters that best promote internal model formation.

Novel targeted sensory reinnervation technique to restore functional hand sensation after transhumeral amputation

We present a case study of a novel variation of the targeted sensory reinnervation technique that provides additional control over sensory restoration after transhumeral amputation. The use of intraoperative somatosensory evoked potentials on individual fascicles of the median and ulnar nerves allowed us to specifically target sensory fascicles to reroute to target cutaneous nerves at a distance away from anticipated motor sites in a transhumeral amputee. This resulted in restored hand maps of the median and ulnar nerve in discrete spatially separated areas. In addition, the subject was able to use native and reinnervated muscle sites to control a robotic arm while simultaneously sensing touch and force feedback from the robotic gripper in a physiologically correct manner. This proof of principle study is the first to demonstrate the ability to have simultaneous dual flow of information (motor and sensory) within the residual limb. In working towards clinical deployment of a sensory integrated prosthetic device, this surgical method addresses the important issue of restoring a usable access point to provide natural hand sensation after upper limb amputation.