Tactile Feedback of Object Slip Facilitates Virtual Object Manipulation

Nonlinear Mapping From EMG to Prosthesis Closing Velocity Improves Force Control With EMG Biofeedback

When using EMG biofeedback to control the grasping force of a myoelectric prosthesis, subjects need to activate their muscles and maintain the myoelectric signal within an appropriate interval. However, their performance decreases for higher forces, because the myoelectric signal is more variable for stronger contractions. Therefore, the present study proposes to implement EMG biofeedback using nonlinear mapping, in which EMG intervals of increasing size are mapped to equal-sized intervals of the prosthesis velocity. To validate this approach, 20 non-disabled subjects performed force-matching tasks using Michelangelo prosthesis with and without EMG biofeedback with linear and nonlinear mapping. Additionally, four transradial amputees performed a functional task in the same feedback and mapping conditions. The success rate in producing desired force was significantly higher with feedback (65.4±15.9%) compared to no feedback (46.2±14.9%) as well as when using nonlinear (62.4±16.8%) versus linear mapping (49.2±17.2%). Overall, in non-disabled subjects, the highest success rate was obtained when EMG biofeedback was combined with nonlinear mapping (72%), and the opposite for linear mapping with no feedback (39.6%). The same trend was registered also in four amputee subjects. Therefore, EMG biofeedback improved prosthesis force control, especially when combined with nonlinear mapping, which showed to be an effective approach to counteract increasing variability of myoelectric signal for stronger contractions.

Neural evidence for functional roles of tactile and visual feedback in the application of myoelectric prosthesis

Objective. The primary purpose of this study was to investigate the electrophysiological mechanism underlying different modalities of sensory feedback and multi-sensory integration in typical prosthesis control tasks.Approach. We recruited 15 subjects and developed a closed-loop setup for three prosthesis control tasks which covered typical activities in the practical prosthesis application, i.e. prosthesis finger position control (PFPC), equivalent grasping force control (GFC) and box and block control (BABC). All the three tasks were conducted under tactile feedback (TF), visual feedback (VF) and tactile-visual feedback (TVF), respectively, with a simultaneous electroencephalography (EEG) recording to assess the electroencephalogram (EEG) response underlying different types of feedback. Behavioral and psychophysical assessments were also administered in each feedback condition.Results. EEG results showed that VF played a predominant role in GFC and BABC tasks. It was reflected by a significantly lower somatosensory alpha event-related desynchronization (ERD) in TVF than in TF and no significant difference in visual alpha ERD between TVF and VF. In PFPC task, there was no significant difference in somatosensory alpha ERD between TF and TVF, while a significantly lower visual alpha ERD was found in TVF than in VF, indicating that TF was essential in situations related to proprioceptive position perception. Tactile-visual integration was found when TF and VF were congruently implemented, showing an obvious activation over the premotor cortex in the three tasks. Behavioral and psychophysical results were consistent with EEG evaluations.Significance. Our findings could provide neural evidence for multi-sensory integration and functional roles of tactile and VF in a practical setting of prosthesis control, shedding a multi-dimensional insight into the functional mechanisms of sensory feedback.

MEMS-Based Tactile Sensors: Materials, Processes and Applications in Robotics

Commonly encountered problems in the manipulation of objects with robotic hands are the contact force control and the setting of approaching motion. Microelectromechanical systems (MEMS) sensors on robots offer several solutions to these problems along with new capabilities. In this review, we analyze tactile, force and/or pressure sensors produced by MEMS technologies including off-the-shelf products such as MEMS barometric sensors. Alone or in conjunction with other sensors, MEMS platforms are considered very promising for robots to detect the contact forces, slippage and the distance to the objects for effective dexterous manipulation. We briefly reviewed several sensing mechanisms and principles, such as capacitive, resistive, piezoresistive and triboelectric, combined with new flexible materials technologies including polymers processing and MEMS-embedded textiles for flexible and snake robots. We demonstrated that without taking up extra space and at the same time remaining lightweight, several MEMS sensors can be integrated into robotic hands to simulate human fingers, gripping, hardness and stiffness sensations. MEMS have high potential of enabling new generation microactuators, microsensors, micro miniature motion-systems (e.g., microrobots) that will be indispensable for health, security, safety and environmental protection.

Evaluating Virtual Hand Illusion through Realistic Appearance and Tactile Feedback

We conducted a virtual reality study to explore virtual hand illusion through three levels of appearance (Appearance dimension: realistic vs. pixelated vs. toon hand appearances) and two levels of tactile feedback (Tactile dimension: no tactile vs. tactile feedback). We instructed our participants to complete a virtual assembly task in this study. Immediately afterward, we asked them to provide self-reported ratings on a survey that captured presence and five embodiment dimensions (hand ownership, touch sensation, agency and motor control, external appearance, and response to external stimuli). The results of our study indicate that (1) tactile feedback generated a stronger sense of presence, touch sensation, and response to external stimuli; (2) the pixelated hand appearance provided the least hand ownership and external appearance; and (3) in the presence of the pixelated hand, prior virtual reality experience of participants impacted their agency and motor control and their response to external stimuli ratings. This paper discusses our findings and provides design considerations for virtual reality applications with respect to the realistic appearance of virtual hands and tactile feedback.

Neurophysiology of slip sensation and grip reaction: insights for hand prosthesis control of slippage

Sensory feedback is pivotal for a proficient dexterity of the hand. By modulating the grip force in function of the quick and not completely predictable change of the load force, grabbed objects are prevented to slip from the hand. Slippage control is an enabling achievement to all manipulation abilities. However, in hand prosthetics, the performance of even the most innovative research solutions proposed so far to control slippage remain distant from the human physiology. Indeed, slippage control involves parallel and compensatory activation of multiple mechanoceptors, spinal and supraspinal reflexes, and higher-order voluntary behavioral adjustments. In this work, we reviewed the literature on physiological correlates of slippage to propose a three-phases model for the slip sensation and reaction. Furthermore, we discuss the main strategies employed so far in the research studies that tried to restore slippage control in amputees. In the light of the proposed three-phase slippage model and from the weaknesses of already implemented solutions, we proposed several physiology-inspired solutions for slippage control to be implemented in the future hand prostheses. Understanding the physiological basis of slip detection and perception and implementing them in novel hand feedback system would make prosthesis manipulation more efficient and would boost its perceived naturalness, fostering the sense of agency for the hand movements.

Evaluating the Effect of Stimulus Duration on Vibrotactile Cue Localizability With a Tactile Sleeve

Vibrotactile arrays are appealing as wearable haptic devices, since designers can vary parameters including cue location and duration to create distinct haptic icons to represent a wide range of information. Vibrotactile sleeves have typically used cues that vary in duration from 100 to 400 ms, but it is not well understood how cue duration might affect localizability of stimuli. Using an experimental protocol typically employed to understand how our visual system can localize stimuli, we examined localization of tactile cues for tactors spaced at fixed locations along the forearm while we varied cue duration between 100 and 400 ms. To validate our experimental methods and hardware, we also evaluated visual cue localization performance. Our visual cue localization results were in agreement with prior experiments showing that varying noise in visual cues affects cue localization. More importantly, this experimental paradigm allowed us to verify that participants could successfully localize tactile cues regardless of duration. Response variance in tactile localizability was much greater than the visual case. There was also an effect of stimulus location on tactile localization performance. Our findings support the variation of tactile cue duration in the 100 to 400 ms range for tactile arrays positioned on the forearm.

Comprehensive Review on Reaching and Grasping of Objects in Robotics

Interaction between a robot and its environment requires perception about the environment, which helps the robot in making a clear decision about the object type and its location. After that, the end effector will be brought to the object’s location for grasping. There are many research studies on the reaching and grasping of objects using different techniques and mechanisms for increasing accuracy and robustness during grasping and reaching tasks. Thus, this paper presents an extensive review of research directions and topics of different approaches such as sensing, learning and gripping, which have been implemented within the current five years.

Psychometric characterization of incidental feedback sources during grasping with a hand prosthesis

Background

A prosthetic system should ideally reinstate the bidirectional communication between the user’s brain and its end effector by restoring both motor and sensory functions lost after an amputation. However, current commercial prostheses generally do not incorporate somatosensory feedback. Even without explicit feedback, grasping using a prosthesis partly relies on sensory information. Indeed, the prosthesis operation is characterized by visual and sound cues that could be exploited by the user to estimate the prosthesis state. However, the quality of this incidental feedback has not been objectively evaluated.

Methods

In this study, the psychometric properties of the auditory and visual feedback of prosthesis motion were assessed and compared to that of a vibro-tactile interface. Twelve able-bodied subjects passively observed prosthesis closing and grasping an object, and they were asked to discriminate (experiment I) or estimate (experiment II) the closing velocity of the prosthesis using visual (VIS), acoustic (SND), or combined (VIS + SND) feedback. In experiment II, the subjects performed the task also with a vibrotactile stimulus (VIB) delivered using a single tactor. The outcome measures for the discrimination and estimation experiments were just noticeable difference (JND) and median absolute estimation error (MAE), respectively.

Results

The results demonstrated that the incidental sources provided a remarkably good discrimination and estimation of the closing velocity, significantly outperforming the vibrotactile feedback. Using incidental sources, the subjects could discriminate almost the minimum possible increment/decrement in velocity that could be commanded to the prosthesis (median JND < 2% for SND and VIS + SND). Similarly, the median MAE in estimating the prosthesis velocity randomly commanded from the full working range was also low, i.e., approximately 5% in SND and VIS + SND.

Conclusions

Since the closing velocity is proportional to grasping force in state-of-the-art myoelectric prostheses, the results of the present study imply that the incidental feedback, when available, could be usefully exploited for grasping force control. Therefore, the impact of incidental feedback needs to be considered when designing a feedback interface in prosthetics, especially since the quality of estimation using supplemental sources (e.g., vibration) can be worse compared to that of the intrinsic cues.

Effects of vibrotactile feedback and grasp interface compliance on perception and control of a sensorized myoelectric hand

Current myoelectric prosthetic limbs are limited in their ability to provide direct sensory feedback to users, which increases attentional demands and reliance on visual cues. Vibrotactile sensory substitution (VSS), which can be used to provide sensory feedback in a non-invasive manner has demonstrated some improvement in myoelectric hand control. In this work, we developed and tested two VSS configurations: one with a single burst-rate modulated actuator and another with a spatially distributed array of five coin tactors. We performed a direct comparative assessment of these two VSS configurations with able-bodied subjects to investigate sensory perception, myoelectric control of grasp force and hand aperture with a prosthesis, and the effects of interface compliance. Six subjects completed a sensory perception experiment under a stimulation only paradigm; sixteen subjects completed experiments to compare VSS performance on perception and graded myoelectric control during grasp force and hand aperture tasks; and ten subjects completed experiments to investigate the effect of mechanical compliance of the myoelectric hand on the ability to control grasp force. Results indicated that sensory perception of vibrotactile feedback was not different for the two VSS configurations in the absence of active myoelectric control, but it was better with feedback from the coin tactor array than with the single actuator during myoelectric control of grasp force. Graded myoelectric control of grasp force and hand aperture was better with feedback from the coin tactor array than with the single actuator, and myoelectric control of grasp force was improved with a compliant grasp interface. Further investigations with VSS should focus on the use of coin tactor arrays by subjects with amputation in real-world settings and on improving control of grasp force by increasing the mechanical compliance of the hand.

Analysis of Al2O3-parylene C bilayer coatings and impact of microelectrode topography on long term stability of implantable neural arrays

OBJECTIVE

Performance of many dielectric coatings for neural electrodes degrades over time, contributing to loss of neural signals and evoked percepts. Studies using planar test substrates have found that a novel bilayer coating of atomic-layer deposited (ALD) Al₂O₃ and parylene C is a promising candidate for neural electrode applications, exhibiting superior stability to parylene C alone. However, initial results from bilayer encapsulation testing on non-planar devices have been less positive. Our aim was to evaluate ALD Al₂O₃-parylene C coatings using novel test paradigms, to rigorously evaluate dielectric coatings for neural electrode applications by incorporating neural electrode topography into test structure design.

APPROACH

Five test devices incorporated three distinct topographical features common to neural electrodes, derived from the utah electrode array (UEA). Devices with bilayer (52 nm Al₂O₃  +  6 µm parylene C) were evaluated against parylene C controls (N  ⩾  6 per device type). Devices were aged in phosphate buffered saline at 67 °C for up to 311 d, and monitored through: (1) leakage current to evaluate encapsulation lifetimes (>1 nA during 5VDC bias indicated failure), and (2) wideband (1-10⁵ Hz) impedance.

MAIN RESULTS

Mean-times-to-failure (MTTFs) ranged from 12 to 506 d for bilayer-coated devices, versus 10 to  >2310 d for controls. Statistical testing (log-rank test, α  =  0.05) of failure rates gave mixed results but favored the control condition. After failure, impedance loss for bilayer devices continued for months and manifested across the entire spectrum, whereas the effect was self-limiting after several days, and restricted to frequencies  <100 Hz for controls. These results correlated well with observations of UEAs encapsulated with bilayer and control films.

SIGNIFICANCE

We observed encapsulation failure modes and behaviors comparable to neural electrode performance which were undetected in studies with planar test devices. We found the impact of parylene C defects to be exacerbated by ALD Al₂O₃, and conclude that inferior bilayer performance arises from degradation of ALD Al₂O₃ when directly exposed to saline. This is an important consideration, given that neural electrodes with bilayer coatings are expected to have ALD Al₂O₃ exposed at dielectric boundaries that delineate electrode sites. Process improvements and use of different inorganic coatings to decrease dissolution in physiological fluids may improve performance. Testing frameworks which take neural electrode complexities into account will be well suited to reliably evaluate such encapsulation schemes.

The clinical relevance of advanced artificial feedback in the control of a multi-functional myoelectric prosthesis

Background

To effectively replace the human hand, a prosthesis should seamlessly respond to user intentions but also convey sensory information back to the user. Restoration of sensory feedback is rated highly by the prosthesis users, and feedback is critical for grasping in able-bodied subjects. Nonetheless, the benefits of feedback in prosthetics are still debated. The lack of consensus is likely due to the complex nature of sensory feedback during prosthesis control, so that its effectiveness depends on multiple factors (e.g., task complexity, user learning).

Methods

We evaluated the impact of these factors with a longitudinal assessment in six amputee subjects, using a clinical setup (socket, embedded control) and a range of tasks (box and blocks, block turn, clothespin and cups relocation). To provide feedback, we have proposed a novel vibrotactile stimulation scheme capable of transmitting multiple variables from a multifunction prosthesis. The subjects wore a bracelet with four by two uniformly placed vibro-tactors providing information on contact, prosthesis state (active function), and grasping force. The subjects also completed a questionnaire for the subjective evaluation of the feedback.

Results

The tests demonstrated that feedback was beneficial only in the complex tasks (block turn, clothespin and cups relocation), and that the training had an important, task-dependent impact. In the clothespin relocation and block turn tasks, training allowed the subjects to establish successful feedforward control, and therefore, the feedback became redundant. In the cups relocation task, however, the subjects needed some training to learn how to properly exploit the feedback. The subjective evaluation of the feedback was consistently positive, regardless of the objective benefits. These results underline the multifaceted nature of closed-loop prosthesis control as, depending on the context, the same feedback interface can have different impact on performance. Finally, even if the closed-loop control does not improve the performance, it could be beneficial as it seems to improve the subjective experience.

Conclusions

Therefore, in this study we demonstrate, for the first time, the relevance of an advanced, multi-variable feedback interface for dexterous, multi-functional prosthesis control in a clinically relevant setting.

Electronic supplementary material

The online version of this article (10.1186/s12984-018-0371-1) contains supplementary material, which is available to authorized users.

Adding vibrotactile feedback to a myoelectric-controlled hand improves performance when online visual feedback is disturbed

We investigated whether adding vibrotactile feedback to a myoelectric-controlled hand, when visual feedback is disturbed, can improve performance during a functional test. For this purpose, able-bodied subjects, activating a myoelectric-controlled hand attached to their right hand performed the modified Box & Blocks test, grasping and manipulating wooden blocks over a partition. This was performed in 3 conditions, using a repeated-measures design: in full light, in a dark room where visual feedback was disturbed and no auditory feedback - one time with the addition of tactile feedback provided during object grasping and manipulation, and one time without any tactile feedback. The average time needed to transfer one block was measured, and an infrared camera was used to give information on the number of grasping errors during performance of the test. Our results show that when vibrotactile feedback was provided, performance time was reduced significantly, compared with when no vibrotactile feedback was available. Furthermore, the accuracy of grasping and manipulation was improved, reflected by significantly fewer errors during test performance. In conclusion, adding vibrotactile feedback to a myoelectric-controlled hand has positive effects on functional performance when visual feedback is disturbed. This may have applications to current myoelectric-controlled hands, as adding tactile feedback may help prosthesis users to improve their functional ability during daily life activities in different environments, particularly when limited visual feedback is available or desirable.

A Novel Wearable Apparatus to Measure Fingertip Forces in Manipulation Tasks Based on MEMS Barometric Sensors

Artificial tactile sensing is a challenging research topic in robotics, motor control, and rehabilitation engineering encompassing multi-disciplinary skills and different technologies. This paper presents the development of a wearable tactile thimble system using MEMS barometric sensors and flexible printed circuit board. Barometric sensors were carefully processed to make them able to transduce contact forces. Thumb, index, and medium fingers were equipped with an array of six sensing elements each, covering the central, lateral, and medial aspects of the fingertip. The sensor integration, signal read-out and processing, hardware architecture of the device, along with the calibration protocol, were described. The test results showed adequate sensitivity at very low forces with an almost linear transduction range up to about 4N (RMSE: 0.04N). Tests on object manipulation tasks highlighted the value of the proposed system demonstrating the ability of measuring both the force amplitude and contact points, demonstrating the suitability of barometric sensors for tactile applications.

GLIMPSE: Google Glass interface for sensory feedback in myoelectric hand prostheses

OBJECTIVE

Providing sensory feedback to the user of the prosthesis is an important challenge. The common approach is to use tactile stimulation, which is easy to implement but requires training and has limited information bandwidth. In this study, we propose an alternative approach based on augmented reality.

APPROACH

We have developed the GLIMPSE, a Google Glass application which connects to the prosthesis via a Bluetooth interface and renders the prosthesis states (EMG signals, aperture, force and contact) using augmented reality (see-through display) and sound (bone conduction transducer). The interface was tested in healthy subjects that used the prosthesis with (FB group) and without (NFB group) feedback during a modified clothespins test that allowed us to vary the difficulty of the task. The outcome measures were the number of unsuccessful trials, the time to accomplish the task, and the subjective ratings of the relevance of the feedback.

MAIN RESULTS

There was no difference in performance between FB and NFB groups in the case of a simple task (basic, same-color clothespins test), but the feedback significantly improved the performance in a more complex task (pins of different resistances). Importantly, the GLIMPSE feedback did not increase the time to accomplish the task. Therefore, the supplemental feedback might be useful in the tasks which are more demanding, and thereby less likely to benefit from learning and feedforward control. The subjects integrated the supplemental feedback with the intrinsic sources (vision and muscle proprioception), developing their own idiosyncratic strategies to accomplish the task.

SIGNIFICANCE

The present study demonstrates a novel self-contained, ready-to-deploy, wearable feedback interface. The interface was successfully tested and was proven to be feasible and functionally beneficial. The GLIMPSE can be used as a practical solution but also as a general and flexible instrument to investigate closed-loop prosthesis control.

Discriminability of Single and Multichannel Intracortical Microstimulation within Somatosensory Cortex

The addition of tactile and proprioceptive feedback to neuroprosthetic limbs is expected to significantly improve the control of these devices. Intracortical microstimulation (ICMS) of somatosensory cortex is a promising method of delivering this sensory feedback. To date, the main focus of somatosensory ICMS studies has been to deliver discriminable signals, corresponding to varying intensity, to a single location in cortex. However, multiple independent and simultaneous streams of sensory information will need to be encoded by ICMS to provide functionally relevant feedback for a neuroprosthetic limb (e.g., encoding contact events and pressure on multiple digits). In this study, we evaluated the ability of an awake, behaving non-human primate (Macaca mulatta) to discriminate ICMS stimuli delivered on multiple electrodes spaced within somatosensory cortex. We delivered serial stimulation on single electrodes to evaluate the discriminability of sensations corresponding to ICMS of distinct cortical locations. Additionally, we delivered trains of multichannel stimulation, derived from a tactile sensor, synchronously across multiple electrodes. Our results indicate that discrimination of multiple ICMS stimuli is a challenging task, but that discriminable sensory percepts can be elicited by both single and multichannel ICMS on electrodes spaced within somatosensory cortex.

Vibrotactile feedback for conveying object shape information as perceived by artificial sensing of robotic arm

This work is a preliminary study towards developing an alternative communication channel for conveying shape information to aid in recognition of items when tactile perception is hindered. Tactile data, acquired during object exploration by sensor fitted robot arm, are processed to recognize four basic geometric shapes. Patterns representing each shape, classified from tactile data, are generated using micro-controller-driven vibration motors which vibrotactually stimulate users to convey the particular shape information. These motors are attached on the subject's arm and their psychological (verbal) responses are recorded to assess the competence of the system to convey shape information to the user in form of vibrotactile stimulations. Object shapes are classified from tactile data with an average accuracy of 95.21 %. Three successive sessions of shape recognition from vibrotactile pattern depicted learning of the stimulus from subjects' psychological response which increased from 75 to 95 %. This observation substantiates the learning of vibrotactile stimulation in user over the sessions which in turn increase the system efficacy. The tactile sensing module and vibrotactile pattern generating module are integrated to complete the system whose operation is analysed in real-time. Thus, the work demonstrates a successful implementation of the complete schema of artificial tactile sensing system for object-shape recognition through vibrotactile stimulations.

Neuromimetic Event-Based Detection for Closed-Loop Tactile Feedback Control of Upper Limb Prostheses

Upper limb amputees lack the valuable tactile sensing that helps provide context about the surrounding environment. Here we utilize tactile information to provide active touch feedback to a prosthetic hand. First, we developed fingertip tactile sensors for producing biomimetic spiking responses for monitoring contact, release, and slip of an object grasped by a prosthetic hand. We convert the sensor output into pulses, mimicking the rapid and slowly adapting spiking responses of receptor afferents found in the human body. Second, we designed and implemented two neuromimetic event-based algorithms, Compliant Grasping and Slip Prevention, on a prosthesis to create a local closed-loop tactile feedback control system (i.e. tactile information is sent to the prosthesis). Grasping experiments were designed to assess the benefit of this biologically inspired neuromimetic tactile feedback to a prosthesis. Results from able-bodied and amputee subjects show the average number of objects that broke or slipped during grasping decreased by over 50% and the average time to complete a grasping task decreased by at least 10% for most trials when comparing neuromimetic tactile feedback with no feedback on a prosthesis. Our neuromimetic method of closed-loop tactile sensing is a novel approach to improving the function of upper limb prostheses.

Effects of Different Tactile Feedback on Myoelectric Closed-Loop Control for Grasping Based on Electrotactile Stimulation

Closed-loop control is important for amputees to manipulate myoelectric prostheses intuitively and dexterously. Tactile feedback can help amputees improve myoelectric control performance for grasping objects. To investigate the effects of different tactile feedback, we performed experiments on six amputees and six able-bodied subjects via electrotactile stimulation. Using a virtual environment, six kinds of objects with different weights and stiffnesses were used for grasping tasks. Five feedback conditions (no feedback, pressure feedback, slip feedback, pressure + slip feedback, and vision feedback) were considered. Nine evaluation indexes and three control objectives (rapidity, economy, and stability) were proposed. Under the five feedback conditions, our study investigated four issues: 1) three types of grasping-related failures; 2) four types of grasping-related time measures; 3) average grasping force; 4) standard deviation of the grasping force. Results indicate that: 1) slip feedback is better than pressure feedback; 2) pressure + slip feedback can improve grasping rapidity; 3) slip feedback significantly contributes to grasping economy and stability; and 4) pressure + slip feedback can perform as well as vision feedback.