Exploring teleimpedance and tactile feedback for intuitive control of the Pisa/IIT SoftHand

Sensor Fusion-Based Anthropomorphic Control of a Robotic Arm

The main goal of this research is to develop a highly advanced anthropomorphic control system utilizing multiple sensor technologies to achieve precise control of a robotic arm. Combining Kinect and IMU sensors, together with a data glove, we aim to create a multimodal sensor system for capturing rich information of human upper body movements. Specifically, the four angles of upper limb joints are collected using the Kinect sensor and IMU sensor. In order to improve the accuracy and stability of motion tracking, we use the Kalman filter method to fuse the Kinect and IMU data. In addition, we introduce data glove technology to collect the angle information of the wrist and fingers in seven different directions. The integration and fusion of multiple sensors provides us with full control over the robotic arm, giving it flexibility with 11 degrees of freedom. We successfully achieved a variety of anthropomorphic movements, including shoulder flexion, abduction, rotation, elbow flexion, and fine movements of the wrist and fingers. Most importantly, our experimental results demonstrate that the anthropomorphic control system we developed is highly accurate, real-time, and operable. In summary, the contribution of this study lies in the creation of a multimodal sensor system capable of capturing and precisely controlling human upper limb movements, which provides a solid foundation for the future development of anthropomorphic control technologies. This technology has a wide range of application prospects and can be used for rehabilitation in the medical field, robot collaboration in industrial automation, and immersive experience in virtual reality environments.

Bio-robotics research for non-invasive myoelectric neural interfaces for upper-limb prosthetic control: a 10-year perspective review

A decade ago, a group of researchers from academia and industry identified a dichotomy between the industrial and academic state-of-the-art in upper-limb prosthesis control, a widely used bio-robotics application. They proposed that four key technical challenges, if addressed, could bridge this gap and translate academic research into clinically and commercially viable products. These challenges are unintuitive control schemes, lack of sensory feedback, poor robustness and single sensor modality. Here, we provide a perspective review on the research effort that occurred in the last decade, aiming at addressing these challenges. In addition, we discuss three research areas essential to the recent development in upper-limb prosthetic control research but were not envisioned in the review 10 years ago: deep learning methods, surface electromyogram decomposition and open-source databases. To conclude the review, we provide an outlook into the near future of the research and development in upper-limb prosthetic control and beyond.

Evaluating the effect of non-invasive force feedback on prosthetic grasp force modulation in participants with and without limb loss

Grasping an object is one of the most common and complex actions performed by humans. The human brain can adapt and update the grasp dynamics through information received from sensory feedback. Prosthetic hands can assist with the mechanical performance of grasping, however currently commercially available prostheses do not address the disruption of the sensory feedback loop. Providing feedback about a prosthetic hand's grasp force magnitude is a top priority for those with limb loss. This study tested a wearable haptic system, i.e., the Clenching Upper-Limb Force Feedback device (CUFF), which was integrated with a novel robotic hand (The SoftHand Pro). The SoftHand Pro was controlled with myoelectrics of the forearm muscles. Five participants with limb loss and nineteen able-bodied participants completed a constrained grasping task (with and without feedback) which required modulation of the grasp to reach a target force. This task was performed while depriving participants of incidental sensory sources (vision and hearing were significantly limited with glasses and headphones). The data were analyzed with Functional Principal Component Analysis (fPCA). CUFF feedback improved grasp precision for participants with limb loss who typically use body-powered prostheses as well as a sub-set of able-bodied participants. Further testing, that is more functional and allows participants to use all sensory sources, is needed to determine if CUFF feedback can accelerate mastery of myoelectric control or would benefit specific patient sub-groups.

Exploring Biomimetic Stiffness Modulation and Wearable Finger Haptics for Improving Myoelectric Control of Virtual Hand

The embodiment of virtual hand (VH) by the user is generally deemed to be important for virtual reality (VR) based hand rehabilitation applications, which may help to engage the user and promote motor skill relearning. In particular, it requires that the VH should produce task-dependent interaction behaviors from rigid to soft. While such a capability is inherent to humans via hand stiffness regulation and haptic interactions, yet it have not been successfully imitated by VH in existing studies. In this paper, we present a work which integrates biomimetic stiffness regulation and wearable finger force feedback in VR scenarios involving myoelectric control of VH. On one hand, the biomimetic stiffness modulation intuitively enables VH to imitate the stiffness profile of the user's hand in real time. On the other hand, the wearable finger force-feedback device elicits a natural and realistic sensation of external force on the fingertip, which provides the user a proper understanding of the environment for enhancing his/her stiffness regulation. The benefits of the proposed integrated system were evaluated with eight healthy subjects that performed two tasks with opposite stiffness requirements. The achieved performance is compared with reduced versions of the integrated system, where either biomimetic impedance control or wearable force feedback is excluded. The results suggest that the proposed integrated system enables the stiffness of VH to be adaptively regulated by the user through the perception of interaction torques and vision, resulting in task-dependent behaviors from rigid to soft for VH.

Dynamical Synergies of Multidigit Hand Prehension

Hand prehension requires highly coordinated control of contact forces. The high-dimensional sensorimotor system of the human hand operates at ease, but poses several challenges when replicated in artificial hands. This paper investigates how the dynamical synergies, coordinated spatiotemporal patterns of contact forces, contribute to the hand grasp, and whether they could potentially capture the force primitives in a low-dimensional space. Ten right-handed subjects were recruited to grasp and hold mass-varied objects. The contact forces during this multidigit prehension were recorded using an instrumented grip glove. The dynamical synergies were derived using principal component analysis (PCA). The contact force patterns during the grasps were reconstructed using the first few synergies. The significance of the dynamical synergies, the influence of load forces and task configurations on the synergies were explained. This study also discussed the contribution of biomechanical constraints on the first few synergies and the current challenges and possible applications of the dynamical synergies in the design and control of exoskeletons. The integration of the dynamical synergies into exoskeletons will be realized in the near future.

A Directional Vibrotactile Feedback Interface for Ergonomic Postural Adjustment

The objective of this paper is to develop and evaluate a directional vibrotactile feedback interface as a guidance tool for postural adjustments during work. In contrast to the existing active and wearable systems such as exoskeletons, we aim to create a lightweight and intuitive interface, capable of guiding its wearers towards more ergonomic and healthy working conditions. To achieve this, a vibrotactile device called ErgoTac is employed to develop three different feedback modalities that are able to provide a directional guidance at the body segments towards a desired pose. In addition, an evaluation is made to find the most suitable, comfortable, and intuitive feedback modality for the user. Therefore, these modalities are first compared experimentally on fifteen subjects wearing eight ErgoTac devices to achieve targeted arm and torso configurations. The most effective directional feedback modality is then evaluated on five subjects in a set of experiments in which an ergonomic optimisation module provides the optimised body posture while performing heavy lifting or forceful exertion tasks. The results yield strong evidence on the usefulness and the intuitiveness of one of the developed modalities in providing guidance towards ergonomic working conditions, by minimising the effect of an external load on body joints. We believe that the integration of such low-cost devices in workplaces can help address the well-known and complex problem of work-related musculoskeletal disorders.

Object Recognition via Evoked Sensory Feedback during Control of a Prosthetic Hand

Haptic and proprioceptive feedback is critical for sensorimotor integration when we use our hand to perform daily tasks. Here, we evaluated how externally evoked haptic and proprioceptive feedback and myoelectric control strategies affected the recognition of object properties when participants controlled a prosthetic hand. Fingertip haptic sensation was elicited using a transcutaneous nerve stimulation grid to encode the prosthetic's fingertip forces. An array of tactors elicited patterned vibratory stimuli to encode tactile-proprioceptive kinematic information of the prosthetic finger joint. Myoelectric signals of the finger flexor and extensor were used to control the position or velocity of joint angles of the prosthesis. Participants were asked to perform object property (stiffness and size) recognition, by controlling the prosthetic hand with concurrent haptic and tactile-proprioceptive feedback. With the evoked feedback, intact and amputee participants recognized the object stiffness and size at success rates ranging from 50% to 100% in both position and velocity control with no significant difference across control schemes. Our findings show that evoked somatosensory feedback in a non-invasive manner can facilitate closed-loop control of the prosthetic hand and allowed for simultaneous recognition of different object properties. The outcomes can facilitate our understanding on the role of sensory feedback during bidirectional human-machine interactions, which can potentially promote user experience in object interactions using prosthetic hands.

Biorealistic Control of Hand Prosthesis Augments Functional Performance of Individuals With Amputation

The human hand has compliant properties arising from muscle biomechanics and neural reflexes, which are absent in conventional prosthetic hands. We recently proved the feasibility to restore neuromuscular reflex control (NRC) to prosthetic hands using real-time computing neuromorphic chips. Here we show that restored NRC augments the ability of individuals with forearm amputation to complete grasping tasks, including standard Box and Blocks Test (BBT), Golf Balls Test (GBT), and Potato Chips Test (PCT). The latter two were more challenging, but novel to prosthesis tests. Performance of a biorealistic controller (BC) with restored NRC was compared to that of a proportional linear feedback (PLF) controller. Eleven individuals with forearm amputation were divided into two groups: one with experience of myocontrol of a prosthetic hand and another without any. Controller performances were evaluated by success rate, failure (drop/break) rate in each grasping task. In controller property tests, biorealistic control achieved a better compliant property with a 23.2% wider range of stiffness adjustment than that of PLF control. In functional grasping tests, participants could control prosthetic hands more rapidly and steadily with neuromuscular reflex. For participants with myocontrol experience, biorealistic control yielded 20.4, 39.4, and 195.2% improvements in BBT, GBT, and PCT, respectively, compared to PLF control. Interestingly, greater improvements were achieved by participants without any myocontrol experience for BBT, GBT, and PCT at 27.4, 48.9, and 344.3%, respectively. The functional gain of biorealistic control over conventional control was more dramatic in more difficult grasp tasks of GBT and PCT, demonstrating the advantage of NRC. Results support the hypothesis that restoring neuromuscular reflex in hand prosthesis can improve neural motor compatibility to human sensorimotor system, hence enabling individuals with amputation to perform delicate grasps that are not tested with conventional prosthetic hands.

Bipolar versus high-density surface electromyography for evaluating risk in fatiguing frequency-dependent lifting activities

Workers often develop low back pain due to manually lifting heavy loads. Instrumental-based assessment tools are used to quantitatively assess the biomechanical risk in lifting activities. This study aims to verify the hypothesis that high-density surface electromyography (HDsEMG) allows an optimized discrimination of risk levels associated with different fatiguing lifting conditions compared to traditional bipolar sEMG. 15 participants performed three lifting tasks with a progressively increasing lifting index (LI) each lasting 15 min. Erector spinae (ES) activity was recorded using both bipolar and HDsEMG systems. The amplitude of both bipolar and HDsEMG can significantly discriminate each pair of LI. HDsEMG data could discriminate across the different LIs starting from the fourth minute of the task while bipolar sEMG could only do so towards the end. The higher discriminative power of HDsEMG data across the lifting tasks makes such methodology a valuable tool to be used to monitor fatigue while lifting and could extend the possibilities offered by currently available instrumental-based tools.

Motion Analysis and Tactile-Based Impedance Control of the Chest Holder of a Piggyback Patient Transfer Robot

Patient transfer, such as carrying a bedridden patient from a bed to a pedestal pan or a wheelchair and back, is one of the most physically challenging tasks in nursing care facilities. To reduce the intensity of physical labor on nurses or caregivers, a piggyback transfer robot has been developed by imitating the motion when a person holds another person on his/her back. As the chest holder supports most of the weight of the care-receiver during transfer, a human-robot dynamic model was built to analyze the influences of the motion of the chest holder on comfort. Simulations and experiments were conducted, and the results demonstrated that the rotational motion of the chest holder is the key factor affecting comfort. A tactile-based impedance control law was developed to adjust the rotational motion. Subjective evaluations of ten healthy subjects showed that adjusting the rotational motion of the chest holder is a useful way to achieve a comfortable transfer.

Myoelectric prosthesis users and non-disabled individuals wearing a simulated prosthesis exhibit similar compensatory movement strategies

Background

Research studies on upper limb prosthesis function often rely on the use of simulated myoelectric prostheses (attached to and operated by individuals with intact limbs), primarily to increase participant sample size. However, it is not known if these devices elicit the same movement strategies as myoelectric prostheses (operated by individuals with amputation). The objective of this study was to address the question of whether non-disabled individuals using simulated prostheses employ the same compensatory movements (measured by hand and upper body kinematics) as individuals who use actual myoelectric prostheses.

Methods

The upper limb movements of two participant groups were investigated: (1) twelve non-disabled individuals wearing a simulated prosthesis, and (2) three individuals with transradial amputation using their custom-fitted myoelectric devices. Motion capture was used for data collection while participants performed a standardized functional task. Performance metrics, hand movements, and upper body angular kinematics were calculated. For each participant group, these measures were compared to those from a normative baseline dataset. Each deviation from normative movement behaviour, by either participant group, indicated that compensatory movements were used during task performance.

Results

Results show that participants using either a simulated or actual myoelectric prosthesis exhibited similar deviations from normative behaviour in phase durations, hand velocities, hand trajectories, number of movement units, grip aperture plateaus, and trunk and shoulder ranges of motion.

Conclusions

This study suggests that the use of a simulated prosthetic device in upper limb research offers a reasonable approximation of compensatory movements employed by a low- to moderately-skilled transradial myoelectric prosthesis user.

Critical Issues and Imminent Challenges in the Use of sEMG in Return-To-Work Rehabilitation of Patients Affected by Neurological Disorders in the Epoch of Human–Robot Collaborative Technologies

Patients affected by neurological pathologies with motor disorders when they are of working age have to cope with problems related to employability, difficulties in working, and premature work interruption. It has been demonstrated that suitable job accommodation plans play a beneficial role in the overall quality of life of pathological subjects. A well-designed return-to-work program should consider several recent innovations in the clinical and ergonomic fields. One of the instrument-based methods used to monitor the effectiveness of ergonomic interventions is surface electromyography (sEMG), a multi-channel, non-invasive, wireless, wearable tool, which allows in-depth analysis of motor coordination mechanisms. Although the scientific literature in this field is extensive, its use remains significantly underexploited and the state-of-the-art technology lags expectations. This is mainly attributable to technical and methodological (electrode-skin impedance, noise, electrode location, size, configuration and distance, presence of crosstalk signals, comfort issues, selection of appropriate sensor setup, sEMG amplitude normalization, definition of correct sEMG-related outcomes and normative data) and cultural limitations. The technical and methodological problems are being resolved or minimized also thanks to the possibility of using reference books and tutorials. Cultural limitations are identified in the traditional use of qualitative approaches at the expense of quantitative measurement-based monitoring methods to design and assess ergonomic interventions and train operators. To bridge the gap between the return-to-work rehabilitation and other disciplines, several teaching courses, accompanied by further electrodes and instrumentations development, should be designed at all Bachelor, Master and PhD of Science levels to enhance the best skills available among physiotherapists, occupational health and safety technicians and ergonomists.

A method for autonomous robotic manipulation through exploratory interactions with uncertain environments

Expanding robot autonomy can deliver functional flexibility and enable fast deployment of robots in challenging and unstructured environments. In this direction, significant advances have been recently made in visual-perception driven autonomy, which is mainly due to the availability of rich sensory data-sets. However, current robots’ physical interaction autonomy levels still remain at a basic level. Towards providing a systematic approach to this problem, this paper presents a new context-aware and adaptive method that allows a robotic platform to interact with unknown environments. In particular, a multi-axes self-tuning impedance controller is introduced to regulate quasi-static parameters of the robot based on previous experience in interacting with similar environments and the real-time sensory data. The proposed method is also capable of differentiating internal and external disruptions, and responding to them accordingly and appropriately. An agricultural experiment with different deformable material is presented to validate robot interaction autonomy improvements, and the capability of the proposed methodology in detecting and responding to unexpected events (e.g., faults).

Hand–object configuration estimation using particle filters for dexterous in-hand manipulation

We consider the problem of in-hand dexterous manipulation with a focus on unknown or uncertain hand–object parameters, such as hand configuration, object pose within hand, and contact positions. In particular, in this work we formulate a generic framework for hand–object configuration estimation using underactuated hands as an example. Owing to the passive reconfigurability and the lack of encoders in the hand’s joints, it is challenging to estimate, plan, and actively control underactuated manipulation. By modeling the grasp constraints, we present a particle filter-based framework to estimate the hand configuration. Specifically, given an arbitrary grasp, we start by sampling a set of hand configuration hypotheses and then randomly manipulate the object within the hand. While observing the object’s movements as evidence using an external camera, which is not necessarily calibrated with the hand frame, our estimator calculates the likelihood of each hypothesis to iteratively estimate the hand configuration. Once converged, the estimator is used to track the hand configuration in real time for future manipulations. Thereafter, we develop an algorithm to precisely plan and control the underactuated manipulation to move the grasped object to desired poses. In contrast to most other dexterous manipulation approaches, our framework does not require any tactile sensing or joint encoders, and can directly operate on any novel objects, without requiring a model of the object a priori. We implemented our framework on both the Yale Model O hand and the Yale T42 hand. The results show that the estimation is accurate for different objects, and that the framework can be easily adapted across different underactuated hand models. In the end, we evaluated our planning and control algorithm with handwriting tasks, and demonstrated the effectiveness of the proposed framework.

Skin Stretch Haptic Feedback to Convey Closure Information in Anthropomorphic, Under-Actuated Upper Limb Soft Prostheses

Restoring hand function in individuals with upper limb loss is a challenging task, made difficult by the complexity of human hands from both a functional and sensory point of view. Users of commercial prostheses, even sophisticated devices, must visually attend to the hand to know its state, since in most cases they are not provided with any direct sensory information. Among the different types of haptic feedback that can be delivered, particularly information on hand opening is likely to reduce the requirement of constant visual attention. In recent years, there has been a trend of using underactuated, compliant multi-fingered hands as upper limb prostheses, in part due to their simplicity and ease of use attributed to low degree-of-freedom (d.o.f.) actuation. The trend toward underactuation encourages the design of one d.o.f. haptic devices to provide intuitive sensory feedback from the prosthesis. However, mapping the closure of a multi-d.o.f. prosthetic hand to a simple and intuitive haptic cue is not a trivial task. In this paper, we explore the use of a one d.o.f. skin stretch haptic device, the rice haptic rocker, to provide intuitive proprioceptive feedback indicating overall hand closure of an underactuated prosthesis. The benefits and challenges of the system are assessed in multi-tasking and reduced vision scenarios for an object-size discrimination task, in an effort to simulate challenges in daily life, and are compared against the haptic resolution of the device using the just noticeable difference. Finally, an evaluation done with a prosthesis user, in the form of a truncated version of the Activities Measure for Upper Limb Amputees (AM-ULA), shows possible benefits of the addition of haptic feedback in tasks with reduced visual attention.

Inter-Limb Transfer of Grasp Force Perception With Closed-Loop Hand Prosthesis

Sensory feedback of grasp forces provides important information about physical interactions between the hand and objects, enabling both reactive and anticipatory neural control mechanisms. The numerous studies have shown artificial sensory feedback of various forms improves force control during grasping tasks by prosthetic hand users through a closed-feedback loop. However, little is known about how perceptual information is transferred between an intact limb and a closed-loop prosthetic limb, and the extent to which training inter-limb transfer may improve myoelectric prosthetic control. We addressed these gaps by using a contralateral force-matching task in which able-bodied participants were asked to generate grasp forces with their native hand, and then match it using the contralateral hand or a soft-synergy prosthetic hand worn on the contralateral arm that was coupled with a mechanotactile feedback device. We found that absolute matching error and matching time were greater when using the prosthetic system than the native hand. However, with contralateral specific training, subjects were able to produce similar relative matching error with the prosthetic system and the native hand, especially at the untrained force level. These findings suggest that an association can be established between the perception produced by the prosthetic limb and the contralateral intact limb, and provide novel insights about potential applications to training and design of the closed-loop prosthesis.

Supplemental vibrotactile feedback of real-time limb position enhances precision of goal-directed reaching

We examined vibrotactile stimulation as a form of supplemental limb state feedback to enhance planning and ongoing control of goal-directed movements. Subjects wore a two-dimensional vibrotactile display on their nondominant arm while performing horizontal planar reaching with the dominant arm. The vibrotactile display provided feedback of hand position such that small hand displacements were more easily discriminable using vibrotactile feedback than with intrinsic proprioceptive feedback. When subjects relied solely on proprioception to capture visuospatial targets, performance was degraded by proprioceptive drift and an expansion of task space. By contrast, reach accuracy was enhanced immediately when subjects were provided vibrotactile feedback and further improved over 2 days of training. Improvements reflected resolution of proprioceptive drift, which occurred only when vibrotactile feedback was active, demonstrating that benefits of vibrotactile feedback are due, in part to its integration into the ongoing control of movement. A partial resolution of task space expansion persisted even when vibrotactile feedback was inactive, demonstrating that training with vibrotactile feedback also induced changes in movement planning. However, the benefits of vibrotactile feedback come at a cognitive cost. All subjects adopted a stereotyped strategy wherein they attempted to capture targets by moving first along one axis of the vibrotactile display and then the other. For most subjects, this inefficient approach did not resolve over two bouts of training performed on separate days, suggesting that additional training is needed to integrate vibrotactile feedback into the planning and online control of goal-directed reaching in a way that promotes smooth and efficient movement. NEW & NOTEWORTHY A two-dimensional vibrotactile display provided state (not error) feedback to enhance control of a moving limb. Subjects learned to use state feedback to perform blind reaches with accuracy and precision exceeding that attained using intrinsic proprioception alone. Feedback utilization incurred substantial cognitive cost: subjects moved first along one axis of the vibrotactile display, then the other. This stereotyped control strategy must be overcome if vibrotactile limb state feedback is to promote naturalistic limb movements.

A survey on what Australians with upper limb difference want in a prosthesis: justification for using soft robotics and additive manufacturing for customized prosthetic hands

Purpose: Upper limb prostheses are part of a rapidly changing market place. Despite development in device design, surveys report low levels of uptake and dissatisfaction with current prosthetic design. In this study, we present the results of a survey conducted with people with upper limb difference in Australia on their use of current prostheses and preferences in a prosthetic in order to inform future prosthetic hand design.Methods: An online survey was conducted on upper limb amputees, with 27 respondents that completed the survey. The survey was a mixture of open-ended questions, ranking design features and quantitative questions on problems experienced and desired attributes of future prosthesis designs.Results: Common key issues and concerns were isolated in the survey related to the weight, manipulation and dexterity, aesthetics, sensory feedback and financial cost; each of which could be addressed by additive manufacturing and soft robotics techniques.Conclusions: The adaptability of additive manufacturing and soft robotics to the highlighted concerns of participants shows that further research into these techniques is a feasible method to improve patient satisfaction and acceptance in prosthetic hands.Implications for rehabilitationEven with recent developments and advances in prosthetic design, the needs and desires of prosthetic users are not being met with current products.The desires and needs of those with upper limb difference are diverse.Using additive manufacturing to produce prosthetics allows for mass customization of prosthetics to meet these diverse needs while reducing costs.A soft robotic approach to prosthetics can help meet the desires of reducing weight and costs, while maintaining functionality.

Effects of vibrotactile feedback and grasp interface compliance on perception and control of a sensorized myoelectric hand

Current myoelectric prosthetic limbs are limited in their ability to provide direct sensory feedback to users, which increases attentional demands and reliance on visual cues. Vibrotactile sensory substitution (VSS), which can be used to provide sensory feedback in a non-invasive manner has demonstrated some improvement in myoelectric hand control. In this work, we developed and tested two VSS configurations: one with a single burst-rate modulated actuator and another with a spatially distributed array of five coin tactors. We performed a direct comparative assessment of these two VSS configurations with able-bodied subjects to investigate sensory perception, myoelectric control of grasp force and hand aperture with a prosthesis, and the effects of interface compliance. Six subjects completed a sensory perception experiment under a stimulation only paradigm; sixteen subjects completed experiments to compare VSS performance on perception and graded myoelectric control during grasp force and hand aperture tasks; and ten subjects completed experiments to investigate the effect of mechanical compliance of the myoelectric hand on the ability to control grasp force. Results indicated that sensory perception of vibrotactile feedback was not different for the two VSS configurations in the absence of active myoelectric control, but it was better with feedback from the coin tactor array than with the single actuator during myoelectric control of grasp force. Graded myoelectric control of grasp force and hand aperture was better with feedback from the coin tactor array than with the single actuator, and myoelectric control of grasp force was improved with a compliant grasp interface. Further investigations with VSS should focus on the use of coin tactor arrays by subjects with amputation in real-world settings and on improving control of grasp force by increasing the mechanical compliance of the hand.

Human-Robotic Variable-Stiffness Grasps of Small-Fruit Containers Are Successful Even Under Severely Impaired Sensory Feedback

Application areas of robotic grasping extend to delicate objects like groceries. The intrinsic elasticity offered by variable-stiffness actuators (VSA) appears to be promising in terms of being able to adapt to the object shape, to withstand collisions with the environment during the grasp acquisition, and to resist the weight applied to the fingers by a lifted object during the actual grasp. It is hypothesized that these properties are particularly useful in the absence of high-quality sensory feedback, which would otherwise be able to guide the shape adaptation and collision avoidance, and that in this case, VSA hands perform better than hands with fixed stiffness. This hypothesis is tested in an experiment where small-fruit containers are picked and placed using a newly developed variable-stiffness robotic hand. The grasp performance is measured under different sensory feedback conditions: full or impaired visual feedback, full or impaired force feedback. The hand is switched between a variable-stiffness mode and two fixed-stiffness modes. Strategies for modulating the stiffness and exploiting environmental constraints are observed from human operators that control the robotic hand. The results show consistently successful grasps under all stiffness and feedback conditions. However, the performance is affected by the amount of available visual feedback. Different stiffness modes turn out to be beneficial in different feedback conditions and with respect to different performance criteria, but a general advantage of VSA over fixed stiffness cannot be shown for the present task. Guidance of the fingers along cracks and gaps is observed, which may inspire the programming of autonomously grasping robots.

The SoftHand Pro: Functional evaluation of a novel, flexible, and robust myoelectric prosthesis

Roughly one quarter of active upper limb prosthetic technology is rejected by the user, and user surveys have identified key areas requiring improvement: function, comfort, cost, durability, and appearance. Here we present the first systematic, clinical assessment of a novel prosthetic hand, the SoftHand Pro (SHP), in participants with transradial amputation and age-matched, limb-intact participants. The SHP is a robust and functional prosthetic hand that minimizes cost and weight using an underactuated design with a single motor. Participants with limb loss were evaluated on functional clinical measures before and after a 6-8 hour training period with the SHP as well as with their own prosthesis; limb-intact participants were tested only before and after SHP training. Participants with limb loss also evaluated their own prosthesis and the SHP (following training) using subjective questionnaires. Both objective and subjective results were positive and illuminated the strengths and weaknesses of the SHP. In particular, results pre-training show the SHP is easy to use, and significant improvement in the Activities Measure for Upper Limb Amputees in both groups following a 6-8 hour training highlights the ease of learning the unique features of the SHP (median improvement: 4.71 and 3.26 and p = 0.009 and 0.036 for limb loss and limb-intact groups, respectively). Further, we found no difference in performance compared to participant's own commercial devices in several clinical measures and found performance surpassing these devices on two functional tasks, buttoning a shirt and using a cell phone, suggesting a functional prosthetic design. Finally, improvements are needed in the SHP design and/or training in light of poor results in small object manipulation. Taken together, these results show the promise of the SHP, a flexible and adaptive prosthetic hand, and pave a path forward to ensuring higher functionality in future.

Soft Poly-Limbs: Toward a New Paradigm of Mobile Manipulation for Daily Living Tasks

We present the design and development of the fluid-driven, wearable, Soft Poly-Limb (SPL), from the Greek word polys, meaning many. The SPL utilizes the numerous traits of soft robotics to enable a novel approach in providing safe and compliant mobile manipulation assistance to healthy and impaired users. This wearable system equips the user with a controllable additional limb that is capable of complex three-dimensional motion in space. Similar to an elephant trunk, the SPL is able to manipulate objects using a variety of end effectors, such as suction adhesion or a soft grasper, as well as its entire soft body to conform around an object, able to lift 2.35 times its own weight. To develop these highly articulated soft robotic limbs, we provide a novel set of systematic design rules, obtained through varying geometrical parameters of the SPL through experimentally verified finite element method models. We investigate performance of the limb by testing the lifetime of the new SPL actuators, evaluating its payload capacity, operational workspace, and capability of interacting close to a user through a spatial mobility test. Furthermore, we are able to demonstrate limb controllability through multiple user-intent detection modalities. Finally, we explore the limb's ability to assist in multitasking and pick and place scenarios with varying mounting locations of the SPL around the user's body. Our results highlight the SPL's ability to safely interact with the user while demonstrating promising performance in assisting with a wide variety of tasks, in both work and general living settings.

A Bio-inspired Grasp Stiffness Control for Robotic Hands

This work presents a bio-inspired grasp stiffness control for robotic hands based on the concepts of Common Mode Stiffness (CMS) and Configuration Dependent Stiffness (CDS). Using an ellipsoid representation of the desired grasp stiffness, the algorithm focuses on achieving its geometrical features. Based on preliminary knowledge of the fingers workspace, the method starts by exploring the possible hand poses that maintain the grasp contacts on the object. This outputs a first selection of feasible grasp configurations providing the base for the CDS control. Then, an optimization is performed to find the minimum joint stiffness (CMS control) that would stabilize these grasps. This joint stiffness can be increased afterwards depending on the task requirements. The algorithm finally chooses among all the found stable configurations the one that results in a better approximation of the desired grasp stiffness geometry (CDS). The proposed method results in a reduction of the control complexity, needing to independently regulate the joint positions, but requiring only one input to produce the desired joint stiffness. Moreover, the usage of the fingers pose to attain the desired grasp stiffness results in a more energy-efficient configuration than only relying on the joint stiffness (i.e., joint torques) modifications. The control strategy is evaluated using the fully actuated Allegro Hand while grasping a wide variety of objects. Different desired grasp stiffness profiles are selected to exemplify several stiffness geometries.

HapPro: A Wearable Haptic Device for Proprioceptive Feedback

OBJECTIVE

Myoelectric hand prostheses have reached a considerable technological level and gained an increasing attention in assistive robotics. However, their abandonment rate remains high, with unintuitive control and lack of sensory feedback being major causes. Among the different types of sensory information, proprioception, e.g., information on hand aperture, is crucial to successfully perform everyday actions. Despite the many attempts in literature to restore and convey this type of feedback, much remains to be done to close the action-perception loop in prosthetic devices.

METHODS

With this as motivation, in this paper we introduce HapPro, a wearable, noninvasive haptic device that can convey proprioceptive information for a prosthetic hand. The device was used with an under-actuated, simple to control anthropomorphic robotic hand, providing information about hand aperture by mapping it to the position of a wheel that can run on the user's forearm. Tests with 43 able bodied subjects and one amputee subject were conducted in order to quantify the effectiveness of HapPro as a feedback device.

RESULTS

HapPro provided a good level of accuracy for item discrimination. Participants also reported the device to be intuitive and effective in conveying proprioceptive cues. Similar results were obtained in the proof-of-concept experiment with an amputee subject.

CONCLUSIONS

Results show that HapPro is able to convey information on the opening of a prosthetic hand in a noninvasive way.

SIGNIFICANCE

Using this device for proprioceptive feedback could improve usability of myoelectric prostheses, potentially reducing abandonment and increasing quality of life for their users.

Minute hands of clocks indicating the same time are not perceived as haptically parallel

Many studies have already shown that a large idiosyncratic orientation difference is needed to perceive two bars that are far apart as haptically parallel. There exist also strong indications that if such bars are imagined to be minute hands of clocks, errors made in clock time estimates and clock time settings are much smaller. The current study investigated this seemingly discrepancy. Participants partook in three experiments: parallel setting, clock time estimate and clock time setting, in this order. As the individual parallel settings were used in the subsequent clock time estimate experiment, and the estimated clock times in the clock time setting experiment, the deviations could be compared directly. In all three experiments, the deviations were systematic and idiosyncratic, and consistent with a biasing influence of an egocentric reference frame. However, the deviations in the two clock time experiments were indeed much smaller than in the parallel setting experiment. Task instruction and strengthened focus on an allocentric reference frame are the most likely explanations. These findings provide fundamental insights in the processing of spatial information. Taking these findings into account when designing haptic devices may make these more intuitive.

Improving Fine Control of Grasping Force during Hand-Object Interactions for a Soft Synergy-Inspired Myoelectric Prosthetic Hand

The concept of postural synergies of the human hand has been shown to potentially reduce complexity in the neuromuscular control of grasping. By merging this concept with soft robotics approaches, a multi degrees of freedom soft-synergy prosthetic hand [SoftHand-Pro (SHP)] was created. The mechanical innovation of the SHP enables adaptive and robust functional grasps with simple and intuitive myoelectric control from only two surface electromyogram (sEMG) channels. However, the current myoelectric controller has very limited capability for fine control of grasp forces. We addressed this challenge by designing a hybrid-gain myoelectric controller that switches control gains based on the sensorimotor state of the SHP. This controller was tested against a conventional single-gain (SG) controller, as well as against native hand in able-bodied subjects. We used the following tasks to evaluate the performance of grasp force control: (1) pick and place objects with different size, weight, and fragility levels using power or precision grasp and (2) squeezing objects with different stiffness. Sensory feedback of the grasp forces was provided to the user through a non-invasive, mechanotactile haptic feedback device mounted on the upper arm. We demonstrated that the novel hybrid controller enabled superior task completion speed and fine force control over SG controller in object pick-and-place tasks. We also found that the performance of the hybrid controller qualitatively agrees with the performance of native human hands.

Grasp Performance of a Soft Synergy-Based Prosthetic Hand: A Pilot Study

Current prosthetic hands are frequently rejected in part due to limited functionality and versatility. We assessed the feasibility of a novel prosthetic hand, the SoftHand Pro (SHP), whose design combines soft robotics and hand postural synergies. Able-bodied subjects ( ) tracked cursor motion by opening and closing the SHP and performed a grasp-lift-hold-release (GLHR) task with a sensorized cylindrical object of variable weight. The SHP control was driven by electromyographic (EMG) signals from two antagonistic muscles. Although the time to perform the GLHR task was longer for the SHP than native hand for the first few trials (10.2 ± 1.4 s and 2.13 ± 0.09 s, respectively), performance was much faster on subsequent trials (~5 s). The SHP steady-state grip force was significantly modulated as a function of object weight ( ). For the native hand, however, peak and steady-state grip forces were modulated to a greater extent (+68% and +91%, respectively). These changes were mediated by the modulation of EMG amplitude and co-contraction. These data suggest that the SHP has a promise for prosthetic applications and point-to-design modifications that could improve the SHP.

Influence of force feedback on grasp force modulation in prosthetic applications: a preliminary study

In typical movement, humans use a combination of feed-forward and feedback motor control strategies to interact with the world around them. However, when sensory input is impaired or absent, as in the case of various neuropathies or amputation, the ability to perform everyday tasks, like modulating grip force to object weight, can be affected. In this study, we show the results of a preliminary study using a pressure cuff-like force feedback device (CUFF) with the SoftHand Pro (SHP) prosthetic hand. Subjects lifted an object of various weights using their own hand, with the SHP without feedback, and the SHP with force feedback. As expected, significant differences were found between the two SHP conditions and the native hand, but surprisingly not between the SHP conditions. A closer look at the data suggests the feedback may help diminish the overall grip force used during grasping even if it does not alter the grip force modulation to object weight. The lack of significance may be due in part to high intra- and inter-subject variability. Additional training with the CUFF and/or customization of the feedback may enhance the effects and warrants further study.

Synergy-Based Bilateral Port: A Universal Control Module for Tele-Manipulation Frameworks Using Asymmetric Master-Slave Systems

Endowing tele-manipulation frameworks with the capability to accommodate a variety of robotic hands is key to achieving high performances through permitting to flexibly interchange the end-effector according to the task considered. This requires the development of control policies that not only cope with asymmetric master–slave systems but also whose high-level components are designed in a unified space in abstraction from the devices specifics. To address this dual challenge, a novel synergy port is developed that resolves the kinematic, sensing, and actuation asymmetries of the considered system through generating motion and force feedback references in the hardware-independent hand postural synergy space. It builds upon the concept of the Cartesian-based synergy matrix, which is introduced as a tool mapping the fingertips Cartesian space to the directions oriented along the grasp principal components. To assess the effectiveness of the proposed approach, the synergy port has been integrated into the control system of a highly asymmetric tele-manipulation framework, in which the 3-finger hand exoskeleton HEXOTRAC is used as a master device to control the SoftHand, a robotic hand whose transmission system relies on a single motor to drive all joints along a soft synergistic path. The platform is further enriched with the vision-based motion capture system Optitrack to monitor the 6D trajectory of the user’s wrist, which is used to control the robotic arm on which the SoftHand is mounted. Experiments have been conducted with the humanoid robot COMAN and the KUKA LWR robotic manipulator. Results indicate that this bilateral interface is highly intuitive and allows users with no prior experience to reach, grasp, and transport a variety of objects exhibiting very different shapes and impedances. In addition, the hardware and control solutions proved capable of accommodating users with different hand kinematics. Finally, the proposed control framework offers a universal, flexible, and intuitive interface allowing for the performance of effective tele-manipulations.

A Human-Robot Co-Manipulation Approach Based on Human Sensorimotor Information

This paper aims to improve the interaction and coordination between the human and the robot in cooperative execution of complex, powerful, and dynamic tasks. We propose a novel approach that integrates online information about the human motor function and manipulability properties into the hybrid controller of the assistive robot. Through this human-in-the-loop framework, the robot can adapt to the human motor behavior and provide the appropriate assistive response in different phases of the cooperative task. We experimentally evaluate the proposed approach in two human-robot co-manipulation tasks that require specific complementary behavior from the two agents. Results suggest that the proposed technique, which relies on a minimum degree of task-level pre-programming, can achieve an enhanced physical human-robot interaction performance and deliver appropriate level of assistance to the human operator.

Can a Soft Robotic Probe Use Stiffness Control Like a Human Finger to Improve Efficacy of Haptic Perception?

When humans are asked to palpate a soft tissue to locate a hard nodule, they regulate the stiffness, speed, and force of the finger during examination. If we understand the relationship between these behavioral variables and haptic information gain (transfer entropy) during manual probing, we can improve the efficacy of soft robotic probes for soft tissue palpation, such as in tumor localization in minimally invasive surgery. Here, we recorded the muscle co-contraction activity of the finger using EMG sensors to address the question as to whether joint stiffness control during manual palpation plays an important role in the haptic information gain. To address this question, we used a soft robotic probe with a controllable stiffness joint and a force sensor mounted at the base to represent the function of the tendon in a biological finger. Then, we trained a Markov chain using muscle co-contraction patterns of human subjects, and used it to control the stiffness of the soft robotic probe in the same soft tissue palpation task. The soft robotic experiments showed that haptic information gain about the depth of the hard nodule can be maximized by varying the internal stiffness of the soft probe.

Assessment of Myoelectric Controller Performance and Kinematic Behavior of a Novel Soft Synergy-Inspired Robotic Hand for Prosthetic Applications

Myoelectric artificial limbs can significantly advance the state of the art in prosthetics, since they can be used to control mechatronic devices through muscular activity in a way that mimics how the subjects used to activate their muscles before limb loss. However, surveys indicate that dissatisfaction with the functionality of terminal devices underlies the widespread abandonment of prostheses. We believe that one key factor to improve acceptability of prosthetic devices is to attain human likeness of prosthesis movements, a goal which is being pursued by research on social and human-robot interactions. Therefore, to reduce early abandonment of terminal devices, we propose that controllers should be designed so as to ensure effective task accomplishment in a natural fashion. In this work, we have analyzed and compared the performance of three types of myoelectric controller algorithms based on surface electromyography to control an underactuated and multi-degrees of freedom prosthetic hand, the SoftHand Pro. The goal of the present study was to identify the myoelectric algorithm that best mimics the native hand movements. As a preliminary step, we first quantified the repeatability of the SoftHand Pro finger movements and identified the electromyographic recording sites for able-bodied individuals with the highest signal-to-noise ratio from two pairs of muscles, i.e., flexor digitorum superficialis/extensor digitorum communis, and flexor carpi radialis/extensor carpi ulnaris. Able-bodied volunteers were then asked to execute reach-to-grasp movements, while electromyography signals were recorded from flexor digitorum superficialis/extensor digitorum communis as this was identified as the muscle pair characterized by high signal-to-noise ratio and intuitive control. Subsequently, we tested three myoelectric controllers that mapped electromyography signals to position of the SoftHand Pro. We found that a differential electromyography-to-position mapping ensured the highest coherence with hand movements. Our results represent a first step toward a more effective and intuitive control of myoelectric hand prostheses.

Neuromimetic Event-Based Detection for Closed-Loop Tactile Feedback Control of Upper Limb Prostheses

Upper limb amputees lack the valuable tactile sensing that helps provide context about the surrounding environment. Here we utilize tactile information to provide active touch feedback to a prosthetic hand. First, we developed fingertip tactile sensors for producing biomimetic spiking responses for monitoring contact, release, and slip of an object grasped by a prosthetic hand. We convert the sensor output into pulses, mimicking the rapid and slowly adapting spiking responses of receptor afferents found in the human body. Second, we designed and implemented two neuromimetic event-based algorithms, Compliant Grasping and Slip Prevention, on a prosthesis to create a local closed-loop tactile feedback control system (i.e. tactile information is sent to the prosthesis). Grasping experiments were designed to assess the benefit of this biologically inspired neuromimetic tactile feedback to a prosthesis. Results from able-bodied and amputee subjects show the average number of objects that broke or slipped during grasping decreased by over 50% and the average time to complete a grasping task decreased by at least 10% for most trials when comparing neuromimetic tactile feedback with no feedback on a prosthesis. Our neuromimetic method of closed-loop tactile sensing is a novel approach to improving the function of upper limb prostheses.

Hand synergies: Integration of robotics and neuroscience for understanding the control of biological and artificial hands

The term 'synergy' - from the Greek synergia - means 'working together'. The concept of multiple elements working together towards a common goal has been extensively used in neuroscience to develop theoretical frameworks, experimental approaches, and analytical techniques to understand neural control of movement, and for applications for neuro-rehabilitation. In the past decade, roboticists have successfully applied the framework of synergies to create novel design and control concepts for artificial hands, i.e., robotic hands and prostheses. At the same time, robotic research on the sensorimotor integration underlying the control and sensing of artificial hands has inspired new research approaches in neuroscience, and has provided useful instruments for novel experiments. The ambitious goal of integrating expertise and research approaches in robotics and neuroscience to study the properties and applications of the concept of synergies is generating a number of multidisciplinary cooperative projects, among which the recently finished 4-year European project "The Hand Embodied" (THE). This paper reviews the main insights provided by this framework. Specifically, we provide an overview of neuroscientific bases of hand synergies and introduce how robotics has leveraged the insights from neuroscience for innovative design in hardware and controllers for biomedical engineering applications, including myoelectric hand prostheses, devices for haptics research, and wearable sensing of human hand kinematics. The review also emphasizes how this multidisciplinary collaboration has generated new ways to conceptualize a synergy-based approach for robotics, and provides guidelines and principles for analyzing human behavior and synthesizing artificial robotic systems based on a theory of synergies.

Tactile Feedback of Object Slip Facilitates Virtual Object Manipulation

Recent advances in myoelectric prosthetic technology have enabled more complex movements and interactions with objects, but the lack of natural haptic feedback makes object manipulation difficult to perform. Our research effort aims to develop haptic feedback systems for improving user performance in object manipulation. Specifically, in this work, we explore the effectiveness of vibratory tactile feedback of slip information for grasping objects without slipping. A user interacts with a virtual environment to complete a virtual grasp and hold task using a Sensable Phantom. Force feedback simulates contact with objects, and vibratory tactile feedback alerts the user when a virtual object is slipping from the grasp. Using this task, we found that tactile feedback significantly improved a user's ability to detect and respond to slip and to recover the slipping object when visual feedback was not available. This advantage of tactile feedback is especially important in conjunction with force feedback, which tends to reduce a subject's grasping forces and therefore encourage more slips. Our results demonstrate the potential of slip feedback to improve a prosthesis user's ability to interact with objects with less visual attention, aiding in performance of everyday manipulation tasks.

Design and Characterization of a Fabric-Based Softness Display

To enable a realistic tactile interaction with remote or virtual objects, softness information represents a fundamental property to be rendered via haptic devices. What is challenging is to reduce the complexity of such an information as it arises from contact mechanics and to find suitable simplifications that can lead an effective development of softness displays. A possible approach is to surrogate detailed tactile cues with information on the rate of spread of the contact area between the object and the finger as the contact force increases, i.e. force/area relation. This paradigm is called contact area spread rate. In this paper we discuss how such a paradigm has inspired the design of a tactile device (hereinafter referred to as Fabric Yielding Display, FYD-2), which exploits the elasticity of a fabric to mimic different levels of stiffness, while the contact area on the finger indenting the fabric is measured. In this manner, the FYD-2 can be controlled to reproduce force-area characteristics. In this work, we describe the FYD-2 architecture and report a psychophysical characterization. FYD-2 is shown to be able to accurately reproduce force-area curves of typical objects and to enable a reliable softness discrimination in human users.

Real-Time Task Discrimination for Myoelectric Control Employing Task-Specific Muscle Synergies

We present a novel formulation that employs task-specific muscle synergies and state-space representation of neural signals to tackle the challenging myoelectric control problem for lower arm prostheses. The proposed framework incorporates information about muscle configurations, e.g., muscles acting synergistically or in agonist/antagonist pairs, using the hypothesis of muscle synergies. The synergy activation coefficients are modeled as the latent system state and are estimated using a constrained Kalman filter. These task-dependent synergy activation coefficients are estimated in real-time from the electromyogram (EMG) data and are used to discriminate between various tasks. The task discrimination is helped by a post-processing algorithm that uses posterior probabilities. The proposed algorithm is robust as well as computationally efficient, yielding a decision with > 90% discrimination accuracy in approximately 3 ms . The real-time performance and controllability of the algorithm were evaluated using the targeted achievement control (TAC) test. The proposed algorithm outperformed common machine learning algorithms for single- as well as multi-degree-of-freedom (DOF) tasks in both off-line discrimination accuracy and real-time controllability (p < 0.01).