Analysis of human grasping behavior: object characteristics and grasp type

Electromyography and kinematics data of the hand in activities of daily living with special interest for ergonomics

This work presents a dataset of human hand kinematics and forearm muscle activation collected during the performance of a wide variety of activities of daily living (ADLs), with tagged characteristics of products and tasks. A total of 26 participants performed 161 ADLs selected to be representative of common elementary tasks, grasp types, product orientations and performance heights. 105 products were used, being varied regarding shape, dimensions, weight and type (common products and assistive devices). The data were recorded using CyberGlove instrumented gloves on both hands measuring 18 degrees of freedom on each and seven surface EMG sensors per arm recording muscle activity. Data of more than 4100 ADLs is presented in this dataset as MATLAB structures with full continuous recordings, which may be used in applications such as machine learning or to characterize healthy human hand behaviour. The dataset is accompanied with a custom data visualization application (ERGOMOVMUS) as a tool for ergonomics applications, allowing visualization and calculation of aggregated data from specific task, product and/or participants' characteristics.

Recognition of Grasping Patterns Using Deep Learning for Human-Robot Collaboration

Recent advances in the field of collaborative robotics aim to endow industrial robots with prediction and anticipation abilities. In many shared tasks, the robot's ability to accurately perceive and recognize the objects being manipulated by the human operator is crucial to make predictions about the operator's intentions. In this context, this paper proposes a novel learning-based framework to enable an assistive robot to recognize the object grasped by the human operator based on the pattern of the hand and finger joints. The framework combines the strengths of the commonly available software MediaPipe in detecting hand landmarks in an RGB image with a deep multi-class classifier that predicts the manipulated object from the extracted keypoints. This study focuses on the comparison between two deep architectures, a convolutional neural network and a transformer, in terms of prediction accuracy, precision, recall and F1-score. We test the performance of the recognition system on a new dataset collected with different users and in different sessions. The results demonstrate the effectiveness of the proposed methods, while providing valuable insights into the factors that limit the generalization ability of the models.

Development of Immersive Virtual Reality-Based Hand Rehabilitation System Using a Gesture-Controlled Rhythm Game With Vibrotactile Feedback: An fNIRS Pilot Study

Recently, virtua reality (VR) has been widely utilized with rehabilitation to promote user engagement, which has been shown to induce brain plasticity. In this study, we developed a VR-based hand rehabilitation system consisting of a personalized gesture-controlled rhythm game with vibrotactile feedback and investigated the cortical activation pattern induced by our system using functional near-infrared spectroscopy (fNIRS). Our system provides vibrotactile feedback as the user matches their hand gestures to VR targets customized to their pre-recorded hand gestures. Cortical activation was measured via fNIRS during 420 seconds of alternating gameplay and rest in 11 healthy subjects and one stroke survivor. Regions of interest (ROI) were the prefrontal cortex (PFC), the premotor cortex & the supplementary motor area (PMC&SMA), the primary sensorimotor cortex (SM1), and the somatosensory association cortex (SAC). The mean success rate of gesture matching among healthy subjects was 90 % with a standard deviation of 10.7 %, and the success rate of the stroke survivor was 79.6 %. The averaged cortical activation map for the 11 healthy subjects and the individual cortical activation map for the single stroke survivor showed increased hemodynamic responses of oxygenated hemoglobin (HbO) during the VR-based hand rehabilitation compared to the resting condition. Paired t-test analysis demonstrated a significant increase in HbO activation values in 19 out of 51 channels, corresponding to all ROIs except the left PFC and PMC&SMA, which exhibited high subject variability. The experimental results indicate that the proposed system successfully activated brain areas related to motor planning/execution, multisensory integration, and attention.

Functional MRI Assessment of Brain Activity During Hand Rehabilitation with an MR-Compatible Soft Glove in Chronic Stroke Patients: A Preliminary Study

Brain plasticity plays a significant role in functional recovery after stroke, but the specific benefits of hand rehabilitation robot therapy remain unclear. Evaluating the specific effects of hand rehabilitation robot therapy is crucial in understanding how it impacts brain activity and its relationship to rehabilitation outcomes. This study aimed to investigate the brain activity pattern during hand rehabilitation exercise using functional magnetic resonance imaging (fMRI), and to compare it before and after 3-week hand rehabilitation robot training. To evaluate it, an fMRI experimental environment was constructed to facilitate the same hand posture used in rehabilitation robot therapy. Two stroke survivors participated and the conjunction analysis results from fMRI scans showed that patient 1 exhibited a significant improvement in activation profile after hand rehabilitation robot training, indicative of improved motor function in the bilateral motor cortex. However, activation profile of patient 2 exhibited a slight decrease, potentially due to habituation to the rehabilitation task. Clinical results supported these findings, with patient 1 experiencing a greater increase in FMA score than patient 2. These results suggest that hand rehabilitation robot therapy can induce different brain activity patterns in stroke survivors, which may be linked to patient-specific training outcomes. Further studies with larger sample sizes are necessary to confirm these findings.

Object use in communication of semi-wild chimpanzees

Object interactions play an important role in human communication but the extent to which nonhuman primates incorporate objects in their social interactions remains unknown. To better understand the evolution of object use, this study explored how objects are used in social interactions in semi-wild chimpanzees (Pan troglodytes). We used an observational approach focusing on naturally occurring object actions where we examined their use and tested whether the production of object actions was influenced by the recipients' visual attention as well as by colony membership. The results show that chimpanzees adjusted both the type of object used, and the modality of object actions to match the visual attention of the recipient, as well as colony differences in the use of targeted object actions. These results provide empirical evidence highlighting that chimpanzees use objects in diverse ways to communicate with conspecifics and that their use may be shaped by social factors, contributing to our understanding of the evolution of human nonverbal communication, language, and tool use.

Advanced Stiffness Sensing through the Pincer Grasping of Soft Pneumatic Grippers

In this study, a comprehensive approach for sensing object stiffness through the pincer grasping of soft pneumatic grippers (SPGs) is presented. This study was inspired by the haptic sensing of human hands that allows us to perceive object properties through grasping. Many researchers have tried to imitate this capability in robotic grippers. The association between gripper performance and object reaction must be determined for this purpose. However, soft pneumatic actuators (SPA), the main components of SPGs, are extremely compliant. SPA compliance makes the determination of the association challenging. Methodologically, the connection between the behaviors of grasped objects and those of SPAs was clarified. A new concept of SPA modeling was then introduced. A method for stiffness sensing through SPG pincer grasping was developed based on this connection, and demonstrated on four samples. This method was validated through compression testing on the same samples. The results indicate that the proposed method yielded similar stiffness trends with slight deviations in compression testing. A main limitation in this study was the occlusion effect, which leads to dramatic deviations when grasped objects greatly deform. This is the first study to enable stiffness sensing and SPG grasping to be carried out in the same attempt. This study makes a major contribution to research on soft robotics by progressing the role of sensing for SPG grasping and object classification by offering an efficient method for acquiring another effective class of classification input. Ultimately, the proposed framework shows promise for future applications in inspecting and classifying visually indistinguishable objects.

Slip Detection Strategies for Automatic Grasping in Prosthetic Hands

The detection of an object slipping within the grasp of a prosthetic hand enables the hand to react to ensure the grasp is stable. The computer controller of a prosthetic hand needs to be able to unambiguously detect the slide from other signals. Slip can be detected from the surface vibrations made as the contact between object and terminal device shifts. A second method measures the changes in the normal and tangential forces between the object and the digits. After a review of the principles of how the signals are generated and the detection technologies are employed, this paper details the acoustic and force sensors used in versions of the Southampton Hand. Attention is given to the techniques used in the field. The performance of the Southampton tube sensor is explored. Different surfaces are slid past a sensor and the signals analysed. The resulting signals have low-frequency content. The signals are low pass filtered and the resulting processing results in a consistent response across a range of surfaces. These techniques are fast and not computationally intensive, which makes them practical for a device that is to be used daily in the field.

Portable 3D-printed hand orthosis with spatial stiffness distribution personalized for assisting grasping in daily living

Stroke survivors having limited finger coordination require an active hand orthosis to assist them with grasping tasks for daily activities. The orthosis should be portable for constant use; however, portability imposes constraints on the number, size, and weight of the actuators, which increase the difficulty of the design process. Therefore, a tradeoff exists between portability and the assistive force. In this study, a personalized spatial stiffness distribution design is presented for a portable and strengthful hand orthosis. The spatial stiffness distribution of the orthosis was optimized based on measurements of individual hand parameters to satisfy the functional requirements of achieving sufficient grip aperture in the pre-grasping phase and minimal assistive force in the grasping phase. Ten stroke survivors were recruited to evaluate the system. Sufficient grip aperture and high grip strength-to-weight ratio were achieved by the orthosis via a single motor. Moreover, the orthosis significantly restored the range of motion and improved the performance of daily activities. The proposed spatial stiffness distribution can suggest a design solution to make strengthful hand orthoses with reduced weight.

Current Designs of Robotic Arm Grippers: A Comprehensive Systematic Review

Recent technological advances enable gripper-equipped robots to perform many tasks traditionally associated with the human hand, allowing the use of grippers in a wide range of applications. Depending on the application, an ideal gripper design should be affordable, energy-efficient, and adaptable to many situations. However, regardless of the number of grippers available on the market, there are still many tasks that are difficult for grippers to perform, which indicates the demand and room for new designs to compete with the human hand. Thus, this paper provides a comprehensive review of robotic arm grippers to identify the benefits and drawbacks of various gripper designs. The research compares gripper designs by considering the actuation mechanism, degrees of freedom, grasping capabilities with multiple objects, and applications, concluding which should be the gripper design with the broader set of capabilities.

Kinematic analysis of impairments and compensatory motor behavior during prosthetic grasping in below-elbow amputees

After a major upper limb amputation, the use of myoelectric prosthesis as assistive devices is possible. However, these prostheses remain quite difficult to control for grasping and manipulation of daily life objects. The aim of the present observational case study is to document the kinematics of grasping in a group of 10 below-elbow amputated patients fitted with a myoelectric prosthesis in order to describe and better understand their compensatory strategies. They performed a grasping to lift task toward 3 objects (a mug, a cylinder and a cone) placed at two distances within the reaching area in front of the patients. The kinematics of the trunk and upper-limb on the non-amputated and prosthetic sides were recorded with 3 electromagnetic Polhemus sensors placed on the hand, the forearm (or the corresponding site on the prosthesis) and the ipsilateral acromion. The 3D position of the elbow joint and the shoulder and elbow angles were calculated thanks to a preliminary calibration of the sensor position. We examined first the effect of side, distance and objects with non-parametric statistics. Prosthetic grasping was characterized by severe temporo-spatial impairments consistent with previous clinical or kinematic observations. The grasping phase was prolonged and the reaching and grasping components uncoupled. The 3D hand displacement was symmetrical in average, but with some differences according to the objects. Compensatory strategies involved the trunk and the proximal part of the upper-limb, as shown by a greater 3D displacement of the elbow for close target and a greater forward displacement of the acromion, particularly for far targets. The hand orientation at the time of grasping showed marked side differences with a more frontal azimuth, and a more "thumb-up" roll. The variation of hand orientation with the object on the prosthetic side, suggested that the lack of finger and wrist mobility imposed some adaptation of hand pose relative to the object. The detailed kinematic analysis allows more insight into the mechanisms of the compensatory strategies that could be due to both increased distal or proximal kinematic constraints. A better knowledge of those compensatory strategies is important for the prevention of musculoskeletal disorders and the development of innovative prosthetics.

Design and simulation of a gripper structure of cluster tomato based on manual picking behavior

Picking robot technology plays an important role in the rapid promotion of precision agriculture. The development of a successful robot gripper is critical for the final promotion and industrialization of the tomato picking robot. This paper investigated the cluster tomato picking strategy and the gripper structure design based on this strategy to address the problem of poor adaptability of the existing gripper design in the cluster tomato picking scene. Starting from the research on the behavior of artificially picking cluster tomatoes, the grasping method, finger structure parameters and picking movement pattern of the human hand are analyzed. The evaluation criteria of the gripper are summarized, a simplified mathematical model of the gripper is established, and the picking strategy under the model of the gripper is proposed. Furthermore, according to the simplified gripper model, a rigid-flexible coupling gripper structure is designed, and the gripping simulation analysis is carried out. According to the simulation results, the gripper can smoothly grab medium and large tomatoes with diameter of 65∼95 mm. The peak force and fluctuation force of tomato with different sizes during grasping were less than the tomato's minimum damage force. The gripper has adaptability and stability characteristics, providing technical support for gripper manufacturing and the construction of a picking system for a tomato picking robot.

An Accessible, Open-Source Dexterity Test: Evaluating the Grasping and Dexterous Manipulation Capabilities of Humans and Robots

Evaluating the dexterity of human and robotic hands through appropriate benchmarks, scores, and metrics is of paramount importance for determining how skillful humans are and for designing and developing new bioinspired or even biomimetic end-effectors (e.g., robotic grippers and hands). Dexterity tests have been used in industrial and medical settings to assess how dexterous the hands of workers and surgeons are as well as in robotic rehabilitation settings to determine the improvement or deterioration of the hand function after a stroke or a surgery. In robotics, having a comprehensive dexterity test can allow us to evaluate and compare grippers and hands irrespectively of their design characteristics. However, there is a lack of well defined metrics, benchmarks, and tests that quantify robot dexterity. Previous work has focused on a number of widely accepted functional tests that are used for the evaluation of manual dexterity and human hand function improvement post injury. Each of these tests focuses on a different set of specific tasks and objects. Deriving from these tests, this work proposes a new modular, affordable, accessible, open-source dexterity test for both humans and robots. This test evaluates the grasping and manipulation capabilities by combining the features and best practices of the aforementioned tests, as well as new task categories specifically designed to evaluate dexterous manipulation capabilities. The dexterity test and the accompanying benchmarks allow us to determine the overall hand function recovery and dexterity of robotic end-effectors with ease. More precisely, a dexterity score that ranges from 0 (simplistic, non-dexterous system) to 1 (human-like system) is calculated using the weighted sum of the accuracy and task execution speed subscores. It should also be noted that the dexterity of a robotic system can be evaluated assessing the efficiency of either the robotic hardware, or the robotic perception system, or both. The test and the benchmarks proposed in the study have been validated using extensive human and robot trials. The human trials have been used to determine the baseline scores for the evaluation system. The results show that the time required to complete the tasks reduces significantly with trials indicating a clear learning curve in mastering the dexterous manipulation capabilities associated with the imposed tasks. Finally, the time required to complete the tasks with restricted tactile feedback is significantly higher indicating its importance.

A straightforward and miniature implementation method of postural synergies to replicate human grasp characteristics accurately and intuitively

Postural synergies have great potential for replicating human grasp characteristics, simplify grasp control and reduce the number of hardware actuators required. However, due to their complex mapping relationship and jagged transmission ratio, the implemented mechanisms are always too bulky and loose, which greatly limits their application. With current solutions, the replication accuracy of motion characteristics or intuitive control is compromised, and hitherto no work in the literature has reported the replication errors. To overcome these limitations, we present a novel design framework to determine the actuation configuration, implementation scheme and physical parameters. In this way, the mechanism is miniaturized and can be compactly embedded in the palm of the hand. A self-contained synergistic robot hand with integrated mechanism, sensors and a suitable electrical system is built. The experiments demonstrate that the robot hand can accurately replicate the motion characteristics of two primary synergies, maintain intuitive control to simplify grasp control, has a good capability for anthropomorphic motion and can grasp different objects with versatile grasp functionality.

Marine Robotics for Deep-Sea Specimen Collection: A Taxonomy of Underwater Manipulative Actions

In order to develop a gripping system or control strategy that improves scientific sampling procedures, knowledge of the process and the consequent definition of requirements is fundamental. Nevertheless, factors influencing sampling procedures have not been extensively described, and selected strategies mostly depend on pilots' and researchers' experience. We interviewed 17 researchers and remotely operated vehicle (ROV) technical operators, through a formal questionnaire or in-person interviews, to collect evidence of sampling procedures based on their direct field experience. We methodologically analyzed sampling procedures to extract single basic actions (called atomic manipulations). Available equipment, environment and species-specific features strongly influenced the manipulative choices. We identified a list of functional and technical requirements for the development of novel end-effectors for marine sampling. Our results indicate that the unstructured and highly variable deep-sea environment requires a versatile system, capable of robust interactions with hard surfaces such as pushing or scraping, precise tuning of gripping force for tasks such as pulling delicate organisms away from hard and soft substrates, and rigid holding, as well as a mechanism for rapidly switching among external tools.

Reducing Anthropomorphic Hand Degrees of Actuation with Grasp-Function-Dependent and Joint-Element-Sparse Hand Synergies

In this paper, a set of grasp-function-dependent and joint-element-sparse hand synergies was proposed. First, hand synergies were extracted from five basic categories of movements by principal component analysis (PCA). Then, varimax rotation was applied on these synergies, so each sparse synergy only represented a limited number of joints. Next, according to the contribution to these sparse synergies, finger joints were clustered into different joint modules. Finally, integrating the joint modules in different categories of hand movements, the minimum number of actuators and joint synergic modules for anthropomorphic hands were determined. The results showed that using 5 groups of joint modules and 7–9 actuators we can achieve the best performance of grasp function and motion flexibility. Furthermore, through the reasonable design of adaptive and hyperextension functional joint modules, anthropomorphic hands can better meet the requirements of different tasks like power grasping and precision pinching. Comparing with traditional finger-based actuation strategy, the joint coupling scheme achieved better anthropomorphic performance and larger workspace. These above findings will benefit the development of mechanical structure design and control method of anthropomorphic hands.

Quantitative Investigation of Hand Grasp Functionality: Hand Joint Motion Correlation, Independence, and Grasping Behavior

Modeling and understanding human grasp functionality are fundamental in prosthetics, robotics, medicine, and rehabilitation, since they contribute to exploring motor control mechanism, evaluating grasp function, and designing and controlling prosthetic hands or exoskeletons. However, there are still limitations in providing a comprehensive and quantitative understanding of hand grasp functionality. After simultaneously considering three significant and essential influence factors in daily grasping contained relative position, object shape, and size, this paper presents the tolerance grasping to provide a more comprehensive understanding of human grasp functionality. The results of joint angle distribution and variance explained by PCs supported that tolerance grasping can represent hand grasp functionality more comprehensively. Four synergies are found and account for 93% ± 1.5% of the overall variance. The ANOVA confirmed that there was no significant individual difference in the first four postural synergies. The common patterns of grasping behavior were found and characterized by the mean value of postural synergy across 10 subjects. The independence analysis demonstrates that the tolerance grasping results highly correlate with unstructured natural grasping and more accurately correspond to cortical representation size of finger movement. The potential for exploring the neuromuscular control mechanism of human grasping is discussed. The analysis of hand grasp characteristics that contained joint angle distribution, correlation, independence, and postural synergies, presented here, should be more representative to provide a more comprehensive understanding of hand grasp functionality.

Quantitative Investigation of Hand Grasp Functionality: Thumb Grasping Behavior Adapting to Different Object Shapes, Sizes, and Relative Positions

This paper is the first in the two-part series quantitatively modelling human grasp functionality and understanding the way human grasp objects. The aim is to investigate the thumb movement behavior influenced by object shapes, sizes, and relative positions. Ten subjects were requested to grasp six objects (3 shapes × 2 sizes) in 27 different relative positions (3 X deviation × 3 Y deviation × 3 Z deviation). Thumb postures were investigated to each specific joint. The relative position (X, Y, and Z deviation) significantly affects thumb opposition rotation (Rot) and flexion (interphalangeal (IP) and metacarpo-phalangeal (MCP)), while the object property (object shape and size) significantly affects thumb abduction/adduction (ABD) motion. Based on the F value, the Y deviation has the primary effects on thumb motion. When the Y deviation changing from proximal to distal, thumb opposition rotation (Rot) and flexion (IP and MCP joint) angles were increased and decreased, respectively. For principal component analysis (PCA) results, thumb grasp behavior can be accurately reconstructed by first two principal components (PCs) which variance explanation ratio reached 93.8% and described by the inverse and homodromous coordination movement between thumb opposition and IP flexion. This paper provides a more comprehensive understanding of thumb grasp behavior. The postural synergies can reproduce the anthropomorphic motion, reduce the robot hardware, and control dimensionality. All of these provide a more accurate and general basis for the design and control of the bionic thumb and novel wearable assistant robot, thumb function assessment, and rehabilitation.

Learning Grasp Configuration Through Object-Specific Hand Primitives for Posture Planning of Anthropomorphic Hands

The proposal of postural synergy theory has provided a new approach to solve the problem of controlling anthropomorphic hands with multiple degrees of freedom. However, generating the grasp configuration for new tasks in this context remains challenging. This study proposes a method to learn grasp configuration according to the shape of the object by using postural synergy theory. By referring to past research, an experimental paradigm is first designed that enables the grasping of 50 typical objects in grasping and operational tasks. The angles of the finger joints of 10 subjects were then recorded when performing these tasks. Following this, four hand primitives were extracted by using principal component analysis, and a low-dimensional synergy subspace was established. The problem of planning the trajectories of the joints was thus transformed into that of determining the synergy input for trajectory planning in low-dimensional space. The average synergy inputs for the trajectories of each task were obtained through the Gaussian mixture regression, and several Gaussian processes were trained to infer the inputs trajectories of a given shape descriptor for similar tasks. Finally, the feasibility of the proposed method was verified by simulations involving the generation of grasp configurations for a prosthetic hand control. The error in the reconstructed posture was compared with those obtained by using postural synergies in past work. The results show that the proposed method can realize movements similar to those of the human hand during grasping actions, and its range of use can be extended from simple grasping tasks to complex operational tasks.

Bioinspired High-Degrees of Freedom Soft Robotic Glove for Restoring Versatile and Comfortable Manipulation

The human hand is one of the most complex and compact grippers that has arisen as a product of natural genetic engineering; it is highly versatile, as it handles power and precision tasks. Since proper contact points and force directions are required to ensure versatility and secure a stable grip on an object, there must be a large workspace and controllable tip force directions for the digits. Although they are important, many individuals with neuromuscular diseases experience loss of these features. Thus, we propose a high-degree-of-freedom (DOF) soft robotic glove inspired by the anatomical features of human hands. The mechanism for adjusting the position and force direction of each tip is based on the structure of the extrinsic and intrinsic muscle-tendon units. The large thumb workspace was achieved by assisting opposition/reposition and flexion/extension to enable various grasping postures. A bidirectional actuation control mechanism with a cable-actuated agonist and an elastomer antagonist increased the assisted DOF and maintained compactness. The kinematic and kinetic performances of our device were evaluated by performing tests with eight stroke survivors. The thumb workspace increased by 43%, 207%, and 248% in the distal-proximal, dorsal-palmar, and radial-ulnar directions, respectively. The pinching shear force decreased by 54% and 45% for the nonthumb digits and thumb, respectively. These device-assisted improvements allowed objects to be stably grasped and manipulated in various postures. The novel device can assist individuals with impaired hand function to improve their grasping performance. Clinical Research Information Service (CRIS) Registration Number: KCT0004855.

A Systematic Analysis of Hand Movement Functionality: Qualitative Classification and Quantitative Investigation of Hand Grasp Behavior

Understanding human hand movement functionality is fundamental in neuroscience, robotics, prosthetics, and rehabilitation. People are used to investigate movement functionality separately from qualitative or quantitative perspectives. However, it is still limited to providing an integral framework from both perspectives in a logical manner. In this paper, we provide a systematic framework to qualitatively classify hand movement functionality, build prehensile taxonomy to explore the general influence factors of human prehension, and accordingly design a behavioral experiment to quantitatively understand the hand grasp. In qualitative analysis, two facts are explicitly proposed: (1) the arm and wrist make a vital contribution to hand movement functionality; (2) the relative position (relative position in this paper is defined as the distance between the center of the human wrist and the object center of gravity) is a general influence factor significantly impacting human prehension. In quantitative analysis, the significant influence of three factors, object shape, size, and relative position, is quantitatively demonstrated. Simultaneously considering the impact of relative position, object shape, and size, the prehensile taxonomy and behavioral experiment results presented here should be more representative and complete to understand human grasp functionality. The systematic framework presented here is general and applicable to other body parts, such as wrist, arm, etc. Finally, many potential applications and the limitations are clarified.

On the choice of grasp type and location when handing over an object

The human hand is capable of performing countless grasps and gestures that are the basis for social activities. However, which grasps contribute the most to the manipulation skills needed during collaborative tasks, and thus which grasps should be included in a robot companion, is still an open issue. Here, we investigated grasp choice and hand placement on objects during a handover when subsequent tasks are performed by the receiver and when in-hand and bimanual manipulation are not allowed. Our findings suggest that, in this scenario, human passers favor precision grasps during such handovers. Passers also tend to grasp the purposive part of objects and leave "handles" unobstructed to the receivers. Intuitively, this choice allows receivers to comfortably perform subsequent tasks with the objects. In practice, many factors contribute to a choice of grasp, e.g., object and task constraints. However, not all of these factors have had enough emphasis in the implementation of grasping by robots, particularly the constraints introduced by a task, which are critical to the success of a handover. Successful robotic grasping is important if robots are to help humans with tasks. We believe that the results of this work can benefit the wider robotics community, with applications ranging from industrial cooperative manipulation to household collaborative manipulation.

Effect of object and human-factor characteristics on the preference of thumb-index finger grasp type

This work is to investigate the factors affecting the preference of human thumb-index finger grasping type. A multinomial logistic regression analysis shown that the object characteristics (equivalent diameter and shape) and human-factor characteristics (hand-used, finger-length sum and finger-length ratio) had significant contributions on the preference of thumb-index finger grasp type (p < 0.05) but the gender had not (p > 0.05). Subsequently, two mathematical equations were proposed for predicting the probability at which the precision-pinch and power-grasp were chosen for grasping an object. The probability at which the precision-pinch was chosen gradually decreased with the increase in the equivalent diameter of objects, but it is opposite for the power-grasp case. The shorter the finger-length sum, the more likely the participant was to select the power-grasp for grasping an object compared to the precision-pinch. The power-grasp was the most frequently chosen for the finger-length ratios of 1.0-1.25 and 1.75-2.0. Practitioner summary: This fruitful study gave explanation of the relationship between the object and human-factor characteristics and the preference of human thumb-index finger grasp type, which would be helpful to make intelligent grasping planning strategies for two-finger bionic mechanical hands.

The Grasp Strategy of a Robot Passer Influences Performance and Quality of the Robot-Human Object Handover

Task-aware robotic grasping is critical if robots are to successfully cooperate with humans. The choice of a grasp is multi-faceted; however, the task to perform primes this choice in terms of hand shaping and placement on the object. This grasping strategy is particularly important for a robot companion, as it can potentially hinder the success of the collaboration with humans. In this work, we investigate how different grasping strategies of a robot passer influence the performance and the perceptions of the interaction of a human receiver. Our findings suggest that a grasping strategy that accounts for the subsequent task of the receiver improves substantially the performance of the human receiver in executing the subsequent task. The time to complete the task is reduced by eliminating the need of a post-handover re-adjustment of the object. Furthermore, the human perceptions of the interaction improve when a task-oriented grasping strategy is adopted. The influence of the robotic grasp strategy increases as the constraints induced by the object's affordances become more restrictive. The results of this work can benefit the wider robotics community, with application ranging from industrial to household human-robot interaction for cooperative and collaborative object manipulation.

Analysis of subject specific grasping patterns

Existing haptic feedback devices are limited in their capabilities and are often cumbersome and heavy. In addition, these devices are generic and do not adapt to the users’ grasping behavior. Potentially, a human-oriented design process could generate an improved design. While current research done on human grasping was aimed at finding common properties within the research population, we investigated the dynamic patterns that make human grasping behavior distinct rather than generalized, i.e. subject specific. Experiments were conducted on 31 subjects who performed grasping tasks on five different objects. The kinematics and kinetics parameters were measured using a motion capture system and force sensors. The collected data was processed through a pipeline of dimensionality reduction and clustering algorithms. Using finger joint angles and reaction forces as our features, we were able to classify these tasks with over 95% success. In addition, we examined the effects of the objects’ mechanical properties on those patterns and the significance of the different features for the differentiation. Our results suggest that grasping patterns are, indeed, subject-specific; this, in turn, could suggest that a device capable of providing personalized feedback can improve the user experience and, in turn, increase the usability in different applications. This paper explores an undiscussed aspect of human dynamic patterns. Furthermore, the collected data offer a valuable dataset of human grasping behavior, containing 1083 grasp instances with both kinetics and kinematics data.

Evaluation of Drop Shadows for Virtual Object Grasping in Augmented Reality

This article presents the use of rendered visual cues as drop shadows and their impact on overall usability and accuracy of grasping interactions for monitor-based exocentric augmented reality (AR). We report on two conditions, grasping with drop shadows and without drop shadows, and analyze a total of 1620 grasps of two virtual object types (cubes and spheres). We report on the accuracy of one grasp type, the Medium Wrap grasp, against Grasp Aperture ($GAp$GAp), Grasp Displacement ($GDisp$GDisp), completion time, and usability metrics from 30 participants. A comprehensive statistical analysis of the results is presented giving comparisons of the inclusion of drop shadows in AR grasping. Findings showed that the use of drop shadows increases usability of AR grasping while significantly decreasing task completion times. Furthermore, drop shadows also significantly improve user's depth estimation of AR object position. However, this study also shows that using drop shadows does not improve user's object size estimation, which remains as a problematic element in grasping AR interaction literature.

An inverse optimization approach to understand human acquisition of kinematic coordination in bimanual fine manipulation tasks

Tasks that require the cooperation of both hands and arms are common in human everyday life. Coordination helps to synchronize in space and temporally motion of the upper limbs. In fine bimanual tasks, coordination enables also to achieve higher degrees of precision that could be obtained from a single hand. We studied the acquisition of bimanual fine manipulation skills in watchmaking tasks, which require assembly of pieces at millimeter scale. It demands years of training. We contrasted motion kinematics performed by novice apprentices to those of professionals. Fifteen subjects, ten novices and five experts, participated in the study. We recorded force applied on the watch face and kinematics of fingers and arms. Results indicate that expert subjects wisely place their fingers on the tools to achieve higher dexterity. Compared to novices, experts also tend to align task-demanded force application with the optimal force transmission direction of the dominant arm. To understand the cognitive processes underpinning the different coordination patterns across experts and novice subjects, we followed the optimal control theoretical framework and hypothesize that the difference in task performances is caused by changes in the central nervous system's optimal criteria. We formulated kinematic metrics to evaluate the coordination patterns and exploit inverse optimization approach to infer the optimal criteria. We interpret the human acquisition of novel coordination patterns as an alteration in the composition structure of the central nervous system's optimal criteria accompanied by the learning process.

A Machine-Learning-Based Approach to Solve Both Contact Location and Force in Soft Material Tactile Sensors

This study addresses a design and calibration methodology based on numerical finite element method (FEM) modeling for the development of a soft tactile sensor able to simultaneously solve the magnitude and the application location of a normal load exerted onto its surface. The sensor entails the integration of a Bragg grating fiber optic sensor in a Dragon Skin 10 polymer brick (110 mm length, 24 mm width). The soft polymer mediates the transmission of the applied load to the buried fiber Bragg gratings (FBGs), and we also investigated the effect of sensor thickness on receptive field and sensitivity, both with the developed model and experimentally. Force-controlled indentations of the sensor (up to 2.5 N) were carried out through a cylindrical probe applied along the direction of the optical fiber (over an ∼90 mm span in length). A finite element model of the sensor was built and experimentally validated for 1 and 6 mm thicknesses of the soft polymeric encapsulation material, considering that the latter thickness resulted from numerical simulations as leading to optimal cross talk and sensitivity, given the chosen soft material. The FEM model was also used to train a neural network so as to obtain the inverse sensor function. Using four FBG transducers embedded in the 6-mm-thick soft polymer, the proposed machine learning approach managed to accurately detect both load magnitude (R = 0.97) and location (R = 0.99) over the whole experimental range. The proposed system could be used for developing tactile sensors that can be effectively used for a broad range of applications.

Investigation on the Cooperative Grasping Capabilities of Human Thumb and Index Finger

The maximum cooperative grasping mass and diameter of the human thumb and index finger were investigated by 7560 grasp-release trials on various masses of solid cylinders and various sizes of rings. The maximum grasping mass of the participants’ thumb-index finger depended on gender, age and the sum of thumb-index finger lengths (P < 0.05), but not on the hand-used and ratio of index finger to thumb length (P > 0.05). The maximum grasping diameter of the participants’ thumb-index finger depended on the age, sum of thumb-index finger lengths and ratio of index finger to thumb length (P < 0.05), but not on the gender and hand-used (P > 0.05). There was a non-linear regression model for the dependence of the maximum grasping mass on gender, age and the sum of thumb-index finger lengths and another non-linear regression model for the dependence of the maximum grasping diameter on the age, sum of thumb-index finger lengths and ratio of index finger to thumb length. Two regression models were useful in the optimal size design of robotic hands intending to replicate thumb-index finger grasping ability. This research can help to define not only a reasonable grasp mass and size for a bionic robotic hand, but also the requirements for hand rehabilitation.

Flexible Tactile Sensor Array for Slippage and Grooved Surface Recognition in Sliding Movement

Flexible tactile sensor with contact force sensing and surface texture recognition abilities is crucial for robotic dexterous grasping and manipulation in daily usage. Different from force sensing, surface texture discrimination is more challenging in the development of tactile sensors because of limited discriminative information. This paper presents a novel method using the finite element modeling (FEM) and phase delay algorithm to investigate the flexible tactile sensor array for slippage and grooved surfaces discrimination when sliding over an object. For FEM modeling, a 3 × 3 tactile sensor array with a multi-layer structure is utilized. For sensor array sliding over a plate surface, the initial slippage occurrence can be identified by sudden changes in normal forces based on wavelet transform analysis. For the sensor array sliding over pre-defined grooved surfaces, an algorithm based on phase delay between different sensing units is established and then utilized to discriminate between periodic roughness and the inclined angle of the grooved surfaces. Results show that the proposed tactile sensor array and surface texture recognition method is anticipated to be useful in applications involving human-robotic interactions.

Passive Force-Feedback Gloves With Joint-Based Variable Impedance Using Layer Jamming

Force feedback gloves have a great potential in enhancing the fidelity of virtual reality and teleoperation systems. It is a challenge to develop multifinger and lightweight force feedback gloves. In this paper, we propose a solution using layer jamming sheet (LJS) on each finger joint. In simulating free space, the LJS is soft and easy to deform, which allows the finger joints to move freely with a small resistance force. In simulating constrained space, the LJS becomes stiff, which provides resistance torques to prevent the rotation of finger joints. Possible solutions for mounting the LJS on finger joints are investigated. Mechanical models of the LJS are derived by quantifying the relationship between the bending stiffness and the pressure, material, and geometry of the layer. Experiments are performed to characterize the mechanical behavior of the LJS actuator and to validate the performance of the different design solutions in simulating free space and constrained space. Experimental results indicate the potential of the proposed joint-based LJS-actuated approach in developing lightweight force feedback gloves.

Wearable Nail Deformation Sensing for Behavioral and Biomechanical Monitoring and Human-Computer Interaction

The dynamics of the human fingertip enable haptic sensing and the ability to manipulate objects in the environment. Here we describe a wearable strain sensor, associated electronics, and software to detect and interpret the kinematics of deformation in human fingernails. Differential forces exerted by fingertip pulp, rugged connections to the musculoskeletal system and physical contact with the free edge of the nail plate itself cause fingernail deformation. We quantify nail warpage on the order of microns in the longitudinal and lateral axes with a set of strain gauges attached to the nail. The wearable device transmits raw deformation data to an off-finger device for interpretation. Simple motions, gestures, finger-writing, grip strength, and activation time, as well as more complex idioms consisting of multiple grips, are identified and quantified. We demonstrate the use of this technology as a human-computer interface, clinical feature generator, and means to characterize workplace tasks.

The DEKA hand: A multifunction prosthetic terminal device-patterns of grip usage at home

BACKGROUND

Research is needed to understand how upper limb prosthesis users take advantage of multiple grip options.

OBJECTIVES

To quantify usage of DEKA hand grip patterns during home use and compare patterns of usage at home to test sessions.

STUDY DESIGN

Observational study design.

METHODS

Data were collected from 21 subjects. Engineering data on grip were downloaded at various intervals. Proportion of time in each grip was calculated for the first 4 weeks of home use, later months, and test sessions (testing use) and compared statistically across intervals. Exploratory analyses compared grip proportion by DEKA Arm level and prior prosthesis use.

RESULTS

Three most commonly used grips during home use were power, pinch open, and lateral pinch. There were no significant differences between grip use during the first month and later months. Power grip was used 55% of the time at home and 23% of the time in testing use. Pinch closed, lateral, and chuck grip were used less at home than in tests. Comparisons were by configuration level and prosthetic use and no significant differences were found.

CONCLUSION

Patterns of DEKA hand grip usage differed between home and test environments, suggesting that users relied on fewer grip patterns at home. Clinical relevance These findings have implications for prosthetic training with multi-articulating terminal devices.

Effect of Object and Task Properties on Bimanual Transport

The use of both hands simultaneously when manipulating objects is fairly commonplace, but it is not known what factors encourage people to use two hands as opposed to one during simple tasks such as transport. In particular, we are interested in three possible transport strategies: unimanual transport, handing off between hands, and symmetric bimanual transport. In this study, we investigate the effect of object size, weight, and starting and ending position (configuration) as well as the need to balance the object on the use of these three strategies in a bowl-moving task. We find that configuration and balance have a strong effect on choice of strategy, and size and weight have a weaker effect. Hand-offs are most often used when the task requires moving an object from left to right and vice versa, while the unimanual strategy was frequently used when passing front to back. The bimanual strategy is only weakly affected by configuration. The need to balance an object causes subjects to favor unimanual and bimanual strategies over the hand-off. In addition, an analysis of transport duration and body rotation suggests that strategy choice may be driven by the desire to minimize body rotation.

Analyzing at-home prosthesis use in unilateral upper-limb amputees to inform treatment & device design

New upper limb prosthetic devices are continuously being developed by a variety of industrial, academic, and hobbyist groups. Yet, little research has evaluated the long term use of currently available prostheses in daily life activities, beyond laboratory or survey studies. We seek to objectively measure how experienced unilateral upper limb prosthesis-users employ their prosthetic devices and unaffected limb for manipulation during everyday activities. In particular, our goal is to create a method for evaluating all types of amputee manipulation, including non-prehensile actions beyond conventional grasp functions, as well as to examine the relative use of both limbs in unilateral and bilateral cases. This study employs a head-mounted video camera to record participant's hands and arms as they complete unstructured domestic tasks within their own homes. A new 'Unilateral Prosthesis-User Manipulation Taxonomy' is presented based observations from 10 hours of recorded videos. The taxonomy addresses manipulation actions of the intact hand, prostheses, bilateral activities, and environmental feature-use (aiïordances). Our preliminary results involved tagging 23 minute segments of the full videos from 3 amputee participants using the taxonomy. This resulted in over 2,300 tag instances. Observations included that non-prehensile interactions outnumbered prehensile interactions in the affected limb for users with more distal amputation that allowed arm mobility.

Fingertip-Based Feature Analysis for the Push and Stroke Manipulation of Elastic Objects

In this study, to quantitatively understand finger operations used to manipulate elastic objects, we explore robust fingertip-based feature descriptors that are invariant to operator, finger position, and target object. To measure the tactile information generated when an object is directly touched by a fingertip, we used a wearable system that enables the simultaneous measurement of fingertip position and strain without inhibiting the operator's sense of touch. This paper focuses on the quantitative classification of the push and stroke operations of a single finger, and conducted user experiments to obtain time-series fingertip position and strain from 10 subjects touching nine types of elastic objects. The recognition rate was investigated by binary classification using a support vector machine and cross validation. The results show that the two-dimensional features obtained from fingertip position and strain within a 0.9-s time frame can stably recognize push and stroke operations on elastic bodies of different shapes, stiffnesses, and thicknesses at a higher recognition rate.

Form and function - Exemplary analysis of the significance for the design of rehabilitation devices

Stroke often leads to motor impairment that could be recovered by extensive training. Multiple devices exist to support the rehabilitation process. Most systems interact directly with the patient and therefore, depend strongly on the ergonomic design which is determined by its form and function. In this work the interdependence of form and function and their particular significance for the development of rehabilitation devices are outlined. As a case study the development of a hand rehabilitation device is presented, where two approaches answering different key questions to focus either on function or form were realized at the same time to generate different concepts. The function-oriented approach led to a concept based on linkages and the form-oriented approach to one using leaf springs. In the discussion, the differences between the approaches are analyzed in respect to the creation of a geometrical-material entireness. Furthermore, new findings are discussed and the integration of both concepts into a final prototype is shown.

Exploration of Force Myography and surface Electromyography in hand gesture classification

Whereas pressure sensors increasingly have received attention as a non-invasive interface for hand gesture recognition, their performance has not been comprehensively evaluated. This work examined the performance of hand gesture classification using Force Myography (FMG) and surface Electromyography (sEMG) technologies by performing 3 sets of 48 hand gestures using a prototyped FMG band and an array of commercial sEMG sensors worn both on the wrist and forearm simultaneously. The results show that the FMG band achieved classification accuracies as good as the high quality, commercially available, sEMG system on both wrist and forearm positions; specifically, by only using 8 Force Sensitive Resisters (FSRs), the FMG band achieved accuracies of 91.2% and 83.5% in classifying the 48 hand gestures in cross-validation and cross-trial evaluations, which were higher than those of sEMG (84.6% and 79.1%). By using all 16 FSRs on the band, our device achieved high accuracies of 96.7% and 89.4% in cross-validation and cross-trial evaluations.

Analysis of Hand and Wrist Postural Synergies in Tolerance Grasping of Various Objects

Human can successfully grasp various objects in different acceptable relative positions between human hand and objects. This grasp functionality can be described as the grasp tolerance of human hand, which is a significant functionality of human grasp. To understand the motor control of human hand completely, an analysis of hand and wrist postural synergies in tolerance grasping of various objects is needed. Ten healthy right-handed subjects were asked to perform the tolerance grasping with right hand using 6 objects of different shapes, sizes and relative positions between human hand and objects. Subjects were wearing CyberGlove attaching motion tracker on right hand, allowing a measurement of the hand and wrist postures. Correlation analysis of joints and inter-joint/inter-finger modules were carried on to explore the coordination between joints or modules. As the correlation between hand and wrist module is not obvious in tolerance grasping, individual analysis of wrist synergies would be more practical. In this case, postural synergies of hand and wrist were then presented separately through principal component analysis (PCA), expressed through the principal component (PC) information transmitted ratio, PC elements distribution and reconstructed angle error of joints. Results on correlation comparison of different module movements can be well explained by the influence factors of the joint movement correlation. Moreover, correlation analysis of joints and modules showed the wrist module had the lowest correlation among all inter-finger and inter-joint modules. Hand and wrist postures were both sufficient to be described by a few principal components. In terms of the PC elements distribution of hand postures, compared with previous investigations, there was a greater proportion of movement in the thumb joints especially the interphalangeal (IP) and opposition rotation (ROT) joint. The research could serve to a complete understanding of hand grasp, and the design, control of the anthropomorphic hand and wrist.

Automatic Grasp Selection using a Camera in a Hand Prosthesis

In this paper, we demonstrate how automatic grasp selection can be achieved by placing a camera in the palm of a prosthetic hand and training a convolutional neural network on images of objects with corresponding grasp labels. Our labeled dataset is built from common graspable objects curated from the ImageNet dataset and from images captured from our own camera that is placed in the hand. We achieve a grasp classification accuracy of 93.2% and show through real-time grasp selection that using a camera to augment current electromyography controlled prosthetic hands may be useful.

The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments

This paper presents a dataset of human grasping behavior in unstructured environments. Wide-angle head-mounted camera video was recorded from two housekeepers and two machinists during their regular work activities, and the grasp types, objects, and tasks were analyzed and coded by study staff. The full dataset contains 27.7 hours of tagged video and represents a wide range of manipulative behaviors spanning much of the typical human hand usage. We provide the original videos, a spreadsheet including the tagged grasp type, object, and task parameters, time information for each successive grasp, and video screenshots for each instance. Example code is provided for MATLAB and R, demonstrating how to load in the dataset and produce simple plots.

Analysis of human grasping behavior: correlating tasks, objects and grasps

This paper is the second in a two-part series analyzing human grasping behavior during a wide range of unstructured tasks. It investigates the tasks performed during the daily work of two housekeepers and two machinists and correlates grasp type and object properties with the attributes of the tasks being performed. The task or activity is classified according to the force required, the degrees of freedom, and the functional task type. We found that 46 percent of tasks are constrained, where the manipulated object is not allowed to move in a full six degrees of freedom. Analyzing the interrelationships between the grasp, object, and task data show that the best predictors of the grasp type are object size, task constraints, and object mass. Using these attributes, the grasp type can be predicted with 47 percent accuracy. Those parameters likely make useful heuristics for grasp planning systems. The results further suggest the common sub-categorization of grasps into power, intermediate, and precision categories may not be appropriate, indicating that grasps are generally more multi-functional than previously thought. We find large and heavy objects are grasped with a power grasp, but small and lightweight objects are not necessarily grasped with precision grasps-even with grasped object size less than 2 cm and mass less than 20 g, precision grasps are only used 61 percent of the time. These results have important implications for robotic hand design and grasp planners, since it appears while power grasps are frequently used for heavy objects, they can still be quite practical for small, lightweight objects.