Estimation of Soft Tissue Mechanical Parameters from Robotic Manipulation Data

Visual cues of soft-tissue behaviour in minimal-invasive and robotic surgery

Minimal-invasive surgery (MIS) and robotic surgery (RS) offer multiple advantages over open surgery (Vajsbaher et al. in Cogn Syst Res 64:08, 2020). However, the lack of haptic feedback is still a limitation. Surgeons learn to adapt to this lack of haptic feedback using visual cues to make judgements about tissue deformation. Experienced robotic surgeons use the visual interpretation of tissue as a surrogate for tactile feedback. The aim of this review is to identify the visual cues that are consciously or unconsciously used by expert surgeons to manipulate soft tissue safely during Minimally Invasive Surgery (MIS) and Robotic Surgery (RS). We have conducted a comprehensive literature review with papers on visual cue identification and their application in education, as well as skill assessment and surgeon performance measurement with respect to visual feedback. To visualise our results, we provide an overview of the state-of-the-art in the form of a matrix across identified research features, where papers are clustered and grouped in a comparative way. The clustering of the papers showed explicitly that state-of-the-art research does not in particular study the direct effects of visual cues in relation to the manipulation of the tissue and training for that purpose, but is more concentrated on tissue identification. We identified a gap in the literature about the use of visual cues for educational design solutions, that aid the training of soft-tissue manipulation in MIS and in RS. There appears to be a need RS education to make visual cue identification more accessible and set it in the context of manipulation tasks.

Deformation and mechanics of a pulvinus-inspired material

Mimosa pudicarapidly folds leaves when touched. Motion is created by pulvini, 'the plant muscles' that allow plants to produce various complex motions. Plants rely on local control of the turgor pressure to create on-demand motion. In this paper, the mechanics of a cellular material inspired from pulvinus ofM. pudicais studied. First, the manufacturing process of a cell-controllable material is described. Its deformation behaviour when pressured is tested, focusing on three pressure patterns of reference. The deformations are modelled based on the minimisation of elastic energy framework. Depending on pressurisation pattern and magnitude, reversible buckling-induced motion may occur.

Real-Time Numerical Simulation for Accurate Soft Tissues Modeling during Haptic Interaction

The simulation of fabrics physics and its interaction with the human body has been largely studied in recent years to provide realistic-looking garments and wears specifically in the entertainment business. When the purpose of the simulation is to obtain scientific measures and detailed mechanical properties of the interaction, the underlying physical models should be enhanced to obtain better simulation accuracy increasing the modeling complexity and relaxing the simulation timing constraints to properly solve the set of equations under analysis. However, in the specific field of haptic interaction, the desiderata are to have both physical consistency and high frame rate to display stable and coherent stimuli as feedback to the user requiring a tradeoff between accuracy and real-time interaction. This work introduces a haptic system for the evaluation of the fabric hand of specific garments either existing or yet to be produced in a virtual reality simulation. The modeling is based on the co-rotational Finite Element approach that allows for large displacements but the small deformation of the elements. The proposed system can be beneficial for the fabrics industry both in the design phase or in the presentation phase, where a virtual fabric portfolio can be shown to customers around the world. Results exhibit the feasibility of high-frequency real-time simulation for haptic interaction with virtual garments employing realistic mechanical properties of the fabric materials.

Modeling, learning, perception, and control methods for deformable object manipulation

Perceiving and handling deformable objects is an integral part of everyday life for humans. Automating tasks such as food handling, garment sorting, or assistive dressing requires open problems of modeling, perceiving, planning, and control to be solved. Recent advances in data-driven approaches, together with classical control and planning, can provide viable solutions to these open challenges. In addition, with the development of better simulation environments, we can generate and study scenarios that allow for benchmarking of various approaches and gain better understanding of what theoretical developments need to be made and how practical systems can be implemented and evaluated to provide flexible, scalable, and robust solutions. To this end, we survey more than 100 relevant studies in this area and use it as the basis to discuss open problems. We adopt a learning perspective to unify the discussion over analytical and data-driven approaches, addressing how to use and integrate model priors and task data in perceiving and manipulating a variety of deformable objects.

Twining plant inspired pneumatic soft robotic spiral gripper with a fiber optic twisting sensor

The field of soft robotics has been significantly advanced with the recent developments of pneumatic techniques, soft materials, and high-precision motion control. While comprehensive motions can be achieved by sophisticated soft robots, multiple coordinated pneumatic controls are usually required to achieve even the simplest motions. Furthermore, most soft robotics are lacking the ability to sense the environment and provide feedback to the pneumatic control system. In this work, we design a twining plant inspired soft-robotic spiral gripper that requires only one single pneumatic control to perform the twining motion and to firmly hold onto a target object. The soft-robotic spiral gripper has an embedded high-birefringence fiber optic twisting sensor to provide critical information, including twining angle, presence of external perturbation, and physical parameter of the target object. Furthermore, finite element analyses (FEA) in parametric studies of the spiral gripper are performed for module design optimization. The unique single pneumatic channel design enables easy manipulation of the soft spiral gripper with a maximum of 540° twining angle and allows a firm grip of a target object as small as 1-mm in diameter. The embedded fiber optic sensor provides useful information of the target object as well as the twining angle of the soft robotic spiral gripper with high twining angle sensitivity of 0.03nm. The unique fiber-optic sensor embedded single-channel pneumatic spiral gripper that is made from non-toxic silicone rubber allows parallel and soft gripping of elongated objects located in a confined area, which is an essential building block for twining and twisting motions in soft robot.

All-in-One, Wireless, Stretchable Hybrid Electronics for Smart, Connected, and Ambulatory Physiological Monitoring

Commercially available health monitors rely on rigid electronic housing coupled with aggressive adhesives and conductive gels, causing discomfort and inducing skin damage. Also, research-level skin-wearable devices, while excelling in some aspects, fall short as concept-only presentations due to the fundamental challenges of active wireless communication and integration as a single device platform. Here, an all-in-one, wireless, stretchable hybrid electronics with key capabilities for real-time physiological monitoring, automatic detection of signal abnormality via deep-learning, and a long-range wireless connectivity (up to 15 m) is introduced. The strategic integration of thin-film electronic layers with hyperelastic elastomers allows the overall device to adhere and deform naturally with the human body while maintaining the functionalities of the on-board electronics. The stretchable electrodes with optimized structures for intimate skin contact are capable of generating clinical-grade electrocardiograms and accurate analysis of heart and respiratory rates while the motion sensor assesses physical activities. Implementation of convolutional neural networks for real-time physiological classifications demonstrates the feasibility of multifaceted analysis with a high clinical relevance. Finally, in vivo demonstrations with animals and human subjects in various scenarios reveal the versatility of the device as both a health monitor and a viable research tool.

An Iterative Method for Estimating Nonlinear Elastic Constants of Tumor in Soft Tissue from Approximate Displacement Measurements

Objectives

Various elastography techniques have been proffered based on linear or nonlinear constitutive models with the aim of detecting and classifying pathologies in soft tissues accurately and noninvasively. Biological soft tissues demonstrate behaviors which conform to nonlinear constitutive models, in particular the hyperelastic ones. In this paper, we represent the results of our steps towards implementing ultrasound elastography to extract hyperelastic constants of a tumor inside soft tissue.

Methods

Hyperelastic parameters of the unknown tissue have been estimated by applying the iterative method founded on the relation between stress, strain, and the parameters of a hyperelastic model after (a) simulating the medium's response to a sinusoidal load and extracting the tissue displacement fields in some instants and (b) estimating the tissue displacement fields from the recorded/simulated ultrasound radio frequency signals and images using the cross correlation-based technique.

Results

Our results indicate that hyperelastic parameters of an unidentified tissue could be precisely estimated even in the conditions where there is no prior knowledge of the tissue, or the displacement fields have been approximately calculated using the data recorded by a clinical ultrasound system.

Conclusions

The accurate estimation of nonlinear elastic constants yields to the correct cognizance of pathologies in soft tissues.

Efficient Sensitivity Based Reconstruction Technique to Accomplish Breast Hyperelastic Elastography

Hyperelastic models have been acknowledged as constitutive equations which reliably model the nonlinear behaviors observed from soft tissues under various loading conditions. Among them, the Mooney-Rivlin, Yeoh, and polynomial models have been proved capable of accurately modeling responses of breast tissues to applied compressions. Hyperelastic elastography technique takes advantage of the disparities between hyperelastic parameters of varied tissues and the change in hyperelastic parameters in pathological processes. The precise reconstruction of hyperelastic parameters of a completely unknown pathology in the breast in a noninvasive and nondestructive way using the ultrasound elastography has been scrutinized in this paper. In the ultrasound elastography, tissue displacement field is extracted from radio frequency signals or images recorded using the ultrasound medical imaging system; hence the exact displacement field might not be obtained. Our results indicate that the parameters estimated by manipulating the iterative sensitivity-matrix based method converge to tissue's real hyperelastic parameters providing appropriate parameters are assigned to the hypothetical hyperelastic and regularization parameters. Iterative methods have therefore been proposed to compute proper hypothetical hyperelastic and regularization parameters. Accurate estimates of hyperelastic parameters of obscure breast pathology have been achieved even from imprecise measurements of displacements induced in the tissue by the ramp excitation.

Recursive Least Squares Filtering Algorithms for On-Line Viscoelastic Characterization of Biosamples

The mechanical characterization of biological samples is a fundamental issue in biology and related fields, such as tissue and cell mechanics, regenerative medicine and diagnosis of diseases. In this paper, a novel approach for the identification of the stiffness and damping coefficients of biosamples is introduced. According to the proposed method, a MEMS-based microgripper in operational condition is used as a measurement tool. The mechanical model describing the dynamics of the gripper-sample system considers the pseudo-rigid body model for the microgripper, and the Kelvin–Voigt constitutive law of viscoelasticity for the sample. Then, two algorithms based on recursive least square (RLS) methods are implemented for the estimation of the mechanical coefficients, that are the forgetting factor based RLS and the normalised gradient based RLS algorithms. Numerical simulations are performed to verify the effectiveness of the proposed approach. Results confirm the feasibility of the method that enables the ability to perform simultaneously two tasks: sample manipulation and parameters identification.

Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey

We present a survey of recent work on robot manipulation and sensing of deformable objects, a field with relevant applications in diverse industries such as medicine (e.g. surgical assistance), food handling, manufacturing, and domestic chores (e.g. folding clothes). We classify the reviewed approaches into four categories based on the type of object they manipulate. Furthermore, within this object classification, we divide the approaches based on the particular task they perform on the deformable object. Finally, we conclude this survey with a discussion of the current state-of-the-art approaches and propose future directions within the proposed classification.

New MEMS Tweezers for the Viscoelastic Characterization of Soft Materials at the Microscale

As many studies show, there is a relation between the tissue's mechanical characteristics and some specific diseases. Knowing this relationship would help early diagnosis or microsurgery. In this paper, a new method for measuring the viscoelastic properties of soft materials at the microscale is proposed. This approach is based on the adoption of a microsystem whose mechanical structure can be reduced to a compliant four bar linkage where the connecting rod is substituted by the tissue sample. A procedure to identify both stiffness and damping coefficients of the tissue is then applied to the developed hardware. Particularly, stiffness is calculated solving the static equations of the mechanism in a desired configuration, while the damping coefficient is inferred from the dynamic equations, which are written under the hypothesis that the sample tissue is excited by a variable compression force characterized by a suitable wave form. The whole procedure is implemented by making use of a control system.

A NEW PARAMETER ESTIMATION METHOD FOR ONLINE SOFT TISSUE CHARACTERIZATION

Dynamic soft tissue characterization is of importance to robotic-assisted minimally invasive surgery. The traditional linear regression method is unsuited to handle the non-linear Hunt–Crossley (HC) model and its linearization process involves a linearization error. This paper presents a new non-linear estimation method for dynamic characterization of mechanical properties of soft tissues. In order to deal with non-linear and dynamic conditions involved in soft tissue characterization, this method improves the non-linearity and dynamics of the HC model by treating parameter [Formula: see text] as independent variable. Based on this, an unscented Kalman filter is developed for online estimation of soft tissue parameters. Simulations and comparison analysis demonstrate that the proposed method is able to estimate mechanical parameters for both homogeneous tissues and heterogeneous and multi-layer tissues, and the achieved performance is much better than that of the linear regression method.

Encapsulated, High-Performance, Stretchable Array of Stacked Planar Micro-Supercapacitors as Waterproof Wearable Energy Storage Devices

We report the fabrication of an encapsulated, high-performance, stretchable array of stacked planar micro-supercapacitors (MSCs) as a wearable energy storage device for waterproof applications. A pair of planar all-solid-state MSCs with spray-coated multiwalled carbon nanotube electrodes and a drop-cast UV-patternable ion-gel electrolyte was fabricated on a polyethylene terephthalate film using serial connection to increase the operation voltage of the MSC. Additionally, multiple MSCs could be vertically stacked with parallel connections to increase both the total capacitance and the areal capacitance owing to the use of a solid-state patterned electrolyte. The overall device of five parallel-connected stacked MSCs, a microlight-emitting diode (μ-LED), and a switch was encapsulated in thin Ecoflex film so that the capacitance remained at 82% of its initial value even after 4 d in water; the μ-LED was lit without noticeable decrease in brightness under deformation including bending and stretching. Furthermore, an Ecoflex encapsulated oximeter wound around a finger was operated using the stored energy of the MSC array attached to the hand (even in water) to give information on arterial pulse rate and oxygen saturation in the blood. This study suggests potential applications of our encapsulated MSC array in wearable energy storage devices especially in water.

Needle Grasp and Entry Port Selection for Automatic Execution of Suturing Tasks in Robotic Minimally Invasive Surgery

This paper presents algorithms for selection of needle grasp and for selection of entry ports of robotic instruments, for autonomous robotic execution of the minimally invasive surgical suturing task. A critical issue for automatic execution of surgical tasks, such as suturing, is the choice of needle grasp for the robotic system. Inappropriate needle grasp increases operating time requiring multiple regrasps to complete the desired task. In robotic minimally invasive surgery, the entry port that the surgical robot goes through into the patient's body has a significant role on the performance of the robot. Improper entry port affects the robot's dexterity, manipulability and reachability. The proposed methods use manipulability, dexterity and torque metrics for needle grasp selection, and employ needle grasp robustness and target location robustness metrics for port selection. The results of a case study simulation in thoracoscopic surgery is also presented to demonstrate the proposed methods. Note to Practitioners-This paper is motivated by the problem of automating low-level surgical tasks in robotic surgery, such as, suturing, retraction, dissection, and providing exposure. Specifically, this paper focuses on needle grasp and entry port selection for automating robotic surgical suturing. Selection of an appropriate way of grasping a needle is critical for successfully and robustly completing autonomous suturing. To the best authors' knowledge, there are no earlier studies in the literature which focus on the needle grasp selection problem. The proposed approach determines how to grasp the needle by optimizing the surgical system's manipulation performance. The existing approaches in the literature for selecting entry ports for the robotic surgical tools only consider the teleoperated robotic minimally invasive surgery, in which the surgeons directly control the robotic instruments. However, automated performance of suturing introduces additional challenges due to uncertainties in needle localization and grasping. This paper proposes two new performance metrics on selecting port locations from the perspective of autonomously performing surgical suturing, without direct involvement of the human user. The paper also presents preliminary experiments which demonstrate the effectiveness of the proposed methods.

Optimal Needle Grasp Selection for Automatic Execution of Suturing Tasks in Robotic Minimally Invasive Surgery

This paper presents algorithms for optimal selection of needle grasp, for autonomous robotic execution of the minimally invasive surgical suturing task. In order to minimize the tissue trauma during the suturing motion, the best practices of needle path planning that are used by surgeons are applied for autonomous robotic surgical suturing tasks. Once an optimal needle trajectory in a well-defined suturing scenario is chosen, another critical issue for suturing is the choice of needle grasp for the robotic system. Inappropriate needle grasp increases operating time requiring multiple re-grasps to complete the desired task. The proposed methods use manipulability, dexterity and torque metrics for needle grasp selection. A simulation demonstrates the proposed methods and recommends a variety of grasps. Then a realistic demonstration compares the performances of the manipulator using different grasps.

Wearable soft sensing suit for human gait measurement

Wearable robots based on soft materials will augment mobility and performance of the host without restricting natural kinematics. Such wearable robots will need soft sensors to monitor the movement of the wearer and robot outside the lab. Until now wearable soft sensors have not demonstrated significant mechanical robustness nor been systematically characterized for human motion studies of walking and running. Here, we present the design and systematic characterization of a soft sensing suit for monitoring hip, knee, and ankle sagittal plane joint angles. We used hyper-elastic strain sensors based on microchannels of liquid metal embedded within elastomer, but refined their design with the use of discretized stiffness gradients to improve mechanical durability. We found that these robust sensors could stretch up to 396% of their original lengths, would restrict the wearer by less than 0.17% of any given joint’s torque, had gauge factor sensitivities of greater than 2.2, and exhibited less than 2% change in electromechanical specifications through 1500 cycles of loading–unloading. We also evaluated the accuracy and variability of the soft sensing suit by comparing it with joint angle data obtained through optical motion capture. The sensing suit had root mean square (RMS) errors of less than 5° for a walking speed of 0.89 m/s and reached a maximum RMS error of 15° for a running speed of 2.7 m/s. Despite the deviation of absolute measure, the relative repeatability of the sensing suit’s joint angle measurements were statistically equivalent to that of optical motion capture at all speeds. We anticipate that wearable soft sensing will also have applications beyond wearable robotics, such as in medical diagnostics and in human–computer interaction.

Identification and Active Exploration of Deformable Object Boundary Constraints through Robotic Manipulation

Robotic motion planning algorithms for manipulation of deformable objects, such as in medical robotics applications, rely on accurate estimations of object deformations that occur during manipulation. An estimation of the tissue response (for off-line planning or real-time on-line re-planning), in turn, requires knowledge of both object constitutive parameters and boundary constraints. In this paper, a novel algorithm for estimating boundary constraints of deformable objects from robotic manipulation data is presented. The proposed algorithm uses tissue deformation data collected with a vision system, and employs a multi-stage hill climbing procedure to estimate the boundary constraints of the object. An active exploration technique, which uses an information maximization approach, is also proposed to extend the identification algorithm. The effects of uncertainties on the proposed methods are analyzed in simulation. The results of experimental evaluation of the methods are also presented.