Real-time deformable models for surgery simulation: a survey

Atlas-Based Shared-Boundary Deformable Multi-Surface Models through Multi-Material and Two-Manifold Dual Contouring

This paper presents a multi-material dual “contouring” method used to convert a digital 3D voxel-based atlas of basal ganglia to a deformable discrete multi-surface model that supports surgical navigation for an intraoperative MRI-compatible surgical robot, featuring fast intraoperative deformation computation. It is vital that the final surface model maintain shared boundaries where appropriate so that even as the deep-brain model deforms to reflect intraoperative changes encoded in ioMRI, the subthalamic nucleus stays in contact with the substantia nigra, for example, while still providing a significantly sparser representation than the original volumetric atlas consisting of hundreds of millions of voxels. The dual contouring (DC) algorithm is a grid-based process used to generate surface meshes from volumetric data. The DC method enables the insertion of vertices anywhere inside the grid cube, as opposed to the marching cubes (MC) algorithm, which can insert vertices only on the grid edges. This multi-material DC method is then applied to initialize, by duality, a deformable multi-surface simplex model, which can be used for multi-surface atlas-based segmentation. We demonstrate our proposed method on synthetic and deep-brain atlas data, and a comparison of our method’s results with those of traditional DC demonstrates its effectiveness.

Physical modeling and geometric shape simulation for one-dimensional flexible objects with cylindrical surface constraints

This study develops forces equilibrium differential equations for the geometric modeling of 1D flexible objects with surface constraints. These second-order equations are an extension of the Cosserat elastic rod theory and include both bending and torsion. Variables were established for the centerline and attitude in the Cartesian coordinate system of the cross section. This paper specifically investigates the case of a 1D flexible object constrained by a cylindrical surface. To solve this problem, a novel hybrid semi-analytical numerical method is proposed. In this process, a Hamiltonian function and an initial integral operator are introduced in a cylindrical coordinate system. The analytical solution, decoupled in polar coordinates, is then derived. The improved finite difference method was then used to obtain three cylindrical coordinates, which ensured numerical stability and efficiency. The results of a geometric shape simulation with differing boundary conditions demonstrate that this proposed method is capable of real-time modeling. As such, this technique could be a promising new tool for use in graphics simulations of elongated structures, such as DNA molecules, drill pipes, and submarine cables.

Deformable cardiac surface tracking by adaptive estimation algorithms

This study presents a particle filter based framework to track cardiac surface from a time sequence of single magnetic resonance imaging (MRI) slices with the future goal of utilizing the presented framework for interventional cardiovascular magnetic resonance procedures, which rely on the accurate and online tracking of the cardiac surface from MRI data. The framework exploits a low-order parametric deformable model of the cardiac surface. A stochastic dynamic system represents the cardiac surface motion. Deformable models are employed to introduce shape prior to control the degree of the deformations. Adaptive filters are used to model complex cardiac motion in the dynamic model of the system. Particle filters are utilized to recursively estimate the current state of the system over time. The proposed method is applied to recover biventricular deformations and validated with a numerical phantom and multiple real cardiac MRI datasets. The algorithm is evaluated with multiple experiments using fixed and varying image slice planes at each time step. For the real cardiac MRI datasets, the average root-mean-square tracking errors of 2.61 mm and 3.42 mm are reported respectively for the fixed and varying image slice planes. This work serves as a proof-of-concept study for modeling and tracking the cardiac surface deformations via a low-order probabilistic model with the future goal of utilizing this method for the targeted interventional cardiac procedures under MR image guidance. For the real cardiac MRI datasets, the presented method was able to track the points-of-interests located on different sections of the cardiac surface within a precision of 3 pixels. The analyses show that the use of deformable cardiac surface tracking algorithm can pave the way for performing precise targeted intracardiac ablation procedures under MRI guidance. The main contributions of this work are twofold. First, it presents a framework for the tracking of whole cardiac surface from a time sequence of single image slices. Second, it employs adaptive filters to incorporate motion information in the tracking of nonrigid cardiac surface motion for temporal coherence.

A new soft tissue deformation model based on Runge-Kutta: Application in lung

Flexible body deformation model is the most critical research in the field of telemedicine, which decides whether the deformation process of tissues and organs can be simulated in real time and realistically. Basically, the improvement of model accuracy often means the loss of real-time performance. In order to effectively balance between accuracy and real-time performance, this paper proposes a new model, which uses the step-variable fourth-order Runge-Kutta method for the first time to solve the relationship problem between the external force and displacement of the nodes in the finite element deformation of the lung. To improve the real-time performance of the model, a one-stage solution optimization algorithm is proposed to optimize the step size selection problem. Finally, an accelerated two-level node update algorithm is proposed to further improve the real-time performance. A lung surgery simulation platform with 3DMax2020 and OpenGL4.5 is built to verify the accuracy, efficiency, realism and applicability of the model. Experimental results show that the proposed lung model can achieve real-world visual reproduction during remote surgery, and exceeds other 13 reference models in real-time performance, with natural deformation effect, high simulation accuracy, which is suitable for modeling normal lung and four types of lungs suffering from diseases.

A Semi-Automated 3D-Printed Chainmail Design Algorithm with Preprogrammed Directional Functions for Hand Exoskeleton

The problem of computerising the design and development of 3D-printed chainmail with programmed directional functions provides a basis for further research, including the automation of medical devices. The scope of the present research was focused on computational optimisation of the selection of materials and shapes for 3D printing, including the design of medical devices, which constitutes a significant scientific, technical, and clinical problem. The aim of this article was to solve the scientific problem of automated or semi-automated efficient and practical design of 3D-printed chainmail with programmed directional functions (variable stiffness/elasticity depending on the direction). We demonstrate for the first time that 3D-printed particles can be arranged into single-layer chainmail with a tunable one- or two-directional bending modulus for use in a medical hand exoskeleton. In the present work, we accomplished this in two ways: based on traditional programming and based on machine learning. This paper presents the novel results of our research, including 3D printouts, providing routes toward the wider implementation of adaptive chainmails. Our research resulted in an automated or semi-automated efficient and practical 3D printed chainmail design with programmed directional functions for a wrist exoskeleton with variable stiffness/flexibility, depending on the direction. We also compared two methodologies of planning and construction: the use of traditional software and machine-learning-based software, with the latter being more efficient for more complex chainmail designs.

Soft Tissue Hybrid Model for Real-Time Simulations

In this article, a recent formulation for real-time simulation is developed combining the strain energy density of the Spring Mass Model (SMM) with the equivalent representation of the Strain Energy Density Function (SEDF). The resulting Equivalent Energy Spring Model (EESM) is expected to provide information in real-time about the mechanical response of soft tissue when subjected to uniaxial deformations. The proposed model represents a variation of the SMM and can be used to predict the mechanical behavior of biological tissues not only during loading but also during unloading deformation states. To assess the accuracy achieved by the EESM, experimental data was collected from liver porcine samples via uniaxial loading and unloading tensile tests. Validation of the model through numerical predictions achieved a refresh rate of 31 fps (31.49 ms of computation time for each frame), achieving a coefficient of determination R² from 93.23% to 99.94% when compared to experimental data. The proposed hybrid formulation to characterize soft tissue mechanical behavior is fast enough for real-time simulation and captures the soft material nonlinear virgin and stress-softened effects with high accuracy.

Finite-element kalman filter with state constraint for dynamic soft tissue modelling

This research work proposes a novel method for realistic and real-time modelling of deformable biological tissues by the combination of the traditional finite element method (FEM) with constrained Kalman filtering. This methodology transforms the problem of deformation modelling into a problem of constrained filtering to estimate physical tissue deformation online. It discretises the deformation of biological tissues in 3D space according to linear elasticity using FEM. On the basis of this, a constrained Kalman filter is derived to dynamically compute mechanical deformation of biological tissues by minimizing the error between estimated reaction forces and applied mechanical load. The proposed method solves the disadvantage of costly computation in FEM while inheriting the superiority of physical fidelity.

Extended Kalman Filter Nonlinear Finite Element Method for Nonlinear Soft Tissue Deformation

BACKGROUND AND OBJECTIVE

Soft tissue modelling is crucial to surgery simulation. This paper introduces an innovative approach to realistic simulation of nonlinear deformation behaviours of biological soft tissues in real time.

METHODS

This approach combines the traditional nonlinear finite-element method (NFEM) and nonlinear Kalman filtering to address both physical fidelity and real-time performance for soft tissue modelling. It defines tissue mechanical deformation as a nonlinear filtering process for dynamic estimation of nonlinear deformation behaviours of biological tissues. Tissue mechanical deformation is discretized in space using NFEM in accordance with nonlinear elastic theory and in time using the central difference scheme to establish the nonlinear state-space models for dynamic filtering.

RESULTS

An extended Kalman filter is established to dynamically estimate nonlinear mechanical deformation of biological tissues. Interactive deformation of biological soft tissues with haptic feedback is accomplished as well for surgery simulation.

CONCLUSIONS

The proposed approach conquers the NFEM limitation of step computation but without trading off the modelling accuracy. It not only has a similar level of accuracy as NFEM, but also meets the real-time requirement for soft tissue modelling.

Inferring causal networks of dynamical systems through transient dynamics and perturbation

Inferring causal relations from time series measurements is an ill-posed mathematical problem, where typically an infinite number of potential solutions can reproduce the given data. We explore in depth a strategy to disambiguate between possible underlying causal networks by perturbing the network, where the forcings are either targeted or applied at random. The resulting transient dynamics provide the critical information necessary to infer causality. Two methods are shown to provide accurate causal reconstructions: Granger causality (GC) with perturbations, and our proposed perturbation cascade inference (PCI). Perturbed GC is capable of inferring smaller networks under low coupling strength regimes. Our proposed PCI method demonstrated consistently strong performance in inferring causal relations for small (2-5 node) and large (10-20 node) networks, with both linear and nonlinear dynamics. Thus, the ability to apply a large and diverse set of perturbations to the network is critical for successfully and accurately determining causal relations and disambiguating between various viable networks.

Characterizing Limits of Vision-Based Force Feedback in Simulated Surgical Tool-Tissue Interaction

Haptic feedback can render real-time force interactions with computer simulated objects. In several telerobotic applications, it is desired that a haptic simulation reflects a physical task space or interaction accurately. This is particularly true when excessive applied force can result in disastrous consequences, as with the case of robot-assisted minimally invasive surgery (RMIS) and tissue damage. Since force cannot be directly measured in RMIS, non-contact methods are desired. A promising direction of non-contact force estimation involves the primary use of vision sensors to estimate deformation. However, the required fidelity of non-contact force rendering of deformable interaction to maintain surgical operator performance is not well established. This work attempts to empirically evaluate the degree to which haptic feedback may deviate from ground truth yet result in acceptable teleoperated performance in a simulated RMIS-based palpation task. A preliminary user-study is conducted to verify the utility of the simulation platform, and the results of this work have implications in haptic feedback for RMIS and inform guidelines for vision-based tool-tissue force estimation. An adaptive thresholding method is used to collect the minimum and maximum tolerable errors in force orientation and magnitude of presented haptic feedback to maintain sufficient performance.

Survey of Finite Element Method-Based Real-Time Simulations

The finite element method (FEM) has deservedly gained the reputation of the most powerful, highly efficient, and versatile numerical method in the field of structural analysis. Though typical application of FE programs implies the so-called “off-line” computations, the rapid pace of hardware development over the past couple of decades was the major impetus for numerous researchers to consider the possibility of real-time simulation based on FE models. Limitations of available hardware components in various phases of developments demanded remarkable innovativeness in the quest for suitable solutions to the challenge. Different approaches have been proposed depending on the demands of the specific field of application. Though it is still a relatively young field of work in global terms, an immense amount of work has already been done calling for a representative survey. This paper aims to provide such a survey, which of course cannot be exhaustive.

A review on modeling of flexible deformable object for dexterous robotic manipulation

In this article, we present a review on the recent advancement in flexible deformable object modeling for dexterous manipulation in robotic system. Flexible deformable object is one of the most research topics in computer graphic, computer vision, and robotic literature. The deformable models are known as the construction of object with material parameters in virtual environment to describe the deformation behavior. Existing modeling techniques and different types of deformable model are described. Various approaches of deformable object modeling have been used in robotic recognition and manipulation in order to reduce the time and cost to obtain more accurate result. In robotic manipulation, object detection, classification, and recognition of deformable objects are always a challenging problem and required as a first step to imbue the robot to able handle these deformable objects. Furthermore, the dexterity of robot control is also another essential key in handling of deformable object which its manipulation strategies need to plan intelligently for each sequence process. We also discuss some deserving direction for further research based on most current contribution.

A patient-specific mass-spring model for biomechanical simulation of aortic root tissue during transcatheter aortic valve implantation

The success of transcatheter aortic valve implantation (TAVI) is highly dependent on the prediction of the interaction between the prosthesis and the aortic root anatomy. The simulation of the surgical procedure may be useful to guide artificial valve selection and delivery, nevertheless the introduction of simulation models into the clinical workflow is often hindered by model complexity and computational burden. To address this point, we introduced a patient-specific mass-spring model (MSM) with viscous damping, as a good trade-off between simulation accuracy and time-efficiency. The anatomical model consisted of a hexahedral mesh, segmented from pre-procedural patient-specific cardiac computer tomographic (CT) images of the aortic root, including valve leaflets and attached calcifications. Nodal forces were represented by linear-elastic springs acting on edges and angles. A fast integration approach based on the modulation of nodal masses was also tested. The model was validated on seven patients, comparing simulation results with post-procedural CT images with respect to calcification and aortic wall position. The validation showed that the MSM was able to predict calcification displacement with an average accuracy of 1.72 mm and 1.54 mm for the normal and fast integration approaches, respectively. Wall displacement root mean squared error after valve expansion was about 1 mm for both approaches, showing an improved matching with respect to the pre-procedural configuration. In terms of computational burden, the fast integration approach allowed a consistent reduction of the computational times, which decreased from 36 h to 21.8 min per 100 K hexahedra. Our findings suggest that the proposed linear-elastic MSM model may provide good accuracy and reduced computational times for TAVI simulations, fostering its inclusion in clinical routines.

Virtual reality simulation of robotic transsphenoidal brain tumor resection: Evaluating dynamic motion scaling in a master-slave system

Background

Integrating simulators with robotic surgical procedures could assist in designing and testing of novel robotic control algorithms and further enhance patient‐specific pre‐operative planning and training for robotic surgeries.

Methods

A virtual reality simulator, developed to perform the transsphenoidal resection of pituitary gland tumours, tested the usability of robotic interfaces and control algorithms. It used position‐based dynamics to allow soft‐tissue deformation and resection with haptic feedback; dynamic motion scaling control was also incorporated into the simulator.

Results

Neurosurgeons and residents performed the surgery under constant and dynamic motion scaling conditions (CMS vs DMS). DMS increased dexterity and reduced the risk of damage to healthy brain tissue. Post‐experimental questionnaires indicated that the system was well‐evaluated by experts.

Conclusion

The simulator was intuitively and realistically operated. It increased the safety and accuracy of the procedure without affecting intervention time. Future research can investigate incorporating this simulation into a real micro‐surgical robotic system.

Improving Haptic Transparency for Uncertain Virtual Environments Using Adaptive Control and Gain-Scheduled Prediction

The realism or transparency of haptic interfaces is becoming more critical as they are applied to training in fields like minimally invasive surgery (MIS). Surgical training simulators must provide a transparent virtual environment (VE) at a high update rate. Complex, deformable, cuttable tissue models have nonlinear dynamics and are computationally expensive, making it difficult to provide sufficient update rates. The objective of this work is to improve the transparency for this type of VE by formulating the unknown nonlinear dynamics as a quasi-linear parameter varying (LPV) system and designing a predictor to provide an output at a much higher update rate. An adaptive controller based on gain-scheduled prediction is considered for a nonlinear haptic device and a nonlinear, delayed, and sampled VE. The predictor uses feedback from the more accurate but slow-updating VE to update a simplified dynamic model. The predictor is designed based on numerical solutions to a linear matrix inequality derived using Lyapunov-based methods. Experimental results demonstrate the effectiveness of the gain-scheduled predictor approach and compare it to previous work using a constant-gain predictor. The gain-scheduled predictor results in significant performance improvements compared to a haptic system without prediction, but less significant improvement compared to the constant-gain approach.

A review of haptic simulator for oral and maxillofacial surgery based on virtual reality

INTRODUCTION

Traditional medical training in oral and maxillofacial surgery (OMFS) may be limited by its low efficiency and high price due to the shortage of cadaver resources. With the combination of visual rendering and feedback force, surgery simulators become increasingly popular in hospitals and medical schools as an alternative to the traditional training.

AREAS COVERED

The major goal of this review is to provide a comprehensive reference source of current and future developments of haptic OMFS simulators based on virtual reality (VR) for relevant researchers.

EXPERT COMMENTARY

Visual rendering, haptic rendering, tissue deformation, and evaluation are key components of haptic surgery simulator based on VR. Compared with traditional medical training, virtual and tactical fusion of virtual environment in surgery simulator enables considerably vivid sensation, and the operators have more opportunities to practice surgical skills and receive objective evaluation as reference.

Navigation of guidewires and catheters in the body during intervention procedures: a review of computer-based models

Guidewires and catheters are used during minimally invasive interventional procedures to traverse in vascular system and access the desired position. Computer models are increasingly being used to predict the behavior of these instruments. This information can be used to choose the right instrument for each case and increase the success rate of the procedure. Moreover, a designer can test the performance of instruments before the manufacturing phase. A precise model of the instrument is also useful for a training simulator. Therefore, to identify the strengths and weaknesses of different approaches used to model guidewires and catheters, a literature review of the existing techniques has been performed. The literature search was carried out in Google Scholar and Web of Science and limited to English for the period 1960 to 2017. For a computer model to be used in practice, it should be sufficiently realistic and, for some applications, real time. Therefore, we compared different modeling techniques with regard to these requirements, and the purposes of these models are reviewed. Important factors that influence the interaction between the instruments and the vascular wall are discussed. Finally, different ways used to evaluate and validate the models are described. We classified the developed models based on their formulation into finite-element method (FEM), mass-spring model (MSM), and rigid multibody links. Despite its numerical stability, FEM requires a very high computational effort. On the other hand, MSM is faster but there is a risk of numerical instability. The rigid multibody links method has a simple structure and is easy to implement. However, as the length of the instrument is increased, the model becomes slower. For the level of realism of the simulation, friction and collision were incorporated as the most influential forces applied to the instrument during the propagation within a vascular system. To evaluate the accuracy, most of the studies compared the simulation results with the outcome of physical experiments on a variety of phantom models, and only a limited number of studies have done face validity. Although a subset of the validated models is considered to be sufficiently accurate for the specific task for which they were developed and, therefore, are already being used in practice, these models are still under an ongoing development for improvement. Realism and computation time are two important requirements in catheter and guidewire modeling; however, the reviewed studies made a trade-off depending on the purpose of their model. Moreover, due to the complexity of the interaction with the vascular system, some assumptions have been made regarding the properties of both instruments and vascular system. Some validation studies have been reported but without a consistent experimental methodology.

Real-time deformation of human soft tissues: A radial basis meshless 3D model based on Marquardt's algorithm

BACKGROUND

When the meshless method is used to establish the mathematical-mechanical model of human soft tissues, it is necessary to define the space occupied by human tissues as the problem domain and the boundary of the domain as the surface of those tissues. Nodes should be distributed in both the problem domain and on the boundaries. Under external force, the displacement of the node is computed by the meshless method to represent the deformation of biological soft tissues. However, computation by the meshless method consumes too much time, which will affect the simulation of real-time deformation of human tissues in virtual surgery.

METHODS

In this article, the Marquardt's Algorithm is proposed to fit the nodal displacement at the problem domain's boundary and obtain the relationship between surface deformation and force. When different external forces are applied, the deformation of soft tissues can be quickly obtained based on this relationship.

RESULTS AND CONCLUSIONS

The analysis and discussion show that the improved model equations with Marquardt's Algorithm not only can simulate the deformation in real-time but also preserve the authenticity of the deformation model's physical properties.

Cellular neural network modelling of soft tissue dynamics for surgical simulation

BACKGROUND

Currently, the mechanical dynamics of soft tissue deformation is achieved by numerical time integrations such as the explicit or implicit integration; however, the explicit integration is stable only under a small time step, whereas the implicit integration is computationally expensive in spite of the accommodation of a large time step.

OBJECTIVE

This paper presents a cellular neural network method for stable simulation of soft tissue deformation dynamics.

METHOD

The non-rigid motion equation is formulated as a cellular neural network with local connectivity of cells, and thus the dynamics of soft tissue deformation is transformed into the neural dynamics of the cellular neural network.

RESULTS

Results show that the proposed method can achieve good accuracy at a small time step. It still remains stable at a large time step, while maintaining the computational efficiency of the explicit integration.

CONCLUSION

The proposed method can achieve stable soft tissue deformation with efficiency of explicit integration for surgical simulation.

On the Transparency of Client/Server-Based Haptic Interaction with Deformable Objects

This paper studies the transparency of client/server-based haptic interaction with simulated deformable objects. In the considered remote interaction scenario, the server simulates the computationally expensive finite-element-based object deformation at a low temporal update rate and transmits the result to the clients. There, the received deformation data is applied to the polygon mesh, which is used to locally render force feedback with a penalty-based force rendering algorithm at the required high rate. Based on a one-dimensional deformable object example, we analyze the transparency of this multi-rate architecture for a two-user interaction. Communication delay leads to increased force magnitudes and an increased impedance displayed to the clients that actively interact with the object. We propose a method that adjusts the stiffness used in the local force rendering at the clients to compensate for this effect. The conducted objective and subjective evaluations show that the proposed method successfully compensates for the effect of communication delay in the tested delay range of up to 100 ms.

Visualization of vascular injuries in extremity trauma

A tandem of particle-based computational methods is adapted to simulate injury and hemorrhage in the human body. In order to ensure anatomical fidelity, a three-dimensional model of a targeted portion of the human body is reconstructed from a dense sequence of CT scans of an anonymized patient. Skin, bone and muscular tissue are distinguished in the imaging data and assigned with their respective material properties. An injury geometry is then generated by simulating the mechanics of a ballistic projectile passing through the anatomical model with the material point method. From the injured vascular segments identified in the resulting geometry, smoothed particle hydrodynamics (SPH) is employed to simulate bleeding, based on inflow boundary conditions obtained from a network model of the systemic arterial tree. Computational blood particles interact with the stationary particles representing impermeable bone and skin and permeable muscular tissue through the Brinkman equations for porous media. The SPH results are rendered in post-processing for improved visual fidelity. The overall simulation strategy is demonstrated on an injury scenario in the lower leg.

Deformable Model-Based Methods for Shape Control of a Haptic Jamming Surface

Haptic Jamming, the approach of simultaneously controlling mechanical properties and surface deformation of a tactile display via particle jamming and pneumatics, shows promise as a tangible, shape-changing human-computer interface. Previous research introduced device design and described the force-displacement interactions for individual jamming cells. The work in this article analyzes the shape output capabilities of a multi-cell array. A spring-mass deformable body simulation combines models of the three actuation inputs of a Haptic Jamming surface: node pinning, chamber pressurization, and cell jamming. Surface measurements of a 12-cell prototype from a depth camera fit the mass and stiffness parameters to the device during pressurization tests and validate the accuracy of the model for various actuation sequences. The simulator is used to develop an algorithm that generates a sequence of actuation inputs for a Haptic Jamming array of any size in order to match a desired surface output shape. Data extracted from topographical maps and three-dimensional solid object models are used to evaluate the shape-matching algorithm and assess the utility of increasing array size and resolution. Results show that a discrete Laplace operator applied to the input is a suitable predictor of the correlation coefficient between the desired shape and the device output.

NOViSE: a virtual natural orifice transluminal endoscopic surgery simulator

PURPOSE

Natural orifice transluminal endoscopic surgery (NOTES) is a novel technique in minimally invasive surgery whereby a flexible endoscope is inserted via a natural orifice to gain access to the abdominal cavity, leaving no external scars. This innovative use of flexible endoscopy creates many new challenges and is associated with a steep learning curve for clinicians.

METHODS

We developed NOViSE-the first force-feedback-enabled virtual reality simulator for NOTES training supporting a flexible endoscope. The haptic device is custom-built, and the behaviour of the virtual flexible endoscope is based on an established theoretical framework-the Cosserat theory of elastic rods.

RESULTS

We present the application of NOViSE to the simulation of a hybrid trans-gastric cholecystectomy procedure. Preliminary results of face, content and construct validation have previously shown that NOViSE delivers the required level of realism for training of endoscopic manipulation skills specific to NOTES.

CONCLUSIONS

VR simulation of NOTES procedures can contribute to surgical training and improve the educational experience without putting patients at risk, raising ethical issues or requiring expensive animal or cadaver facilities. In the context of an experimental technique, NOViSE could potentially facilitate NOTES development and contribute to its wider use by keeping practitioners up to date with this novel surgical technique. NOViSE is a first prototype, and the initial results indicate that it provides promising foundations for further development.

Parallel deformation of heterogeneous ChainMail models: Application to interactive deformation of large medical volumes

In this work we present a new solution for correctly handling heterogeneous materials in ChainMail models, which are widely used in medical applications. Our core method relies on two main components: (1) a novel timestamp-based propagation scheme that tracks the propagation speed of a deformation through the model and allows to correct ambiguous configurations, and (2) a novel relaxation stage that performs an energy minimization process taking into account the heterogeneity of the model. In addition, our approach extends the SP-ChainMail algorithm by supporting interactive topology changes and handling multiple concurrent deformations, increasing its range of applicability. Finally, we present an improved blocking scheme that efficiently handles the sparse computation, greatly increasing the performance of our algorithm. Our proposed solution has been applied to interactive deformation of large medical datasets. The simulation model is directly generated from the input dataset and a user defined material transfer function, while the visualization of the deformations is performed by rendering the resampled deformed model using direct volume rendering techniques. In our results, we show that our parallel pipeline is capable of interactively deforming models with several million elements. A comparison is finally discussed, analyzing the properties of our approach with respect to previous work. The results show that our algorithm correctly handles very large heterogeneous ChainMail models in an interactive manner, increasing the applicability of the ChainMail approach for more demanding scenarios both in response time and material modeling.

Implementation and clinical application of a deformation method for fast simulation of biological tissue formed by fibers and fluid

Background

The aim of this paper is to provide a general discussion, algorithm, and actual working programs of the deformation method for fast simulation of biological tissue formed by fibers and fluid. In order to demonstrate the benefit of the clinical applications software, we successfully used our computational program to deform a 3D breast image acquired from patients, using a 3D scanner, in a real hospital environment.

Results

The method implements a quasi-static solution for elastic global deformations of objects. Each pair of vertices of the surface is connected and defines an elastic fiber. The set of all the elastic fibers defines a mesh of smaller size than the volumetric meshes, allowing for simulation of complex objects with less computational effort. The behavior similar to the stress tensor is obtained by the volume conservation equation that mixes the 3D coordinates. Step by step, we show the computational implementation of this approach.

Conclusions

As an example, a 2D rectangle formed by only 4 vertices is solved and, for this simple geometry, all intermediate results are shown. On the other hand, actual implementations of these ideas in the form of working computer routines are provided for general 3D objects, including a clinical application.

Quasi-non-linear deformation modeling of a human liver based on artificial and experimental data

BACKGROUND

Researchers working on error-prevention theories have shown that the use of replica models within simulation systems has improved operating skills, resulting in better patient outcomes.

METHODS

This study aims to provide material test data specifically for a human liver to validate the accuracy of viscoelastic soft tissue models. This allows the validation of virtual surgery simulators by comparison with physical test data obtained from material tests on a viscoelastic silicone gel pad.

RESULTS

The results proved that stress behavior and relaxation curves of Aquaflex® experiment and FEM simulation are close if average liver response and respective material parameters and model are used.

CONCLUSIONS

The precise representation of manipulated tissues used in virtual surgery trainers involves the accurate characterization of mechanical properties of the tissue. Consequently, successful implementations of these mechanical properties in a mathematical model of the deforming organ are of major importance. Copyright © 2015 John Wiley & Sons, Ltd.

Comparison and calibration of a real-time virtual stenting algorithm using Finite Element Analysis and Genetic Algorithms

In this paper, we perform a comparative analysis between two computational methods for virtual stent deployment: a novel fast virtual stenting method, which is based on a spring-mass model, is compared with detailed finite element analysis in a sequence of in silico experiments. Given the results of the initial comparison, we present a way to optimise the fast method by calibrating a set of parameters with the help of a genetic algorithm, which utilises the outcomes of the finite element analysis as a learning reference. As a result of the calibration phase, we were able to substantially reduce the force measure discrepancy between the two methods and validate the fast stenting method by assessing the differences in the final device configurations.

NiftySim: A GPU-based nonlinear finite element package for simulation of soft tissue biomechanics

PURPOSE

NiftySim, an open-source finite element toolkit, has been designed to allow incorporation of high-performance soft tissue simulation capabilities into biomedical applications. The toolkit provides the option of execution on fast graphics processing unit (GPU) hardware, numerous constitutive models and solid-element options, membrane and shell elements, and contact modelling facilities, in a simple to use library.

METHODS

The toolkit is founded on the total Lagrangian explicit dynamics (TLEDs) algorithm, which has been shown to be efficient and accurate for simulation of soft tissues. The base code is written in C[Formula: see text], and GPU execution is achieved using the nVidia CUDA framework. In most cases, interaction with the underlying solvers can be achieved through a single Simulator class, which may be embedded directly in third-party applications such as, surgical guidance systems. Advanced capabilities such as contact modelling and nonlinear constitutive models are also provided, as are more experimental technologies like reduced order modelling. A consistent description of the underlying solution algorithm, its implementation with a focus on GPU execution, and examples of the toolkit's usage in biomedical applications are provided.

RESULTS

Efficient mapping of the TLED algorithm to parallel hardware results in very high computational performance, far exceeding that available in commercial packages.

CONCLUSION

The NiftySim toolkit provides high-performance soft tissue simulation capabilities using GPU technology for biomechanical simulation research applications in medical image computing, surgical simulation, and surgical guidance applications.

Coupling strategies for multi-resolution deformable meshes: expanding the pyramid approach beyond its one-way nature

PURPOSE

With higher resolutions, medical image processing operations like segmentation take more time to calculate per step. The pyramid technique is a common approach to solving this problem. Starting with a low resolution, a stepwise refinement is applied until the original resolution is reached.

METHODS

Our work proposes a method for deformable model segmentation that generally utilizes the common pyramid technique with our improvement, to calculate and keep synchronized all mesh resolution levels in parallel. The models are coupled to propagate their changes. It presents coupling techniques and shows approaches for synchronization. The interaction with the models is realized using springs and volcanoes, and it is evaluated for the semantics of the operation to share them across the different levels.

RESULTS

The locking overhead has been evaluated for different synchronization techniques with meshes of individual resolutions. The partial update strategy has been found to have the least locking overhead.

CONCLUSION

Running multiple models with individual resolutions in parallel is feasible. The synchronization approach has to be chosen carefully, so that an interactive modification of the segmentation remains possible. The proposed technique is aimed at making medical image segmentation more usable while delivering high performance.

Volume Preserved Mass-Spring Model with Novel Constraints for Soft Tissue Deformation

An interactive surgical simulation system needs to meet three main requirements, speed, accuracy, and stability. In this paper, we present a stable and accurate method for animating mass-spring systems in real time. An integration scheme derived from explicit integration is used to obtain interactive realistic animation for a multiobject environment. We explore a predictor-corrector approach by correcting the estimation of the explicit integration in a poststep process. We introduce novel constraints on positions into the mass-spring model (MSM) to model the nonlinearity and preserve volume for the realistic simulation of the incompressibility. We verify the proposed MSM by comparing its deformations with the reference deformations of the nonlinear finite-element method. Moreover, experiments on porcine organs are designed for the evaluation of the multiobject deformation. Using a pair of freshly harvested porcine liver and gallbladder, the real organ deformations are acquired by computed tomography and used as the reference ground truth. Compared to the porcine model, our model achieves a 1.502 mm mean absolute error measured at landmark locations for cases with small deformation (the largest deformation is 49.109 mm) and a 3.639 mm mean absolute error for cases with large deformation (the largest deformation is 83.137 mm). The changes of volume for the two deformations are limited to 0.030% and 0.057%, respectively. Finally, an implementation in a virtual reality environment for laparoscopic cholecystectomy demonstrates that our model is capable to simulate large deformation and preserve volume in real-time calculations.

Haptic simulation of organ deformation and hybrid contacts in dental operations

There are two main challenges in simulating bi-manual dental operations with six-degrees-of-freedom (6-DoF) haptic rendering. One is to simulate large deformation and force response of a tongue under multi-region contacts with a dental mirror, and the other is to simulate the force response when a dental probe inserts into a narrow periodontal pocket, which leads to simultaneous contacts of different types between the probe and both rigid and deformable objects (i.e., a rigid tooth and its surrounding deformable gingiva), which we call hybrid contacts, as well as frequent contact switches. In this paper, we address both challenges. We first introduce a novel method for modeling deformation based on a sphere-tree representation of deformable objects. A configuration-based constrained optimization method is utilized for determining the six-dimensional configuration of the graphic tool and the contact force/torque. This approach conducts collision detection, deformation computation, and tool configuration optimization very efficiently, avoids inter-penetration, and maintains stability of haptic display without using virtual coupling. To simulate the force response due to fine manipulation of the probe inside a narrow periodontal pocket, we propose an efficient method to simulate the local deformation of the gingiva and stable haptic feedback under frequent contact switches. Simulations on typical dental operations were carried out to validate the efficiency and stability of our approach.

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70-0.95 (P<0.05). A t-test was used for the statistical analysis.

Direct volume manipulation for visualizing intraoperative liver resection process

This paper introduces a new design and application for direct volume manipulation for visualizing the intraoperative liver resection process. So far, interactive volume deformation and resection have been independently handled due to the difficulty of representing elastic behavior of volumetric objects. Our framework models global shape editing and discontinuous local deformation by merging proxy geometry encoding and displacement mapping. A local-frame-based elastic model is presented to allow stable editing of the liver shape including bending and twisting while preserving the volume. Several tests using clinical CT data have confirmed the developed software and interface can represent the intraoperative state of liver and produce local views of reference vascular structures, which provides a "road map of vessels" that are key features when approaching occluded tumors during surgery.

Haptic feedback and control of a flexible surgical endoscopic robot

A flexible endoscope could reach the potential surgical site via a single small incision on the patient or even through natural orifices, making it a very promising platform for surgical procedures. However, endoscopic surgery has strict spatial constraints on both tool-channel size and surgical site volume. It is therefore very challenging to deploy and control dexterous robotic instruments to conduct surgical procedures endoscopically. Pioneering endoscopic surgical robots have already been introduced, but the performance is limited by the flexible neck of the robot that passes through the endoscope tool channel. In this article we present a series of new developments to improve the performance of the robot: a force transmission model to address flexibility, elongation study for precise position control, and tissue property modeling for haptic feedback. Validation experiment results are presented for each sector. An integrated control architecture of the robot system is given in the end.

Synchronous simulation for deformation of liver and gallbladder with stretch and compression compensation

One challenge in surgical simulation is to design stable deformable models to simulate the dynamics of organs synchronously. In this paper, we develop a novel mass-spring model on the tetrahedral meshes for soft organs such as the liver and gallbladder, which can stably deform with large time steps. We model the contact forces between the organs as a kind of forces generated by the tensions of repulsive springs connecting in between the organs. The simulation system couples a pair of constraints on the length of springs with an implicit integration method. Based on the novel constraints, our simulator can efficiently preserve the volumes and geometric properties of the liver and gallbladder during the simulation. The numerical examples demonstrate that the proposed simulation system can provide realistic and stable deformable results.

Interventional radiology virtual simulator for liver biopsy

PURPOSE

Training in Interventional Radiology currently uses the apprenticeship model, where clinical and technical skills of invasive procedures are learnt during practice in patients. This apprenticeship training method is increasingly limited by regulatory restrictions on working hours, concerns over patient risk through trainees' inexperience and the variable exposure to case mix and emergencies during training. To address this, we have developed a computer-based simulation of visceral needle puncture procedures.

METHODS

A real-time framework has been built that includes: segmentation, physically based modelling, haptics rendering, pseudo-ultrasound generation and the concept of a physical mannequin. It is the result of a close collaboration between different universities, involving computer scientists, clinicians, clinical engineers and occupational psychologists.

RESULTS

The technical implementation of the framework is a robust and real-time simulation environment combining a physical platform and an immersive computerized virtual environment. The face, content and construct validation have been previously assessed, showing the reliability and effectiveness of this framework, as well as its potential for teaching visceral needle puncture.

CONCLUSION

A simulator for ultrasound-guided liver biopsy has been developed. It includes functionalities and metrics extracted from cognitive task analysis. This framework can be useful during training, particularly given the known difficulties in gaining significant practice of core skills in patients.

Navigation system with real-time finite element analysis for minimally invasive surgery

This paper presents a navigation system for minimally invasive surgery, especially laparoscopic surgery in which operates in abdomen. Conventional navigation systems show virtual images by superimposing models of target tissues on real endoscopic images. Since soft tissues within the abdomen are deformed during the surgery, the navigation system needs to provide surgeons reliable information by deforming the models according to their biomechanical behavior. However, conventional navigation systems don't consider the tissue deformation during the surgery. We have been developing a new real-time FEM-based simulation for deforming a soft tissue model by using neural network[1]. The network is called the neuroFEM. The incorporation of the neuroFEM into the navigation leads to improve the accuracy of the navigation system. In this paper, we propose a new navigation system with a framework of the neuroFEM.

NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training

BACKGROUND

A virtual reality neurosurgery simulator with haptic feedback may help in the training and assessment of technical skills requiring the use of tactile and visual cues.

OBJECTIVE

To develop a simulator for craniotomy-based procedures with haptic and graphics feedback for implementation by universities and hospitals in the neurosurgery training curriculum.

METHODS

NeuroTouch was developed by a team of more than 50 experts from the National Research Council Canada in collaboration with surgeons from more than 20 teaching hospitals across Canada. Its main components are a stereovision system, bimanual haptic tool manipulators, and a high-end computer. The simulation software engine runs 3 processes for computing graphics, haptics, and mechanics. Training tasks were built from magnetic resonance imaging scans of patients with brain tumors.

RESULTS

Two training tasks were implemented for practicing skills with 3 different surgical tools. In the tumor-debulking task, the objective is complete tumor removal without removing normal tissue, using the regular surgical aspirator (suction) and the ultrasonic aspirator. The objective of the tumor cauterization task is to remove a vascularized tumor with an aspirator while controlling blood loss using bipolar electrocautery.

CONCLUSION

NeuroTouch prototypes have been set up in 7 teaching hospitals across Canada, to be used for beta testing and validation and evaluated for integration in a neurosurgery training curriculum.

Tuning of patient-specific deformable models using an adaptive evolutionary optimization strategy

We present and analyze the behavior of an evolutionary algorithm designed to estimate the parameters of a complex organ behavior model. The model is adaptable to account for patient's specificities. The aim is to finely tune the model to be accurately adapted to various real patient datasets. It can then be embedded, for example, in high fidelity simulations of the human physiology. We present here an application focused on respiration modeling. The algorithm is automatic and adaptive. A compound fitness function has been designed to take into account for various quantities that have to be minimized. The algorithm efficiency is experimentally analyzed on several real test cases: 1) three patient datasets have been acquired with the "breath hold" protocol, and 2) two datasets corresponds to 4-D CT scans. Its performance is compared with two traditional methods (downhill simplex and conjugate gradient descent): a random search and a basic real-valued genetic algorithm. The results show that our evolutionary scheme provides more significantly stable and accurate results.