Real-Time Three-Dimensional Ultrasound for Guiding Surgical Tasks

Photoacoustic-based visual servoing of a needle tip

In intraoperative settings, the presence of acoustic clutter and reflection artifacts from metallic surgical tools often reduces the effectiveness of ultrasound imaging and complicates the localization of surgical tool tips. We propose an alternative approach for tool tracking and navigation in these challenging acoustic environments by augmenting ultrasound systems with a light source (to perform photoacoustic imaging) and a robot (to autonomously and robustly follow a surgical tool regardless of the tissue medium). The robotically controlled ultrasound probe continuously visualizes the location of the tool tip by segmenting and tracking photoacoustic signals generated from an optical fiber inside the tool. System validation in the presence of fat, muscle, brain, skull, and liver tissue with and without the presence of an additional clutter layer resulted in mean signal tracking errors <2 mm, mean probe centering errors <1 mm, and successful recovery from ultrasound perturbations, representing either patient motion or switching from photoacoustic images to ultrasound images to search for a target of interest. A detailed analysis of channel SNR in controlled experiments with and without significant acoustic clutter revealed that the detection of a needle tip is possible with photoacoustic imaging, particularly in cases where ultrasound imaging traditionally fails. Results show promise for guiding surgeries and procedures in acoustically challenging environments with this novel robotic and photoacoustic system combination.

Advanced ultrasound for prenatal interventions

Ultrasound is an integral part of prenatal interventions. Doppler studies and 3-dimensional ultrasound (3DUS) are frequently used to determine whether fetal surgery is required. The operator's experience remains crucial for reducing procedure-related morbidity. Real-time 3DUS (or 4DUS) can simultaneously display the needle tip in three orthogonal planes, providing reassurance that no fetal parts are in the path. In experienced hands, 4DUS guidance may not be more effective than B-mode, but its value for less-experienced operators remains to be determined. Recent developments in needle, shunt, and video endoscopic technologies may compliment the use of image-guided in utero procedures. Future developments of higher-dimensional transducers and image software may improve the utility of ultrasound for invasive obstetric interventions.

Medical Instrument Detection in 3-Dimensional Ultrasound Data Volumes

Ultrasound-guided medical interventions are broadly applied in diagnostics and therapy, e.g., regional anesthesia or ablation. A guided intervention using 2-D ultrasound is challenging due to the poor instrument visibility, limited field of view, and the multi-fold coordination of the medical instrument and ultrasound plane. Recent 3-D ultrasound transducers can improve the quality of the image-guided intervention if an automated detection of the needle is used. In this paper, we present a novel method for detecting medical instruments in 3-D ultrasound data that is solely based on image processing techniques and validated on various ex vivo and in vivo data sets. In the proposed procedure, the physician is placing the 3-D transducer at the desired position, and the image processing will automatically detect the best instrument view, so that the physician can entirely focus on the intervention. Our method is based on the classification of instrument voxels using volumetric structure directions and robust approximation of the primary tool axis. A novel normalization method is proposed for the shape and intensity consistency of instruments to improve the detection. Moreover, a novel 3-D Gabor wavelet transformation is introduced and optimally designed for revealing the instrument voxels in the volume, while remaining generic to several medical instruments and transducer types. Experiments on diverse data sets, including in vivo data from patients, show that for a given transducer and an instrument type, high detection accuracies are achieved with position errors smaller than the instrument diameter in the 0.5-1.5-mm range on average.

2-D and 3-D Ultrasound for Tumor Volume Analysis: A Prospective Study

Ultrasound (US) allows real-time tumor assessment. We evaluated the volumetric limits of 2-D and 3-D US, compared with magnetic resonance imaging (MRI), with a prospective institutional review board-approved clinical evaluation of US-to-MRI volumetric correlation. US images of pre- and post-neoadjuvant breast cancers were obtained. Volume discrepancy was evaluated with the non-parametric Wilcoxon signed-rank test. Expected inter-observer variability <14% was evaluated as relative paired difference (RPD); clinical relevance was gauged with the volumetric standard error of the mean (SEM). For 42 patients, 133 of 170 US examinations were evaluable. For tumors ≤20 cm³, both highly correlated to MRI with RPD within inter-observer variability and Pearson's correlation up to 0.86 (0.80 before and 0.86 after neoadjuvant chemotherapy, respectively). Lesions 20-40 cm³ had US-to-MRI discrepancy within inter-observer variability for 2-D (RPD: 13%), but not 3-D (RPD: 27%) US (SEM: 1.47 cm³ for 2-D, SEM: 2.28 cm³ for 3-D), suggesting clinical utility. Tumors >40 cm³ correlated poorly. Tumor volumes ≤20 cm³ exhibited a good correlation to MRI. Studies of clinical applications are warranted.

A Filtering Approach for Image-Guided Surgery With a Highly Articulated Surgical Snake Robot

GOAL

The objective of this paper is to introduce a probabilistic filtering approach to estimate the pose and internal shape of a highly flexible surgical snake robot during minimally invasive surgery.

METHODS

Our approach renders a depiction of the robot that is registered to preoperatively reconstructed organ models to produce a 3-D visualization that can be used for surgical feedback. Our filtering method estimates the robot shape using an extended Kalman filter that fuses magnetic tracker data with kinematic models that define the motion of the robot. Using Lie derivative analysis, we show that this estimation problem is observable, and thus, the shape and configuration of the robot can be successfully recovered with a sufficient number of magnetic tracker measurements.

RESULTS

We validate this study with benchtop and in-vivo image-guidance experiments in which the surgical robot was driven along the epicardial surface of a porcine heart.

CONCLUSION

This paper introduces a filtering approach for shape estimation that can be used for image guidance during minimally invasive surgery.

SIGNIFICANCE

The methods being introduced in this paper enable informative image guidance for highly articulated surgical robots, which benefits the advancement of robotic surgery.

High-intensity focused ultrasound as a treatment option in renal cell carcinoma

Due to the widespread use of modern imaging modalities, small renal masses are discovered incidentally at increasing rates. Advances in minimally invasive technologies have changed the treatment options for renal cell carcinoma. High-intensity focused ultrasound aims to completely ablate renal tumors in a noninvasive manner. Experimental studies have demonstrated principle feasibility and safety of the technology. However, clinical studies on renal cell carcinoma are very limited and no substantial oncologic results are available to date. Major technical improvements are mandatory to enable high-intensity focused ultrasound as an effective treatment option for patients with small renal masses.

Tubular structure enhancement for surgical instrument detection in 3D ultrasound

Three-dimensional ultrasound has been an effective imaging modality for diagnostics and is now an emerging modality for image-guided minimally-invasive interventions since it enables visualization of both instruments and tissue. Challenges to ultrasound-guided interventions arise, however, due to the low signal-to-noise ratio and the imaging artifacts created by the interventional instruments. Metallic instruments, in particular, are strong scatters and so produce a variety of artifacts. For many interventions, the manual or robotic instrument is comprised of a long curved tubular structure with specialized tooling at its tip. Toward the goal of developing a surgical navigation system, this paper proposes an image processing algorithm for enhancing the tubular structure of imaged instruments while also reducing imaging artifacts. Experiments are presented to evaluate the effectiveness of the approach in the context of robotic instruments whose shape comprises a smooth curve along their length.

Robotics and imaging in congenital heart surgery

The initial success seen in adult cardiac surgery with the application of available robotic systems has not been realized as broadly in pediatric cardiac surgery. The main obstacles include extended set-up time and complexity of the procedures, as well as the large size of the instruments with respect to the size of the child. Moreover, while the main advantage of robotic systems is the ability to minimize incision size, for intracardiac repairs, cardiopulmonary bypass is still required. Catheter-based interventions, on the other hand, have expanded rapidly in both application as well as the complexity of procedures and lesions being treated. However, despite the development of sophisticated devices, robotic systems to aid catheter procedures have not been commonly applied in children. In this article, we describe new catheter-like robotic delivery platforms, which facilitate safe navigation and enable complex repairs, such as tissue approximation and fixation, and tissue removal, inside the beating heart. Additional features including the tracking of rapidly moving tissue targets and novel imaging approaches are described, along with a discussion of future prospects for steerable robotic systems.

Enabling 3D Ultrasound Procedure Guidance through Enhanced Visualization

Real-time 3D ultrasound (3DUS) imaging offers improved spatial orientation information relative to 2D ultrasound. However, in order to improve its efficacy in guiding minimally invasive intra-cardiac procedures where real-time visual feedback of an instrument tip location is crucial, 3DUS volume visualization alone is inadequate. This paper presents a set of enhanced visualization functionalities that are able to track the tip of an instrument in slice views at real-time. User study with in vitro porcine heart indicates a speedup of over 30% in task completion time.

Tubular Enhanced Geodesic Active Contours for Continuum Robot Detection using 3D Ultrasound

Three dimensional ultrasound is a promising imaging modality for minimally invasive robotic surgery. As the robots are typically metallic, they interact strongly with the sound waves in ways that are not modeled by the ultrasound system's signal processing algorithms. Consequently, they produce substantial imaging artifacts that can make image guidance difficult, even for experienced surgeons. This paper introduces a new approach for detecting curved continuum robots in 3D ultrasound images. The proposed approach combines geodesic active contours with a speed function that is based on enhancing the "tubularity" of the continuum robot. In particular, it takes advantage of the known robot diameter along its length. It also takes advantage of the fact that the robot surface facing the ultrasound probe provides the most accurate image. This method, termed Tubular Enhanced Geodesic Active Contours (TEGAC), is demonstrated through ex vivo intracardiac experiments to offer superior performance compared to conventional active contours.

Passive markers for tracking surgical instruments in real-time 3-D ultrasound imaging

A family of passive echogenic markers is presented by which the position and orientation of a surgical instrument can be determined in a 3-D ultrasound volume, using simple image processing. Markers are attached near the distal end of the instrument so that they appear in the ultrasound volume along with the instrument tip. They are detected and measured within the ultrasound image, thus requiring no external tracking device. This approach facilitates imaging instruments and tissue simultaneously in ultrasound-guided interventions. Marker-based estimates of instrument pose can be used in augmented reality displays or for image-based servoing. Design principles for marker shapes are presented that ensure imaging system and measurement uniqueness constraints are met. An error analysis is included that can be used to guide marker design and which also establishes a lower bound on measurement uncertainty. Finally, examples of marker measurement and tracking algorithms are presented along with experimental validation of the concepts.

A fast slam approach to freehand 3-d ultrasound reconstruction for catheter ablation guidance in the left atrium

We present a method for real-time, freehand 3D ultrasound (3D-US) reconstruction of moving anatomy, with specific application towards guiding the catheter ablation procedure in the left atrium. Using an intracardiac echo (ICE) catheter with a pose (position/orientation) sensor mounted to its tip, we continually mosaic 2D-ICE images of a left atrium phantom model to form a 3D-US volume. Our mosaicing strategy employs a probabilistic framework based on simultaneous localization and mapping (SLAM), a technique commonly used in mobile robotics for creating maps of unexplored environments. The measured ICE catheter tip pose provides an initial estimate for compounding 2D-ICE image data into the 3D-US volume. However, we simultaneously consider the overlap-consistency shared between 2D-ICE images and the 3D-US volume, computing a "corrected" tip pose if need be to ensure spatially-consistent reconstruction. This allows us to compensate for anatomic movement and sensor drift that would otherwise cause motion artifacts in the 3D-US volume. Our approach incorporates 2D-ICE data immediately after acquisition, allowing us to continuously update the registration parameters linking sensor coordinates to 3D-US coordinates. This, in turn, enables real-time localization and display of sensorized therapeutic catheters within the 3D-US volume for facilitating procedural guidance.

Detection of Curved Robots using 3D Ultrasound

Three-dimensional ultrasound can be an effective imaging modality for image-guided interventions since it enables visualization of both the instruments and the tissue. For robotic applications, its realtime frame rates create the potential for image-based instrument tracking and servoing. These capabilities can enable improved instrument visualization, compensation for tissue motion as well as surgical task automation. Continuum robots, whose shape comprises a smooth curve along their length, are well suited for minimally invasive procedures. Existing techniques for ultrasound tracking, however, are limited to straight, laparoscopic-type instruments and thus are not applicable to continuum robot tracking. Toward the goal of developing tracking algorithms for continuum robots, this paper presents a method for detecting a robot comprised of a single constant curvature in a 3D ultrasound volume. Computational efficiency is achieved by decomposing the six-dimensional circle estimation problem into two sequential three-dimensional estimation problems. Simulation and experiment are used to evaluate the proposed method.

Real-time 4D ultrasound mosaicing and visualization

Intra-cardiac 3D ultrasound imaging has enabled new minimally invasive procedures. Its narrow field of view, however, limits its efficacy in guiding beating heart procedures where geometrically complex and spatially extended moving anatomic structures are often involved. In this paper, we present a system that performs electrocardiograph gated 4D mosaicing and visualization of 3DUS volumes. Real-time operation is enabled by GPU implementation. The method is validated on phantom and porcine heart data.

Advantages and limitations of intraoperative 3D ultrasound in neurosurgery. Technical note

Three-dimensional ultrasound (US) technology is supposed to help combat some of the orientation difficulties inherent to two-dimensional US. Contemporary navigation solutions combine reconstructed 3D US images with common navigation images and support orientation. New real-time 3D US (without neuronavigation) is more time effective, but whether it further assists in orientation remains to be determined. An integrated US system (IGSonic, VectorVision, BrainLAB, Munich Germany) and a non-integrated system with real-time 3D US (iU22, Philips, Bothell, USA) were recently compared in neurosurgical procedures in our group. The reconstructed navigation view was time-consuming, but images were displayed in familiar planes (e.g., axial, sagittal, coronal). Further potential applications of US angiography and pure US navigation are possible. Real-time 3D images were displayed without the need for an additional acquisition and reconstruction process, but spatial orientation remained challenging in this preliminary testing phase. Reconstructed 3D US navigation appears to be superior with respect to spatial orientation, and the technique can be combined with other imaging data. However, the potential of real-time 3D US imaging is promising.

Robotic tissue tracking for beating heart mitral valve surgery

The rapid motion of the heart presents a significant challenge to the surgeon during intracardiac beating heart procedures. We present a 3D ultrasound-guided motion compensation system that assists the surgeon by synchronizing instrument motion with the heart. The system utilizes the fact that certain intracardiac structures, like the mitral valve annulus, have trajectories that are largely constrained to translation along one axis. This allows the development of a real-time 3D ultrasound tissue tracker that we integrate with a 1 degree-of-freedom (DOF) actuated surgical instrument and predictive filter to devise a motion tracking system adapted to mitral valve annuloplasty. In vivo experiments demonstrate that the system provides highly accurate tracking (1.0 mm error) with 70% less error than manual tracking attempts.

Model fitting using RANSAC for surgical tool localization in 3-D ultrasound images

Ultrasound guidance is used for many surgical interventions such as biopsy and electrode insertion. We present a method to localize a thin surgical tool such as a biopsy needle or a microelectrode in a 3-D ultrasound image. The proposed method starts with thresholding and model fitting using random sample consensus for robust localization of the axis. Subsequent local optimization refines its position. Two different tool image models are presented: one is simple and fast and the second uses learned a priori information about the tool's voxel intensities and the background. Finally, the tip of the tool is localized by finding an intensity drop along the axis. The simulation study shows that our algorithm can localize the tool at nearly real-time speed, even using a MATLAB implementation, with accuracy better than 1 mm. In an experimental comparison with several alternative localization methods, our method appears to be the fastest and the most robust one. We also show the results on real 3-D ultrasound data from a PVA cryogel phantom, turkey breast, and breast biopsy.

A comparison study assessing the feasibility of ultrasound-initialized deformable bone models

This article presents a feasibility and evaluation study for using 2D ultrasound in conjunction with our statistical deformable bone model within the scope of computer-assisted surgery. The final aim is to provide the surgeon with enhanced 3D visualization for surgical navigation in orthopedic surgery without the need for preoperative CT or MRI scans. We unified our earlier work to combine several automatic methods for statistical bone shape prediction and ultrasound segmentation and calibration to provide the intended rapid and accurate visualization. We compared the use of a tracked digitizing pointer and ultrasound for acquiring landmarks and bone surface points for the estimation of two cast proximal femurs.

An Ultrasound-based Localization Algorithm for Catheter Ablation Guidance in the Left Atrium

We present a method for catheter localization in the left atrium based on the unscented particle filter (UPF), a Monte Carlo method employed in stochastic state estimation. Using an intracardiac echo (ICE) ultrasound catheter, we acquire ultrasound images of the atrium from multiple configurations and iteratively determine the catheter’s pose with respect to anatomy. At each time step, the catheter’s change in pose is determined using either a six-degree-of-freedom electromagnetic pose sensor or a robotic guide catheter whose kinematics serve as a pseudo-pose measurement. Sensor and kinematic model uncertainties are explicitly considered when computing the localization estimate. Acquired ultrasound images are compared with simulated ultrasound images based on segmented computed tomography (CT) or magnetic resonance (MR) data of the left atrium. The results of these comparisons are used to refine the localization estimate. After considering less than 30 seconds’ worth of ICE data, our algorithm converges to an accurate pose estimate. Furthermore, our algorithm is robust to sensor drift and kinematic model errors, as well as gradual, unmodeled movements in the anatomy. Such problems typically complicate traditional image-based localization.

Robotic Motion Compensation for Beating Heart Intracardiac Surgery

3D ultrasound imaging has enabled minimally invasive, beating heart intracardiac procedures. However, rapid heart motion poses a serious challenge to the surgeon that is compounded by significant time delays and noise in 3D ultrasound. This paper investigates the concept of using a one-degree-of-freedom motion compensation system to synchronize with tissue motions that may be approximated by 1D motion models. We characterize the motion of the mitral valve annulus and show that it is well approximated by a 1D model. The subsequent development of a motion compensation instrument (MCI) is described, as well as an extended Kalman filter (EKF) that compensates for system delays. The benefits and robustness of motion compensation are tested in user trials under a series of non-ideal tracking conditions. Results indicate that the MCI provides an approximately 50% increase in dexterity and 50% decrease in force when compared with a solid tool, but is sensitive to time delays. We demonstrate that the use of the EKF for delay compensation restores performance, even in situations of high heart rate variability. The resulting system is tested in an in vitro 3D ultrasound-guided servoing task, yielding accurate tracking (1.15 mm root mean square) in the presence of noisy, time-delayed 3D ultrasound measurements.

3D echocardiographic visualization for intracardiac beating heart surgery and intervention

Three-dimensional echocardiography has emerged as an essential tool for visualizing cardiac anatomy and for making more accurate measurements of cardiac structure and function. Recently, improvements in 3D beam-forming and transducer technologies have allowed higher resolution imaging from a transesophageal echocardiographic probe. This is creating new avenues for real-time visualization of intracardiac procedures without the need for cardiopulmonary bypass or opening the beating heart. Evolutions in visualization will allow a wider array of reparative procedures to be performed minimally invasively within a beating heart.

Parallel integral projection transform for straight electrode localization in 3-D ultrasound images

In surgical practice, small metallic instruments are frequently used to perform various tasks inside the human body. We address the problem of their accurate localization in the tissue. Recent experiments using medical ultrasound have shown that this modality is suitable for real-time visualization of anatomical structures as well as the position of surgical instruments. We propose an image-processing algorithm that permits automatic estimation of the position of a line-segment-shaped object. This method was applied to the localization of a thin metallic electrode in biological tissue. We show that the electrode axis can be found through maximizing the parallel integral projection transform that is a form of the Radon transform. To accelerate this step, hierarchical mesh-grid algorithm is implemented. Once the axis position is known, localization of the electrode tip is performed. The method was tested on simulated images, on ultrasound images of a tissue mimicking phantom containing a metallic electrode, and on real ultrasound images from breast biopsy. The results indicate that the algorithm is robust with respect to variations in electrode position and speckle noise. Localization accuracy is of the order of hundreds of micrometers and is comparable to the ultrasound system axial resolution.

Stereo display of 3D ultrasound images for surgical robot guidance

The recent advent of real-time 3-D ultrasound (3DUS) imaging enables a variety of surgical procedures to be performed within the beating heart. Implementation of these procedures is hampered by the difficulty of manipulating tissue guided by the distorted, low resolution 3DUS images and the dexterity constraints imposed by the confined intracardiac space. This paper investigates the use of surgical robotics in conjunction with 3DUS to overcome these limitations. In addition, it describes the development of a graphics processor based volume Tenderer for real-time stereo visualization of the ultrasound data. Stereo displayed 3DUS was compared to ID-displayed 3DUS and endoscopic guidance with a user study. Five subjects performed in vitro surgical tasks using a surgical robot. Results indicate that subjects were able to complete surgical tasks 35 % faster with stereo-displayed 3DUS images compared to conventional two dimensional display of 3DUS.

Fast block flow tracking of atrial septal defects in 4D echocardiography

We are working to develop beating-heart atrial septal defect (ASD) closure techniques using real-time 3D ultrasound guidance. The major image processing challenges are the low-image quality and the processing of information at high-frame rate. This paper presents comparative results for ASD tracking in time sequences of 3D volumes of cardiac ultrasound. We introduce a block flow technique, which combines the velocity computation from optical flow for an entire block with template matching. Enforcing adapted similarity constraints to both the previous and first frames ensures optimal and unique solutions. We compare the performance of the proposed algorithm with that of block matching and region-based optical flow on eight in vivo 4D datasets acquired from porcine beating-heart procedures. Results show that our technique is more stable and has higher sensitivity than both optical flow and block matching in tracking ASDs. Computing velocity at the block level, our technique tracks ASD motion at 2 frames/s, much faster than optical flow and comparable in computation cost to block matching, and shows promise for real-time (30 frames/s). We report consistent results on clinical intra-operative images and retrieve the cardiac cycle (in ungated images) from error analysis. Quantitative results are evaluated on synthetic data with maximum tracking errors of 1 voxel.

3D ultrasound-guided motion compensation system for beating heart mitral valve repair

Beating heart intracardiac procedures promise significant benefits for patients, however, the fast motion of the heart poses serious challenges to surgeons. We present a new 3D ultrasound-guided motion (3DUS) compensation system that synchronizes instrument motion with the heart. The system utilizes the fact that the motion of some intracardiac structures, including the mitral valve annulus, is largely constrained to translation along one axis. This allows the development of a real-time 3DUS tissue tracker which we integrate with a 1 degree-of-freedom actuated surgical instrument, real-time 3DUS instrument tracker, and predictive filter to devise a system with synchronization accuracy of 1.8 mm RMSE. User studies involving the deployment of surgical anchors in a simulated mitral annuloplasty procedure demonstrate that the system increases success rates by over 100%. Furthermore, it enables more careful anchor deployment by reducing forces to the tissue by 50% while allowing instruments to remain in contact with the tissue for longer periods.

Real-time tracking and shape analysis of atrial septal defects in 3D echocardiography

RATIONALE AND OBJECTIVES

Real-time cardiac ultrasound (US) allows monitoring the heart motion during intracardiac beating heart procedures. Our application assists pediatric atrial septal defect (ASD) closure techniques using real-time 3D US guidance and rigid instruments. ASD tracking is also an important tool for facilitating systematic clinical studies of the dynamic behavior of the intra-atrial communication. One major image processing challenge is associated with the required processing of information at high frame rate, especially given the low image quality.

MATERIALS AND METHODS

We present an optimization scheme for a block flow technique, which combines the probability-based velocity computation for an entire block (a 3D volume centered on the ASD) with cyclic template matching. The adapted similarity imposes constraints both locally (from frame to frame) to conserve energy, and globally (from a reference template) to minimize cumulative errors. The algorithm is optimized for fast and reliable results. For tests, we use three intra-operational 4D ultrasound sequences of clinical infant beating hearts with ASD.

RESULTS

Computing velocity at the block level with an optimized scheme, our technique tracks ASD motion at a frequency of 60 frames/s on clinical 4D datasets. Results are stable and accurate for changes in resolution and block size. In particular, we show robust real-time tracking and preliminary segmentation results of the ASD shape, size and orientation as a function of time.

CONCLUSIONS

We present an optimized block flow technique for real-time tracking of ASD to assist in minimally invasive beating heart surgery. Our method proposes the standard use of references for processing repetitive data. This paper represents, to our knowledge, the first study on the dynamic morphology of ASD that takes into account the angular effect introduced by the slanted position of the intra-atrial communication with respect to the US probe.

Imaging artifacts of medical instruments in ultrasound-guided interventions

OBJECTIVE

Real-time 3-dimensional (3D) ultrasound imaging has the potential to become a dominant imaging technique for minimally invasive surgery. One barrier to its widespread use is that surgical instruments generate imaging artifacts, which can obfuscate their location, orientation, and geometry and obscure nearby tissue. The purpose of this study was to identify and describe the types of artifacts which could be produced by metallic instruments during interventions guided by 3D ultrasound imaging.

METHODS

Three imaging studies were performed. First, imaging artifacts from stainless steel rods were identified in vitro and acoustically characterized. Second, 3 typical minimally invasive instruments were imaged (in vitro and in vivo), and their artifacts were analyzed. The third study compared the intensity of imaging artifacts (in vitro and in vivo) from stainless steel rods with rods composed of 3 different materials and stainless steel rods with roughened and coated surfaces.

RESULTS

For the stainless steel rods, all observed artifacts are described and illustrated, and their physical origins are explained. Artifacts from the 3 minimally invasive instruments are characterized with the use of the artifacts observed with the rods. Finally, it is shown that artifacts can be greatly reduced through the use of alternate materials or by surface modification.

CONCLUSIONS

Instrument artifacts in 3D ultrasound images can be more confusing than those from the same instruments imaged in 2 dimensions. Real-time 3D ultrasound imaging can, however, be used effectively for in vivo imaging of minimally invasive instruments by using artifact mitigation techniques, including careful selection of probe and incision locations, as well as by instrument modification.

Statistical segmentation of surgical instruments in 3-D ultrasound images

The recent development of real-time 3-D ultrasound (US) enables intracardiac beating-heart procedures, but the distorted appearance of surgical instruments is a major challenge to surgeons. In addition, tissue and instruments have similar gray levels in US images and the interface between instruments and tissue is poorly defined. We present an algorithm that automatically estimates instrument location in intracardiac procedures. Expert-segmented images are used to initialize the statistical distributions of blood, tissue and instruments. Voxels are labeled through an iterative expectation-maximization algorithm using information from the neighboring voxels through a smoothing kernel. Once the three classes of voxels are separated, additional neighboring information is combined with the known shape characteristics of instruments to correct for misclassifications. We analyze the major axis of segmented data through their principal components and refine the results by a watershed transform, which corrects the results at the contact between instrument and tissue. We present results on 3-D in-vitro data from a tank trial and 3-D in-vivo data from cardiac interventions on porcine beating hearts, using instruments of four types of materials. The comparison of algorithm results to expert-annotated images shows the correct segmentation and position of the instrument shaft.

GPU based real-time instrument tracking with three-dimensional ultrasound

Real-time three-dimensional ultrasound enables new intracardiac surgical procedures, but the distorted appearance of instruments in ultrasound poses a challenge to surgeons. This paper presents a detection technique that identifies the position of the instrument within the ultrasound volume. The algorithm uses a form of the generalized Radon transform to search for long straight objects in the ultrasound image, a feature characteristic of instruments and not found in cardiac tissue. When combined with passive markers placed on the instrument shaft, the full position and orientation of the instrument is found in 3D space. This detection technique is amenable to rapid execution on the current generation of personal computer graphics processor units (GPU). Our GPU implementation detected a surgical instrument in 31 ms, sufficient for real-time tracking at the 25 volumes per second rate of the ultrasound machine. A water tank experiment found instrument orientation errors of 1.1 degrees and tip position errors of less than 1.8mm. Finally, an in vivo study demonstrated successful instrument tracking inside a beating porcine heart.

Three-Dimensional Echocardiographic Technology

This article addresses the current state of the art of technology in three-dimensional echocardiography as it applies to transducer design, beam forming, display, and quantification. Because three-dimensional echocardiography encompasses many technical and clinical areas, this article reviews its strengths and limitations and concludes with an analysis of what to use when.

MRI-guided laparoscopic and robotic surgery for malignancies

Endoscopic surgery has some clear benefits, but it also has some disadvantages in reducing surgeons' normal dexterity and limiting their ability to deal with difficult situations. Computer-aided surgery has been proposed to overcome some of the drawbacks of traditional minimally invasive surgery. The proposed systems make possible a secure, precise procedure with no limitations on the operator's freedom of movement. Image-guided surgery is a new technical tool in surgical oncology. Interventional magnetic resonance imaging (MRI) has entered a new stage in which computer-based techniques play an expanding role in planning, monitoring, and controlling procedures. MRI-guided surgery not only represents a technical challenge but is a transformation from conventional hand-eye coordination to interactive navigational operations. We have recently developed an MRI-guided robot-assisted interventional surgical system as well as an MRI-compatible endoscope. They allow the performance of precise image-guided interventional therapy and endoscopic surgery. MRI-guided laparoscopic surgery is now feasible for malignancies and will play an important part in the development of minimally invasive therapy.

GPU Based Real-Time Instrument Tracking with Three Dimensional Ultrasound

Real-time 3D ultrasound can enable new image-guided surgical procedures, but high data rates prohibit the use of traditional tracking techniques. We present a new method based on the modified Radon transform that identifies the axis of instrument shafts as bright patterns in planar projections. Instrument rotation and tip location are then determined using fiducial markers. These techniques are amenable to rapid execution on the current generation of personal computer graphics processor units (GPU). Our GPU implementation detected a surgical instrument in 31 ms, sufficient for real-time tracking at the 26 volumes per second rate of the ultrasound machine. A water tank experiment found instrument tip position errors of less than 0.2 mm, and an in vivo study tracked an instrument inside a beating porcine heart. The tracking results showed good correspondence to the actual movements of the instrument.

3D ultrasound in robotic surgery: performance evaluation with stereo displays

BACKGROUND

The recent advent of real-time 3D ultrasound (3DUS) imaging enables a variety of new surgical procedures. These procedures are hampered by the difficulty of manipulating tissue guided by the distorted, low-resolution 3DUS images. To lessen the effects of these limitations, we investigated stereo displays and surgical robots for 3DUS-guided procedures.

METHODS

By integrating real-time stereo rendering of 3DUS with the binocular display of a surgical robot, we compared stereo-displayed 3DUS with normally displayed 3DUS. To test the efficacy of stereo-displayed 3DUS, eight surgeons and eight non-surgeons performed in vitro tasks with the surgical robot.

RESULTS

Error rates dropped by 50% with a stereo display. In addition, subjects completed tasks faster with the stereo-displayed 3DUS as compared to normal-displayed 3DUS. A 28% decrease in task time was seen across all subjects.

CONCLUSIONS

The results highlight the importance of using a stereo display. By reducing errors and increasing speed, it is an important enhancement to 3DUS-guided robotics procedures.

Three-dimensional echo-guided beating heart surgery without cardiopulmonary bypass: atrial septal defect closure in a swine model

OBJECTIVE

In this study, we tested 3 techniques of atrial septal defect closure under real-time 3-dimensional echocardiography guidance in a swine model.

METHODS

The operations were conducted under the sole guidance of a modified real-time 3-dimensional echocardiography guidance system with a x4 matrix transducer (Sonos 7500, Philips Medical Systems, Andover, Mass). Eighteen swine were anesthetized, and after median sternotomy, the echo probe was applied directly to the surface of the right atrium. To create an atrial septal defect, balloon atrial septostomy and atrial septal defect enlargement were performed. Subsequently, 3 different techniques of atrial septal defect closure were attempted: group I, direct suture closure; group II, closure of the atrial septal defect using the Amplatzer device (AGA Medical Corp, Golden Valley, Minn); and group III, patch closure of the atrial septal defect (n = 6 each).

RESULTS

Real-time 3-dimensional echocardiography guidance provided sufficient spatial resolution and a satisfactory frame rate to provide a "virtual surgeon's view" of the relevant anatomy during the entire procedure. All atrial septal defects were enlarged, and the mean final size was 8.5 +/- 1.8 mm. Atrial septal defect closure was successfully accomplished with all the 3 surgical techniques examined. In groups I and III, the needles (1-3 sutures) and staples (6-12 staples) penetrated the tissue and patch material consistently, whereas in group III, the Amplatzer atrial septal defect device was easily deployed. There was no incident device/staple embolization or air introduction. Neither intraoperative 2-dimensional color Doppler echocardiography nor postmortem macro-evaluation revealed any residual shunts.

CONCLUSIONS

Beating heart atrial septal defect closure under real-time 3-dimensional echocardiographic guidance is feasible and, unlike catheter-based devices, applicable for any type of secundum atrial septal defect.

High-intensity focused ultrasound in the treatment of solid tumours

Traditionally, surgery has been the only cure for many solid tumours. Technological advances have catalysed a shift from open surgery towards less invasive techniques. Laparoscopic surgery and minimally invasive techniques continue to evolve, but for decades high-intensity focused ultrasound has promised to deliver the ultimate objective - truly non-invasive tumour ablation. Only now, however, with recent improvements in imaging, has this objective finally emerged as a real clinical possibility.

Hybrid procedures in pediatric cardiac surgery

Hybrid pediatric cardiac surgery is an emerging field that combines skills and techniques used by pediatric cardiac surgeons and interventional pediatric cardiologists. This article describes the emerging indications and techniques in hybrid pediatric cardiac surgery and discusses potential future applications. It focuses on peratrial and perventricular septal defect closure, intraoperative stenting, hybrid stage I palliation for hypoplastic left heart syndrome, and percutaneous valve implantation.

Robotic pediatric cardiac surgery: Present and future perspectives

Advances in robotic technology and imaging systems have enabled the broad application of minimally invasive techniques in cardiac surgery, including coronary artery bypass grafting and mitral valve repair in adults. In pediatric cardiac surgery, however, current robotic systems have been used primarily to facilitate thoracoscopic pediatric procedures on extracardiac lesions, such as ligation of patent ductus and division of vascular rings. The use of smaller instruments with sophisticated robotic wrists may make it possible to perform more complex extracardiac procedures even in young infants. Additionally, future technological improvements, including incorporation of tactile feedback, instrument tracking, and intracardiac imaging (such as real-time 3-dimensional echocardiography), may enable intracardiac robotic surgery to be performed in children. This article reviews the current and potential future applications of pediatric robotic surgery and the developmental work required to enable performance of these procedures, along with an overview of the problems associated with the use of current robotic surgical systems in children.

Three-dimensional echocardiography–guided beating-heart surgery without cardiopulmonary bypass: A feasibility study

BACKGROUND

There is no current acceptable approach for intracardiac beating-heart interventions. We have adapted real-time 3-dimensional echocardiography with specialized instrumentation to facilitate beating-heart repair of atrial septal defects and mitral valve plasty to investigate the feasibility of real-time 3-dimensional echocardiography-guided cardiac surgery.

METHODS

In experiment I a modified real-time 3-dimensional echocardiography system with x4 matrix transducer was compared with 2-dimensional echocardiography in the performance of common surgical tasks. Completion times, deviation from an ideal trajectory, and an echogenic target were measured. In experiment II porcine atrial septal defects were closed with an original semiautomatic suturing device (n = 4) and with a 5-mm endoscopic stapler and a pericardial or polytetrafluoroethylene patch (n = 4). In experiment III a pulsatile porcine mitral valve model was developed, and suture placement through the anterior and posterior mitral leaflets was performed (n = 8). During all experiments, the operator was blinded to the target and operated on only with ultrasonic guidance.

RESULTS

In experiment I, compared with 2-dimensional echocardiographic guidance, completion times improved by 21% ( P <.01) with high-trajectory accuracy, and suture deviation was significantly smaller (2-dimensional echocardiography, 5.4 +/- 2.7 mm; 3-dimensional echocardiography, 1.7 +/- 0.7 mm; P <.05) in real-time 3-dimensional echocardiography-guided tasks. In experiments II and III in both atrial septal defect closure and mitral valve plasty, real-time 3-dimensional echocardiography provided satisfactory images and sufficient anatomic detail for suturing and patch deployment. All surgical tasks were successfully performed with accuracy.

CONCLUSIONS

Real-time 3-dimensional echocardiography provides adequate imaging and anatomic detail to act as a sole guide for surgical task performance. These initial experiments demonstrate the feasibility of beating-heart direct or patch closure of atrial septal defects and mitral valve plasty without cardiopulmonary bypass.