3D ultrasound-CT registration of the liver using combined landmark-intensity information

Guided ultrasound acquisition for nonrigid image registration using reinforcement learning

We propose a guided registration method for spatially aligning a fixed preoperative image and untracked ultrasound image slices. We exploit the unique interactive and spatially heterogeneous nature of this application to develop a registration algorithm that interactively suggests and acquires ultrasound images at optimised locations (with respect to registration performance). Our framework is based on two trainable functions: (1) a deep hyper-network-based registration function, which is generalisable over varying location and deformation, and adaptable at test-time; (2) a reinforcement learning function for producing test-time estimates of image acquisition locations and adapted deformation regularisation (the latter is required due to varying acquisition locations). We evaluate our proposed method with real preoperative patient data, and simulated intraoperative data with variable field-of-view. In addition to simulation of intraoperative data, we simulate global alignment based on previous work for efficient training, and investigate probe-level guidance towards an improved deformable registration. The evaluation in a simulated environment shows statistically significant improvements in overall registration performance across a variety of metrics for our proposed method, compared to registration without acquisition guidance or adaptable deformation regularisation, and to commonly used classical iterative methods and learning-based registration. For the first time, efficacy of proactive image acquisition is demonstrated in a simulated surgical interventional registration, in contrast to most existing work addressing registration post-data-acquisition, one of the reasons we argue may have led to previously under-constrained nonrigid registration in such applications. Code: https://github.com/s-sd/rl_guided_registration.

From Coarse to Fine: Non-Rigid Sparse-Dense Registration for Deformation-Aware Liver Surgical Navigation

OBJECTIVE

Intraoperative liver deformation poses a considerable challenge during liver surgery, causing significant errors in image-guided surgical navigation systems. This study addresses a critical non-rigid registration problem in liver surgery: the alignment of intrahepatic vascular trees. The goal is to deform the complete vascular shape extracted from preoperative Computed Tomography (CT) volume, aligning it with sparse vascular contour points obtained from intraoperative ultrasound (iUS) images. Challenges arise due to the intricate nature of slender vascular branches, causing existing methods to struggle with accuracy and vascular self-intersection.

METHODS

We present a novel non-rigid sparse-dense registration pipeline structured in a coarse-to-fine fashion. In the initial coarse registration stage, we introduce a parametrization deformation graph and a Welsch function-based error metric to enhance convergence and robustness of non-rigid registration. For the fine registration stage, we propose an automatic curvature-based algorithm to detect and eliminate overlapping regions. Subsequently, we generate the complete vascular shape using posterior computation of a Gaussian Process Shape Model.

RESULTS

Experimental results using simulated data demonstrate the accuracy and robustness of our proposed method. Evaluation results on the target registration error of tumors highlight the clinical significance of our method in tumor location computation. Comparative analysis against related methods reveals superior accuracy and competitive efficiency of our approach. Moreover, Ex vivo swine liver experiments and clinical experiments were conducted to evaluate the method's performance.

CONCLUSION

The experimental results emphasize the accurate and robust performance of our proposed method.

SIGNIFICANCE

Our proposed non-rigid registration method holds significant application potential in clinical practice.

A water-immersible scanning mirror with hybrid polymer and elastomer hinges for high-speed and wide-field 3D ultrasound imaging

This paper reports a new water-immersible single-axis scanning mirror using hybrid polymer and elastomer hinges to achieve both high scanning resonance frequencies and large tilting angles for high-speed and wide-field 3D ultrasound imaging. To demonstrate the concept, a prototype scanning mirror is designed, fabricated, and characterized. The fast- and slow-scanning were achieved by integrating stiff BoPET (biaxially oriented polyethylene terephthalate) and soft elastomer PDMS (Polydimethylsiloxane) hinges, respectively. The testing results have shown a resonance frequency of 270 Hz for the BoPET hinges and a resonance frequency of 10 Hz for the PDMS hinges when the scanning mirror was immersed in water. 3D ultrasound imaging is demonstrated by combining the fast- and slow-scanning together to provide both an augmented field of view (FoV) and high local imaging volume rate.

A robust and automatic CT-3D ultrasound registration method based on segmentation, context, and edge hybrid metric

BACKGROUND

The fusion of computed tomography (CT) and ultrasound (US) image can enhance lesion detection ability and improve the success rate of liver interventional radiology. The image-based fusion methods encounter the challenge of registration initialization due to the random scanning pose and limited field of view of US. Existing automatic methods those used vessel geometric information and intensity-based metric are sensitive to parameters and have low success rate. The learning-based methods require a large number of registered datasets for training.

PURPOSE

The aim of this study is to provide a fully automatic and robust US-3D CT registration method without registered training data and user-specified parameters assisted by the revolutionary deep learning-based segmentation, which can further be used for preparing training samples for the study of learning-based methods.

METHODS

We propose a fully automatic CT-3D US registration method by two improved registration metrics. We propose to use 3D U-Net-based multi-organ segmentation of US and CT to assist the conventional registration. The rigid transform is searched in the space of any paired vessel bifurcation planes where the best transform is decided by a segmentation overlap metric, which is more related to the segmentation precision than Dice coefficient. In nonrigid registration phase, we propose a hybrid context and edge based image similarity metric with a simple mask that can remove most noisy US voxels to guide the B-spline transform registration. We evaluate our method on 42 paired CT-3D US datasets scanned with two different US devices from two hospitals. We compared our methods with other exsiting methods with both quantitative measures of target registration error (TRE) and the Jacobian determinent with paired t-test and qualitative registration imaging results.

RESULTS

The results show that our method achieves fully automatic rigid registration TRE of 4.895 mm, deformable registration TRE of 2.995 mm in average, which outperforms state-of-the-art automatic linear methods and nonlinear registration metrics with paired t-test's p value less than 0.05. The proposed overlap metric achieves better results than self similarity description (SSD), edge matching (EM), and block matching (BM) with p values of 1.624E-10, 4.235E-9, and 0.002, respectively. The proposed hybrid edge and context-based metric outperforms context-only, edge-only, and intensity statistics-only-based metrics with p values of 0.023, 3.81E-5, and 1.38E-15, respectively. The 3D US segmentation has achieved mean Dice similarity coefficient (DSC) of 0.799, 0.724, 0.788, and precision of 0.871, 0.769, 0.862 for gallbladder, vessel, and branch vessel, respectively.

CONCLUSIONS

The deep learning-based US segmentation can achieve satisfied result to assist robust conventional rigid registration. The Dice similarity coefficient-based metrics, hybrid context, and edge image similarity metric contribute to robust and accurate registration.

High-Volume-Rate 3-D Ultrasound Imaging Using Fast-Tilting and Redirecting Reflectors

Three-dimensional ultrasound imaging has many advantages over 2-D imaging such as more comprehensive tissue evaluation and less operator dependence. However, developing a low-cost and accessible 3-D ultrasound solution with high volume rate and imaging quality remains a challenging task. Recently, we proposed a 3-D ultrasound imaging technique: fast acoustic steering via tilting electromechanical reflectors (FASTER), which uses a fast-tilting acoustic reflector to steer ultrafast plane waves elevationally to achieve high-volume-rate 3-D imaging with conventional 1-D transducers. However, the initial FASTER implementation requires a water tank for acoustic wave conduction and cannot be conveniently used for regular handheld scanning. To address these limitations, here, we developed a novel ultrasound probe clip-on device that encloses a fast-tilting reflector, a redirecting reflector, and an acoustic wave conduction medium. The new FASTER 3-D imaging device can be easily attached to or removed from clinical ultrasound transducers, allowing rapid transformation from 2-D to 3-D imaging. In vitro B-mode studies demonstrated that the proposed method provided comparable imaging quality to conventional, mechanical-translation-based 3-D imaging while offering a much faster volume rate (e.g., 300 versus  ∼  10 Hz). We also demonstrated 3-D power Doppler (PD) and 3-D super-resolution ultrasound localization microscopy (ULM) with the FASTER device. An in vivo imaging study showed that the FASTER device could clearly visualize the 3-D anatomy of the basilic vein. These results suggest that the newly developed redirecting reflector and the clip-on device could overcome key hurdles for future clinical translation of the FASTER 3-D imaging technology.

3D US-CT/MRI registration for percutaneous focal liver tumor ablations

PURPOSE

US-guided percutaneous focal liver tumor ablations have been considered promising curative treatment techniques. To address cases with invisible or poorly visible tumors, registration of 3D US with CT or MRI is a critical step. By taking advantage of deep learning techniques to efficiently detect representative features in both modalities, we aim to develop a 3D US-CT/MRI registration approach for liver tumor ablations.

METHODS

Facilitated by our nnUNet-based 3D US vessel segmentation approach, we propose a coarse-to-fine 3D US-CT/MRI image registration pipeline based on the liver vessel surface and centerlines. Then, phantom, healthy volunteer and patient studies are performed to demonstrate the effectiveness of our proposed registration approach.

RESULTS

Our nnUNet-based vessel segmentation model achieved a Dice score of 0.69. In healthy volunteer study, 11 out of 12 3D US-MRI image pairs were successfully registered with an overall centerline distance of 4.03±2.68 mm. Two patient cases achieved target registration errors (TRE) of 4.16 mm and 5.22 mm.

CONCLUSION

We proposed a coarse-to-fine 3D US-CT/MRI registration pipeline based on nnUNet vessel segmentation models. Experiments based on healthy volunteers and patient trials demonstrated the effectiveness of our registration workflow. Our code and example data are publicly available in this r epository.

High Volume Rate 3-D Ultrasound Imaging Using Fast-Tilting and Redirecting Reflectors

3-D ultrasound imaging has many advantages over 2-D imaging such as more comprehensive tissue evaluation and less operator dependence. Although many 3-D ultrasound imaging techniques have been developed in the last several decades, a low-cost and accessible solution with high imaging volume rate and imaging quality remains elusive. Recently we proposed a new, high volume rate 3-D ultrasound imaging technique: Fast Acoustic Steering via Tilting Electromechanical Reflectors (FASTER), which uses a water-immersible and fast-tilting acoustic reflector to steer ultrafast plane waves in the elevational direction to achieve high volume rate 3-D ultrasound imaging with conventional 1-D array transducers. However, the initial implementation of FASTER imaging only involves a single fast-tilting acoustic reflector, which is inconvenient to use because the probe cannot be held in the regular upright position. Also, conventional FASTER imaging can only be performed inside a water tank because of the necessity of using water for acoustic conduction. To address these limitations of conventional FASTER, here we developed a novel ultrasound probe clip-on device that encloses a fast-tilting reflector, a redirecting reflector, and an acoustic wave conduction medium. The new FASTER 3-D imaging device can be easily attached to or removed from clinical ultrasound transducers, allowing rapid transformation from 2-D to 3-D ultrasound imaging. In vitro B-mode imaging studies demonstrated that the proposed method provided comparable imaging quality (e.g., spatial resolution and contrast-to-noise ratio) to conventional, mechanical-translation-based 3-D imaging while providing a much faster 3-D volume rate (e.g., 300 Hz vs ∼10 Hz). In addition to B-mode imaging, we also demonstrated 3-D power Doppler imaging and 3-D super-resolution ultrasound localization microscopy with the newly developed FASTER device. An in vivo imaging study showed that the FASTER device could clearly visualize the 3-D anatomy of the basilic vein of a healthy volunteer, and customized beamforming was implemented to accommodate the speed of sound difference between the acoustic medium and the imaging object (e.g., soft tissue). These results suggest that the newly developed redirecting reflector and the clip-on device could overcome key hurdles for future clinical translation of the FASTER 3-D imaging technology.

Learning to Localize Cross-Anatomy Landmarks in X-Ray Images with a Universal Model

Objective and Impact Statement. In this work, we develop a universal anatomical landmark detection model which learns once from multiple datasets corresponding to different anatomical regions. Compared with the conventional model trained on a single dataset, this universal model not only is more light weighted and easier to train but also improves the accuracy of the anatomical landmark location. Introduction. The accurate and automatic localization of anatomical landmarks plays an essential role in medical image analysis. However, recent deep learning-based methods only utilize limited data from a single dataset. It is promising and desirable to build a model learned from different regions which harnesses the power of big data. Methods. Our model consists of a local network and a global network, which capture local features and global features, respectively. The local network is a fully convolutional network built up with depth-wise separable convolutions, and the global network uses dilated convolution to enlarge the receptive field to model global dependencies. Results. We evaluate our model on four 2D X-ray image datasets totaling 1710 images and 72 landmarks in four anatomical regions. Extensive experimental results show that our model improves the detection accuracy compared to the state-of-the-art methods. Conclusion. Our model makes the first attempt to train a single network on multiple datasets for landmark detection. Experimental results qualitatively and quantitatively show that our proposed model performs better than other models trained on multiple datasets and even better than models trained on a single dataset separately.

Medical image alignment based on landmark- and approximate contour-matching

Purpose: Our goal is to propose a landmark- and contour-matching (LCM) registration method that uses both landmark information and approximate point correspondences to boost the similarity between image pairs with sparse landmark information. Approach: A model for registering two-dimensional (2D) medical images with landmark information and contour-approximating landmarks was proposed. The model was also extended to accommodate the registration of three-dimensional (3D) cardiac images. We validated the LCM method on 2D hand x-rays and 3D porcine cardiac magnetic resonance images. The following metrics were used to assess the quality of specific aspects of the registered images: Dice similarity coefficient for the overall image overlap, target registration error for pointwise correspondence, and interior angle for local curvature. Results: Target registrations were reduced from 27.12 to 0.01 mm post-LCM registration. Implementing the proposed algorithm also led to a 112% average improvement in image similarity in terms of Dice coefficients. In addition, interior angle measurements indicate that the proposed method preserved the local curvature at major reference landmarks and mitigated the appearance of deformities in the registered images. Conclusions: The proposed method addressed several issues associated with purely landmark-based techniques, such as iterative closest point registration and thin plate spline interpolation. Furthermore, it provided accurate registration results even in the presence of landmark localization errors.

Research on registration and navigation technology of augmented reality for ex-vivo hepatectomy

PURPOSE

The application of augmented reality technology to the partial hepatectomy procedure has high practical significance, but the existing augmented reality navigation system has major drawbacks in the display and registration methods, which result in low precision. The augmented reality surgical navigation system proposed in this study has been improved in the above two aspects, which can significantly improve the surgical accuracy.

METHODS

The use of optical see-through head-mounted displays for imaging displays can prevent doctors from reconstructing the patient's two-dimensional image information in their minds and reduce the psychological burden of doctors. In the registration process, the biomechanical properties of the liver are introduced, and a non-rigid registration method based on biomechanics is proposed and realized by a meshless algorithm. In addition, this study uses the moving grid algorithm to carry out clinical experiments on ex-vivo pig liver for experimental verification.

RESULTS

The mark-based interactive registration error is 4.21 ± 1.6 mm, and the registration error is reduced after taking the biomechanical properties of the liver into account, which is - 0.153 ± 0.398 mm. The cutting error of the liver model is 0.159 ± 0.292 mm. In addition, with the aid of the navigation system proposed in this paper, the experiment of ex-vivo pig liver cutting was completed with an error of - 1.164 ± 0.576 mm.

CONCLUSIONS

As a proof-of-concept study, the augmented reality navigation system proposed in this study improves the traditional image-guided surgery in terms of display and registration methods, and the feasibility of the system is verified by ex-vivo pig liver experiments. Therefore, the navigation system has a certain guiding significance for clinical surgery.

Anatomical Landmark Detection using Deep Appearance-Context Network

Accurate identification of anatomical landmarks is a crucial step in medical image analysis. While deep neural networks have shown impressive performance on computer vision tasks, they rely on a large amount of data, which is often not available. In this work, we propose an attention-driven end-to-end deep learning architecture, which learns the local appearance and global context separately that helps in stable training under limited data. The experiments conducted demonstrate the effectiveness of the proposed approach with impressive results in localizing landmarks when evaluated on cephalometric and spine X-ray image data. The predicted landmarks are further utilized in biomedical applications to demonstrate the impact.

Block-matching-based registration to evaluate ultrasound visibility of percutaneous needles in liver-mimicking phantoms

Purpose

To present a novel methodical approach to compare visibility of percutaneous needles in ultrasound images.

Methods

A motor‐driven rotation platform was used to gradually change the needle angle while capturing image data. Data analysis was automated using block‐matching‐based registration, with a tracking and refinement step. Every 25 frames, a Hough transform was used to improve needle alignments after large rotations. The method was demonstrated by comparing three commercial needles (14G radiofrequency ablation, RFA; 18G Trocar; 22G Chiba) and six prototype needles with different sizes, materials, and surface conditions (polished, sand‐blasted, and kerfed), within polyvinyl alcohol phantom tissue and ex vivo bovine liver models. For each needle and angle, a contrast‐to‐noise ratio (CNR) was determined to quantify visibility. CNR values are presented as a function of needle type and insertion angle. In addition, the normalized area under the (CNR‐angle) curve was used as a summary metric to compare needles.

Results

In phantom tissue, the first kerfed needle design had the largest normalized area of visibility and the polished 1 mm diameter stainless steel needle the smallest (0.704 ± 0.199 vs. 0.154 ± 0.027, p < 0.01). In the ex vivo model, the second kerfed needle design had the largest normalized area of visibility, and the sand‐blasted stainless steel needle the smallest (0.470 ± 0.190 vs. 0.127 ± 0.047, p < 0.001). As expected, the analysis showed needle visibility peaks at orthogonal insertion angles. For acute or obtuse angles, needle visibility was similar or reduced. Overall, the variability in needle visibility was considerably higher in livers.

Conclusion

The best overall visibility was found with kerfed needles and the commercial RFA needle. The presented methodical approach to quantify ultrasound visibility allows comparisons of (echogenic) needles, as well as other technological innovations aiming to improve ultrasound visibility of percutaneous needles, such as coatings, material treatments, and beam steering approaches.

Automated brain structures segmentation from PET/CT images based on landmark-constrained dual-modality atlas registration

Automated brain structures segmentation in positron emission tomography (PET) images has been widely investigated to help brain disease diagnosis and follow-up. To relieve the burden of a manual definition of volume of interest (VOI), automated atlas-based VOI definition algorithms were developed, but these algorithms mostly adopted a global optimization strategy which may not be particularly accurate for local small structures (especially the deep brain structures). This paper presents a PET/CT-based brain VOI segmentation algorithm combining anatomical atlas, local landmarks, and dual-modality information. The method incorporates local deep brain landmarks detected by the Deep Q-Network (DQN) to constrain the atlas registration process. Dual-modality PET/CT image information is also combined to improve the registration accuracy of the extracerebral contour. We compare our algorithm with the representative brain atlas registration methods based on 86 clinical PET/CT images. The proposed algorithm obtained accurate delineation of brain VOIs with an average Dice similarity score of 0.79, an average surface distance of 0.97 mm (sub-pixel level), and a volume recovery coefficient close to 1. The main advantage of our method is that it optimizes both global-scale brain matching and local-scale small structure alignment around the key landmarks, it is fully automated and produces high-quality parcellation of the brain structures from brain PET/CT images.

Registration of Untracked 2D Laparoscopic Ultrasound to CT Images of the Liver Using Multi-Labelled Content-Based Image Retrieval

Laparoscopic Ultrasound (LUS) is recommended as a standard-of-care when performing laparoscopic liver resections as it images sub-surface structures such as tumours and major vessels. Given that LUS probes are difficult to handle and some tumours are iso-echoic, registration of LUS images to a pre-operative CT has been proposed as an image-guidance method. This registration problem is particularly challenging due to the small field of view of LUS, and usually depends on both a manual initialisation and tracking to compose a volume, hindering clinical translation. In this paper, we extend a proposed registration approach using Content-Based Image Retrieval (CBIR), removing the requirement for tracking or manual initialisation. Pre-operatively, a set of possible LUS planes is simulated from CT and a descriptor generated for each image. Then, a Bayesian framework is employed to estimate the most likely sequence of CT simulations that matches a series of LUS images. We extend our CBIR formulation to use multiple labelled objects and constrain the registration by separating liver vessels into portal vein and hepatic vein branches. The value of this new labeled approach is demonstrated in retrospective data from 5 patients. Results show that, by including a series of 5 untracked images in time, a single LUS image can be registered with accuracies ranging from 5.7 to 16.4 mm with a success rate of 78%. Initialisation of the LUS to CT registration with the proposed framework could potentially enable the clinical translation of these image fusion techniques.

Intraoperative Correction of Liver Deformation Using Sparse Surface and Vascular Features via Linearized Iterative Boundary Reconstruction

During image guided liver surgery, soft tissue deformation can cause considerable error when attempting to achieve accurate localization of the surgical anatomy through image-to-physical registration. In this paper, a linearized iterative boundary reconstruction technique is proposed to account for these deformations. The approach leverages a superposed formulation of boundary conditions to rapidly and accurately estimate the deformation applied to a preoperative model of the organ given sparse intraoperative data of surface and subsurface features. With this method, tracked intraoperative ultrasound (iUS) is investigated as a potential data source for augmenting registration accuracy beyond the capacity of conventional organ surface registration. In an expansive simulated dataset, features including vessel contours, vessel centerlines, and the posterior liver surface are extracted from iUS planes. Registration accuracy is compared across increasing data density to establish how iUS can be best employed to improve target registration error (TRE). From a baseline average TRE of 11.4 ± 2.2 mm using sparse surface data only, incorporating additional sparse features from three iUS planes improved average TRE to 6.4 ± 1.0 mm. Furthermore, increasing the sparse coverage to 16 tracked iUS planes improved average TRE to 3.9 ± 0.7 mm, exceeding the accuracy of registration based on complete surface data available with more cumbersome intraoperative CT without contrast. Additionally, the approach was applied to three clinical cases where on average error improved 67% over rigid registration and 56% over deformable surface registration when incorporating additional features from one independent tracked iUS plane.

Tissue Structure Updating for In Situ Augmented Reality Navigation Using Calibrated Ultrasound and Two-Level Surface Warping

OBJECTIVE

In minimally invasive surgery (MIS), in situ augmented reality (AR) navigation systems are usually implemented using a glasses-free 3D display to represent the preoperative tissue structure, and can provide intuitive see-through guidance information. However, due to changes in intraoperative tissue, the preoperative tissue structure is not able to exactly correspond to reality, which influences the precision of in situ AR navigation. To solve this problem, we propose a method to update the tissue structure for in situ AR navigation in such way to reflect changes in intraoperative tissue.

METHODS

The proposed method to update the tissue structure is based on the calibrated ultrasound and two-level surface warping technologies. Firstly, the particle filter-based calibration is implemented to perform ultrasound calibration and obtain intraoperative position of anatomical points. Secondly, intraoperative positions of anatomical points are inputted in the two-level surface warping method to update the preoperative tissue structure. Finally, the glasses-free real 3-D display of the updated tissue structure is finished, and is superimposed onto a patient by a translucent mirror for in situ AR navigation.

RESULTS

we validated the proposed method by simulating liver tissue intervention, and achieved the tissue updating accuracy of 92.86%. Furthermore, the targeting error of AR navigation based on the proposed method was also evaluated through minimally invasive liver surgery, and the acquired mean targeting error was 1.92 mm.

CONCLUSION

The results demonstrate that the proposed AR navigation method is effective.

SIGNIFICANCE

The proposed method can facilitate MIS, as it provides accurate 3D navigation.

Weakly Supervised Estimation of Shadow Confidence Maps in Fetal Ultrasound Imaging

Detecting acoustic shadows in ultrasound images is important in many clinical and engineering applications. Real-time feedback of acoustic shadows can guide sonographers to a standardized diagnostic viewing plane with minimal artifacts and can provide additional information for other automatic image analysis algorithms. However, automatically detecting shadow regions using learning-based algorithms is challenging because pixel-wise ground truth annotation of acoustic shadows is subjective and time consuming. In this paper, we propose a weakly supervised method for automatic confidence estimation of acoustic shadow regions. Our method is able to generate a dense shadow-focused confidence map. In our method, a shadow-seg module is built to learn general shadow features for shadow segmentation, based on global image-level annotations as well as a small number of coarse pixel-wise shadow annotations. A transfer function is introduced to extend the obtained binary shadow segmentation to a reference confidence map. In addition, a confidence estimation network is proposed to learn the mapping between input images and the reference confidence maps. This network is able to predict shadow confidence maps directly from input images during inference. We use evaluation metrics such as DICE, inter-class correlation, and so on, to verify the effectiveness of our method. Our method is more consistent than human annotation and outperforms the state-of-the-art quantitatively in shadow segmentation and qualitatively in confidence estimation of shadow regions. Furthermore, we demonstrate the applicability of our method by integrating shadow confidence maps into tasks such as ultrasound image classification, multi-view image fusion, and automated biometric measurements.

Two-dimensional ultrasound-computed tomography image registration for monitoring percutaneous hepatic intervention

PURPOSE

Deformable registration of ultrasound (US) and contrast enhanced computed tomography (CECT) images are essential for quantitative comparison of ablation boundaries and dimensions determined using these modalities. This comparison is essential as stiffness-based imaging using US has become popular and offers a nonionizing and cost-effective imaging modality for monitoring minimally invasive microwave ablation procedures. A sensible manual registration method is presented that performs the required CT-US image registration.

METHODS

The two-dimensional (2D) virtual CT image plane that corresponds to the clinical US B-mode was obtained by "virtually slicing" the 3D CT volume along the plane containing non-anatomical landmarks, namely points along the microwave ablation antenna. The initial slice plane was generated using the vector acquired by rotating the normal vector of the transverse (i.e., xz) plane along the angle subtended by the antenna. This plane was then further rotated along the ablation antenna and shifted along with the direction of normal vector to obtain similar anatomical structures, such as the liver surface and vasculature that is visualized on both the CT virtual slice and US B-mode images on 20 patients. Finally, an affine transformation was estimated using anatomic and non-anatomic landmarks to account for distortion between the colocated CT virtual slice and US B-mode image resulting in a final registered CT virtual slice. Registration accuracy was measured by estimating the Euclidean distance between corresponding registered points on CT and US B-mode images.

RESULTS

Mean and SD of the affine transformed registration error was 1.85 ± 2.14 (mm), computed from 20 coregistered data sets.

CONCLUSIONS

Our results demonstrate the ability to obtain 2D virtual CT slices that are registered to clinical US B-mode images. The use of both anatomical and non-anatomical landmarks result in accurate registration useful for validating ablative margins and comparison to electrode displacement elastography based images.

Multiple-correlation similarity for block-matching based fast CT to ultrasound registration in liver interventions

In this work we present a fast approach to perform registration of computed tomography to ultrasound volumes for image guided intervention applications. The method is based on a combination of block-matching and outlier rejection. The block-matching uses a correlation based multimodal similarity metric, where the intensity and the gradient of the computed tomography images along with the ultrasound volumes are the input images to find correspondences between blocks in the computed tomography and the ultrasound volumes. A variance and octree based feature point-set selection method is used for selecting distinct and evenly spread point locations for block-matching. Geometric consistency and smoothness criteria are imposed in an outlier rejection step to refine the block-matching results. The block-matching results after outlier rejection are used to determine the affine transformation between the computed tomography and the ultrasound volumes. Various experiments are carried out to assess the optimal performance and the influence of parameters on accuracy and computational time of the registration. A leave-one-patient-out cross-validation registration error of 3.6 mm is achieved over 29 datasets, acquired from 17 patients.

A pre-operative planning framework for global registration of laparoscopic ultrasound to CT images

PURPOSE

Laparoscopic ultrasound (LUS) enhances the safety of laparoscopic liver resection by enabling real-time imaging of internal structures such as vessels. However, LUS probes can be difficult to use, and many tumours are iso-echoic and hence are not visible. Registration of LUS to a pre-operative CT or MR scan has been proposed as a method of image guidance. However, the field of view of the probe is very small compared to the whole liver, making the registration task challenging and dependent on a very accurate initialisation.

METHODS

We propose the use of a subject-specific planning framework that provides information on which anatomical liver regions it is possible to acquire vascular data that is unique enough for a globally optimal initial registration. Vessel-based rigid registration on different areas of the pre-operative CT vascular tree is used in order to evaluate predicted accuracy and reliability.

RESULTS

The planning framework is tested on one porcine subject where we have taken 5 independent sweeps of LUS data from different sections of the liver. Target registration error of vessel branching points was used to measure accuracy. Global registration based on vessel centrelines is applied to the 5 datasets. In 3 out of 5 cases registration is successful and in agreement with the planning. Further tests with a CT scan under abdominal insufflation show that the framework can provide valuable information in all of the 5 cases.

CONCLUSIONS

We have introduced a planning framework that can guide the surgeon on how much LUS data to collect in order to provide a reliable globally unique registration without the need for an initial manual alignment. This could potentially improve the usability of these methods in clinic.

Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods

BACKGROUND

Augmented reality (AR) technology is used to reconstruct three-dimensional (3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.

DATA SOURCES

The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the PubMed database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.

RESULTS

In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery, which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.

CONCLUSIONS

With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling, and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.

Potential of metric homotopy between intensity and geometry information for multi-modal 3D registration

This paper focuses on a novel strategy increasing robustness with respect to local optima when using Mutual Information (MI) in multi-modal image registration. This is realized by integrating additional geometry information in the cost function. Hereby, the main innovation is a generalization of multi-metric registration approaches by means of a metric homotopy. Particularly we realize a method which automatically determines the parameters of the metric homotopy. To construct the cost function independent of the choice of the optimizer, the weighting is defined as a function of one of the metrics instead of optimizer steps. In addition, a differentiable cost function is developed. In comparison to the commonly used technique to process an intensity based registration on different resolutions, the proposed method is three times faster with unchanged accuracy. It is also shown that in the presence of large landmark errors the proposed method outperforms an approach in accuracy in which both similarity functionals are applied one after the other. The method is evaluated on 3D multi-modal human brain data sets from the Retrospective Image Registration Evaluation Project (RIRE). The evaluation is performed using the evaluation website of the RIRE project to make the registration results of the proposed method easily comparable to other methods. Therefore, the presented results are also available online on the RIRE project page.

A new device for fiducial registration of image-guided navigation system for liver RFA

PURPOSE

Radiofrequency ablation for liver tumors (liver RFA) is widely performed under ultrasound guidance. However, discriminating between the tumor and the needle is often difficult because of cavitation caused by RFA-induced coagulation. An unclear ultrasound image can lead to complications and tumor residue. Therefore, image-guided navigation systems based on fiducial registration have been developed. Fiducial points are usually set on a patient's skin. But the use of internal fiducial points can improve the accuracy of navigation. In this study, a new device is introduced to use internal fiducial points using 2D US.

METHODS

3D Slicer as the navigation software, Polaris Vicra as the position sensor, and two target tumors in a 3D abdominal phantom as puncture targets were used. Also, a new device that makes it possible to obtain tracking coordinates in the body was invented. First, two-dimensional reslice images from the CT images using 3D Slicer were built. A virtual needle was displayed on the two-dimensional reslice image, reflecting the movement of the actual needle after fiducial registration. A phantom experiment using three sets of fiducial point configurations: one conventional case using only surface points, and two cases in which the center of the target tumor was selected as a fiducial point was performed. For each configuration, one surgeon punctured each target tumor ten times under guidance from the 3D Slicer display. Finally, a statistical analysis examining the puncture error was performed.

RESULTS

The puncture error for each target tumor decreased significantly when the center of the target tumor was included as one of the fiducial points, compared with when only surface points were used.

CONCLUSION

This study introduces a new device to use internal fiducial points and suggests that the accuracy of image-guided navigation systems for liver RFA can be improved by using the new device.

Evaluation of model-based deformation correction in image-guided liver surgery via tracked intraoperative ultrasound

Soft-tissue deformation represents a significant error source in current surgical navigation systems used for open hepatic procedures. While numerous algorithms have been proposed to rectify the tissue deformation that is encountered during open liver surgery, clinical validation of the proposed methods has been limited to surface-based metrics, and subsurface validation has largely been performed via phantom experiments. The proposed method involves the analysis of two deformation-correction algorithms for open hepatic image-guided surgery systems via subsurface targets digitized with tracked intraoperative ultrasound (iUS). Intraoperative surface digitizations were acquired via a laser range scanner and an optically tracked stylus for the purposes of computing the physical-to-image space registration and for use in retrospective deformation-correction algorithms. Upon completion of surface digitization, the organ was interrogated with a tracked iUS transducer where the iUS images and corresponding tracked locations were recorded. Mean closest-point distances between the feature contours delineated in the iUS images and corresponding three-dimensional anatomical model generated from preoperative tomograms were computed to quantify the extent to which the deformation-correction algorithms improved registration accuracy. The results for six patients, including eight anatomical targets, indicate that deformation correction can facilitate reduction in target error of [Formula: see text].

4D Ultrasound Tracking of Liver and its Verification for TIPS Guidance

In this work we describe a 4D registration method for on the fly stabilization of ultrasound volumes for improving image guidance for transjugular intrahepatic portosystemic shunt (TIPS) interventions. The purpose of the method is to enable a continuous visualization of the relevant anatomical planes (determined in a planning stage) in a free breathing patient during the intervention. This requires registration of the planning information to the interventional images, which is achieved in two steps. In the first step tracking is performed across the streaming input. An approximate transformation between the reference image and the incoming image is estimated by composing the intermediate transformations obtained from the tracking. In the second step a subsequent registration is performed between the reference image and the approximately transformed incoming image to account for the accumulation of error. The two step approach helps in reducing the search range and is robust under rotation. We additionally present an approach to initialize and verify the registration. Verification is required when the reference image (containing planning information) is acquired in the past and is not part of the (interventional) 4D ultrasound sequence. The verification score will help in invalidating the registration outcome, for instance, in the case of insufficient overlap or information between the registering images due to probe motion or loss of contact, respectively. We evaluate the method over thirteen 4D US sequences acquired from eight subjects. A graphics processing unit implementation runs the 4D tracking at 9 Hz with a mean registration error of 1.7 mm.

Locally rigid, vessel-based registration for laparoscopic liver surgery

PURPOSE

Laparoscopic liver resection has significant advantages over open surgery due to less patient trauma and faster recovery times, yet is difficult for most lesions due to the restricted field of view and lack of haptic feedback. Image guidance provides a potential solution but is challenging in a soft deforming organ such as the liver. In this paper, we therefore propose a laparoscopic ultrasound (LUS) image guidance system and study the feasibility of a locally rigid registration for laparoscopic liver surgery.

METHODS

We developed a real-time segmentation method to extract vessel centre points from calibrated, freehand, electromagnetically tracked, 2D LUS images. Using landmark-based initial registration and an optional iterative closest point (ICP) point-to-line registration, a vessel centre-line model extracted from preoperative computed tomography (CT) is registered to the ultrasound data during surgery.

RESULTS

Using the locally rigid ICP method, the RMS residual error when registering to a phantom was 0.7 mm, and the mean target registration error (TRE) for two in vivo porcine studies was 3.58 and 2.99 mm, respectively. Using the locally rigid landmark-based registration method gave a mean TRE of 4.23 mm using vessel centre lines derived from CT scans taken with pneumoperitoneum and 6.57 mm without pneumoperitoneum.

CONCLUSION

In this paper we propose a practical image-guided surgery system based on locally rigid registration of a CT-derived model to vascular structures located with LUS. In a physical phantom and during porcine laparoscopic liver resection, we demonstrate accuracy of target location commensurate with surgical requirements. We conclude that locally rigid registration could be sufficient for practically useful image guidance in the near future.

Toward a generic real-time compression correction framework for tracked ultrasound

PURPOSE

Tissue compression during ultrasound imaging leads to error in the location and geometry of subsurface targets during soft tissue interventions. We present a novel compression correction method, which models a generic block of tissue and its subsurface tissue displacements resulting from application of a probe to the tissue surface. The advantages of the new method are that it can be realized independent of preoperative imaging data and is capable of near-video framerate compression compensation for real-time guidance.

METHODS

The block model is calibrated to the tip of any tracked ultrasound probe. Intraoperative digitization of the tissue surface is used to measure the depth of compression and provide boundary conditions to the biomechanical model of the tissue. The tissue displacement field solution of the model is inverted to nonrigidly transform the ultrasound images to an estimation of the tissue geometry prior to compression. This method was compared to a previously developed method using a patient-specific model and within the context of simulation, phantom, and clinical data.

RESULTS

Experimental results with gel phantoms demonstrated that the proposed generic method reduced the mock tumor margin modified Hausdorff distance (MHD) from 5.0 ± 1.6 to 2.1 ± 0.7 mm and reduced mock tumor centroid alignment error from 7.6 ± 2.6 to 2.6 ± 1.1mm. The method was applied to a clinical case and reduced the in vivo tumor margin MHD error from 5.4 ± 0.1 to 2.9 ± 0.1mm, and the centroid alignment error from 7.2 ± 0.2 to 3.8 ± 0.4 mm.

CONCLUSIONS

The correction method was found to effectively improve alignment of ultrasound and tomographic images and was more efficient compared to a previously proposed correction.

Skeleton Graph Matching vs. Maximum Weight Cliques aorta registration techniques

Vascular diseases are one of the most challenging health problems in developed countries. Past as well as ongoing research activities often focus on efficient, robust and fast aorta segmentation, and registration techniques. According to this needs our study targets an abdominal aorta registration method. The investigated algorithms make it possible to efficiently segment and register abdominal aorta in pre- and post-operative Computed Tomography (CT) data. In more detail, a registration technique using the Path Similarity Skeleton Graph Matching (PSSGM), as well as Maximum Weight Cliques (MWCs) are employed to realise the matching based on Computed Tomography data. The presented approaches make it possible to match characteristic voxels belonging to the aorta from different Computed Tomography (CT) series. It is particularly useful in the assessment of the abdominal aortic aneurysm treatment by visualising the correspondence between the pre- and post-operative CT data. The registration results have been tested on the database of 18 contrast-enhanced CT series, where the cross-registration analysis has been performed producing 153 matching examples. All the registration results achieved with our system have been verified by an expert. The carried out analysis has highlighted the advantage of the MWCs technique over the PSSGM method. The verification phase proves the efficiency of the MWCs approach and encourages to further develop this methods.

Quantitative evaluation of noise reduction and vesselness filters for liver vessel segmentation on abdominal CTA images

Liver vessel segmentation in CTA images is a challenging task, especially in the case of noisy images. This paper investigates whether pre-filtering improves liver vessel segmentation in 3D CTA images. We introduce a quantitative evaluation of several well-known filters based on a proposed liver vessel segmentation method on CTA images. We compare the effect of different diffusion techniques i.e. Regularized Perona-Malik, Hybrid Diffusion with Continuous Switch and Vessel Enhancing Diffusion as well as the vesselness approaches proposed by Sato, Frangi and Erdt. Liver vessel segmentation of the pre-processed images is performed using a histogram-based region grown with local maxima as seed points. Quantitative measurements (sensitivity, specificity and accuracy) are determined based on manual landmarks inside and outside the vessels, followed by T-tests for statistic comparisons on 51 clinical CTA images. The evaluation demonstrates that all the filters make liver vessel segmentation have a significantly higher accuracy than without using a filter (p  <  0.05); Hybrid Diffusion with Continuous Switch achieves the best performance. Compared to the diffusion filters, vesselness filters have a greater sensitivity but less specificity. In addition, the proposed liver vessel segmentation method with pre-filtering is shown to perform robustly on a clinical dataset having a low contrast-to-noise of up to 3 (dB). The results indicate that the pre-filtering step significantly improves liver vessel segmentation on 3D CTA images.

Systematic review of the use of pre-operative simulation and navigation for hepatectomy: current status and future perspectives

Pre-operative simulation using three-dimensional (3D) reconstructions have been suggested to enhance surgical planning of hepatectomy. Evidence on its benefits for hepatectomy patients remains limited. This systematic review examined the use and impact of pre-operative simulation and intraoperative navigation on hepatectomy outcomes. A systematical searched electronic databases for studies reporting on the use and results of simulation and navigation for hepatectomy was performed. The primary outcome was change in operative plan based on simulation. Secondary outcomes included operating time (min), estimated blood loss, surgical margins, 30-day postoperative morbidity and mortality, and study-specific outcomes. From 222 citations, we included 11 studies including 497 patients. All were observational cohort studies. No study compared hepatectomy with and without simulation. All studies performed 3D reconstruction and segmentation, most commonly with volumetrics measurements. In six studies reporting intraoperative navigation, five relied on ultrasound, and one on a resection map. Of two studies reporting on it, the resection line was changed intraoperatively in one third of patients, based on simulation. Virtually predicted liver volumes (Pearson correlation r = 0.917 to 0.995) and surgical margins (r = 0.84 to 0.967) correlated highly with actual ones in eight studies. Heterogeneity of the included studies precluded meta-analysis. Pre-operative simulation seems accurate in measuring volumetrics and surgical margins. Current studies lack intraoperative transposition of simulation for direct navigation. Simulation appears useful planning of hepatectomies, but further work is warranted focusing on the development of improved tools and appraisal of their clinical impact compared to traditional resection.

A Mechanics-Based Nonrigid Registration Method for Liver Surgery Using Sparse Intraoperative Data

In open abdominal image-guided liver surgery, sparse measurements of the organ surface can be taken intraoperatively via a laser-range scanning device or a tracked stylus with relatively little impact on surgical workflow. We propose a novel nonrigid registration method which uses sparse surface data to reconstruct a mapping between the preoperative CT volume and the intraoperative patient space. The mapping is generated using a tissue mechanics model subject to boundary conditions consistent with surgical supportive packing during liver resection therapy. Our approach iteratively chooses parameters which define these boundary conditions such that the deformed tissue model best fits the intraoperative surface data. Using two liver phantoms, we gathered a total of five deformation datasets with conditions comparable to open surgery. The proposed nonrigid method achieved a mean target registration error (TRE) of 3.3 mm for targets dispersed throughout the phantom volume, using a limited region of surface data to drive the nonrigid registration algorithm, while rigid registration resulted in a mean TRE of 9.5 mm. In addition, we studied the effect of surface data extent, the inclusion of subsurface data, the trade-offs of using a nonlinear tissue model, robustness to rigid misalignments, and the feasibility in five clinical datasets.

Hybrid feature-based diffeomorphic registration for tumor tracking in 2-D liver ultrasound images

Real-time ultrasound image acquisition is a pivotal resource in the medical community, in spite of its limited image quality. This poses challenges to image registration methods, particularly to those driven by intensity values. We address these difficulties in a novel diffeomorphic registration technique for tumor tracking in series of 2-D liver ultrasound. Our method has two main characteristics: 1) each voxel is described by three image features: intensity, local phase, and phase congruency; 2) we compute a set of forces from either local information (Demons-type of forces), or spatial correspondences supplied by a block-matching scheme, from each image feature. A family of update deformation fields which are defined by these forces, and inform upon the local or regional contribution of each image feature are then composed to form the final transformation. The method is diffeomorphic, which ensures the invertibility of deformations. The qualitative and quantitative results yielded by both synthetic and real clinical data show the suitability of our method for the application at hand.

Two-stage point-based registration method between ultrasound and CT imaging of the liver based on ICP and unscented Kalman filter: a phantom study

PURPOSE

In recent years, image-guided liver surgery based on intraoperative ultrasound (US) imaging has become common. Using an efficient point-based registration method to improve both accuracy and computational time for the registration of predeformation computer tomography, liver images with postdeformation US images are important during surgical procedure. Although iterative closest point (ICP) algorithm is widely used in surface-based registration, its performance strongly depends on the presence of noise and initial alignment. A registration technique based on unscented Kalman filter (UKF), which has been proposed recently, can used to overcome the noise and outliers on an incremental basis; however, the technique is associated with computational complexity.

METHODS

To overcome the limitations of ICP and UKF algorithms, we proposed an incremental two-stage registration method based on the combination of ICP and UKF algorithms to update the registration process with the acquired new points from US images. The registration is based on both the vessels and surface information of the liver.

RESULTS

The two-stage method was examined using numerical simulations and phantom data sets. The results of the phantom data set confirmed that the two-stage method outperforms the accuracy of ICP by 23% and reduces the running time of UKF by 60%.

CONCLUSION

The convergence rate, computational speed, and accuracy of the UKF algorithm can be improved using the two-stage method.

Vessel bifurcation localization based on intraoperative three-dimensional ultrasound and catheter path for image-guided catheter intervention of oral cancers

We present a method to localize intraoperative target vessel bifurcations under bones for ultrasound (US) image-guided catheter interventions. A catheter path is recorded to acquire skeletons for the target vessel bifurcations that cannot be imaged by intraoperative US. The catheter path is combined with the centerlines of the three-dimensional (3D) US image to construct a preliminary skeleton. Based on the preliminary skeleton, the orientations of target vessels are determined by registration with the preoperative image and the bifurcations were localized by computing the vessel length. An accurate intraoperative vessel skeleton is obtained for correcting the preoperative image to compensate for vessel deformation. A reality check of the proposed method was performed in a phantom experiment. Reasonable results were obtained. The in vivo experiment verified the clinical workflow of the proposed method in an in vivo environment. The accuracy of the centerline length of the vessel for localizing the target artery bifurcation was 2.4mm. These results suggest that the proposed method can allow the catheter tip to stop at the target artery bifurcations and enter into the target arteries. This method can be applied for virtual reality-enhanced image-guided catheter intervention of oral cancers.

Risk maps for liver surgery

PURPOSE

Optimal display of surgical planning data in the operating room is challenging. In liver surgery, an expressive and effective intraoperative visualization of 3D planning models is still a pressing need. The objective of this work is to visualize surgical planning information using a map display.

METHODS

An approach for risk analysis and visualization of planning models is presented which provides relevant information at a glance without the need for user interaction. Therefore, we present methods for the identification and classification of critical anatomical structures in the proximity of a preoperatively planned resection surface. Shadow-like distance indicators are introduced to encode the distance from the resection surface to these critical structures on a risk map. The work is demonstrated with examples in liver resection surgery and evaluated within two user studies.

RESULTS

The results of the performed user studies show that the proposed visualization techniques facilitate the process of risk assessment in liver resection surgery and might be a valuable extension to surgical navigations system.

CONCLUSION

The approach provides a new and objective basis for the assessment of risks during liver surgery and has the potential to improve the outcome of surgical interventions.

Image-Guided Abdominal Surgery and Therapy Delivery

Image-Guided Surgery has become the standard of care in intracranial neurosurgery providing more exact resections while minimizing damage to healthy tissue. Moving that process to abdominal organs presents additional challenges in the form of image segmentation, image to physical space registration, organ motion and deformation. In this paper, we present methodologies and results for addressing these challenges in two specific organs: the liver and the kidney.