A pre-operative planning framework for global registration of laparoscopic ultrasound to CT images

Intelligent automatic registration: is it feasible and efficient for application of ultrasound fusion imaging in liver?

PURPOSE

To evaluate the feasibility and efficiency between the two intelligent auto-registrations (based on hepatic vessels or based on liver surface) and manual registration for US-CT/MR fusion imaging of liver tumours.

METHODS

From May 2017 to December 2017, 30 patients with 30 liver tumours were enrolled in this prospectively study. Two intelligent auto-registrations (based on hepatic vessels or based on liver surface) and manual registration were randomly performed, the registration success rate and efficiency were compared.

RESULTS

In terms of success rate, auto-registrations based on the hepatic vessels (80%) was lower than auto-registration base on liver surface and manual registration (96.67%), but with no statistical difference (P = 0.125). In comparison of the registration efficiency, the efficiency of the auto-registration based on the hepatic vessels was superior to auto-registration based on liver surface and manual registration (P < 0.05). The one-step success rate of auto-registration based on the hepatic vessels (53.33%, 16/30) was higher than that of other two registrations (P < 0.05). Stratified analysis showed that, for the lesion with display of hepatic vessels in grade 3, the success rate of auto-registration based on vessels (0%) was lower than that of auto-registration based on liver surface and manual registration (100%) (P = 0.031).

CONCLUSION

Intelligent auto-registration based on hepatic vessels is a feasible and efficient registration method for US-CT/MR fusion imaging of liver tumours for the patients with clear hepatic vessels. The auto-registration based on liver surface and manual registration can be an effective supplement for cases with poor hepatic vessels display.

Guided ultrasound acquisition for nonrigid image registration using reinforcement learning

We propose a guided registration method for spatially aligning a fixed preoperative image and untracked ultrasound image slices. We exploit the unique interactive and spatially heterogeneous nature of this application to develop a registration algorithm that interactively suggests and acquires ultrasound images at optimised locations (with respect to registration performance). Our framework is based on two trainable functions: (1) a deep hyper-network-based registration function, which is generalisable over varying location and deformation, and adaptable at test-time; (2) a reinforcement learning function for producing test-time estimates of image acquisition locations and adapted deformation regularisation (the latter is required due to varying acquisition locations). We evaluate our proposed method with real preoperative patient data, and simulated intraoperative data with variable field-of-view. In addition to simulation of intraoperative data, we simulate global alignment based on previous work for efficient training, and investigate probe-level guidance towards an improved deformable registration. The evaluation in a simulated environment shows statistically significant improvements in overall registration performance across a variety of metrics for our proposed method, compared to registration without acquisition guidance or adaptable deformation regularisation, and to commonly used classical iterative methods and learning-based registration. For the first time, efficacy of proactive image acquisition is demonstrated in a simulated surgical interventional registration, in contrast to most existing work addressing registration post-data-acquisition, one of the reasons we argue may have led to previously under-constrained nonrigid registration in such applications. Code: https://github.com/s-sd/rl_guided_registration.

From Coarse to Fine: Non-Rigid Sparse-Dense Registration for Deformation-Aware Liver Surgical Navigation

OBJECTIVE

Intraoperative liver deformation poses a considerable challenge during liver surgery, causing significant errors in image-guided surgical navigation systems. This study addresses a critical non-rigid registration problem in liver surgery: the alignment of intrahepatic vascular trees. The goal is to deform the complete vascular shape extracted from preoperative Computed Tomography (CT) volume, aligning it with sparse vascular contour points obtained from intraoperative ultrasound (iUS) images. Challenges arise due to the intricate nature of slender vascular branches, causing existing methods to struggle with accuracy and vascular self-intersection.

METHODS

We present a novel non-rigid sparse-dense registration pipeline structured in a coarse-to-fine fashion. In the initial coarse registration stage, we introduce a parametrization deformation graph and a Welsch function-based error metric to enhance convergence and robustness of non-rigid registration. For the fine registration stage, we propose an automatic curvature-based algorithm to detect and eliminate overlapping regions. Subsequently, we generate the complete vascular shape using posterior computation of a Gaussian Process Shape Model.

RESULTS

Experimental results using simulated data demonstrate the accuracy and robustness of our proposed method. Evaluation results on the target registration error of tumors highlight the clinical significance of our method in tumor location computation. Comparative analysis against related methods reveals superior accuracy and competitive efficiency of our approach. Moreover, Ex vivo swine liver experiments and clinical experiments were conducted to evaluate the method's performance.

CONCLUSION

The experimental results emphasize the accurate and robust performance of our proposed method.

SIGNIFICANCE

Our proposed non-rigid registration method holds significant application potential in clinical practice.

Deep hashing for global registration of untracked 2D laparoscopic ultrasound to CT

PURPOSE

The registration of Laparoscopic Ultrasound (LUS) to CT can enhance the safety of laparoscopic liver surgery by providing the surgeon with awareness on the relative positioning between critical vessels and a tumour. In an effort to provide a translatable solution for this poorly constrained problem, Content-based Image Retrieval (CBIR) based on vessel information has been suggested as a method for obtaining a global coarse registration without using tracking information. However, the performance of these frameworks is limited by the use of non-generalisable handcrafted vessel features.

METHODS

We propose the use of a Deep Hashing (DH) network to directly convert vessel images from both LUS and CT into fixed size hash codes. During training, these codes are learnt from a patient-specific CT scan by supplying the network with triplets of vessel images which include both a registered and a mis-registered pair. Once hash codes have been learnt, they can be used to perform registration with CBIR methods.

RESULTS

We test a CBIR pipeline on 11 sequences of untracked LUS distributed across 5 clinical cases. Compared to a handcrafted feature approach, our model improves the registration success rate significantly from 48% to 61%, considering a 20 mm error as the threshold for a successful coarse registration.

CONCLUSIONS

We present the first DH framework for interventional multi-modal registration tasks. The presented approach is easily generalisable to other registration problems, does not require annotated data for training, and may promote the translation of these techniques.

Performance of image guided navigation in laparoscopic liver surgery - A systematic review

BACKGROUND

Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations.

METHODS

Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format.

RESULTS

Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8-15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes.

CONCLUSIONS

Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard.

The overview of the deep learning integrated into the medical imaging of liver: a review

Deep learning (DL) is a recently developed artificial intelligent method that can be integrated into numerous fields. For the imaging diagnosis of liver disease, several remarkable outcomes have been achieved with the application of DL currently. This advanced algorithm takes part in various sections of imaging processing such as liver segmentation, lesion delineation, disease classification, process optimization, etc. The DL optimized imaging diagnosis shows a broad prospect instead of the pathological biopsy for the advantages of convenience, safety, and inexpensiveness. In this paper, we reviewed the published representative DL-related hepatic imaging works, described the general situation of this new-rising technology in medical liver imaging and explored the future direction of DL development.

[Advantages and future perspectives of laparoscopic liver surgery]

BACKGROUND

Laparoscopic liver surgery (LLS) is increasingly utilized worldwide due to several potential advantages over open liver surgery.

OBJECTIVE

Analysis and presentation of the advantages and disadvantages of LLS in comparison to open surgery.

MATERIAL AND METHODS

Analysis of clinically relevant factors of minimally invasive liver surgery in comparison to open liver surgery in the current literature.

RESULTS

In addition to obvious cosmetic advantages, the current literature shows advantages regarding length of hospital stay and quality of life after LLC. In contrast to major liver resections, parenchyma-preserving resections often appear cost-neutral due a shorter postoperative hospital stay compared to conventional liver resections. In addition to particular personnel requirements, LLS also has technical prerequisites, such as a dedicated intraoperative ultrasound system. Furthermore, contrast-enhanced laparoscopic examinations are possible and ultrasound information can be fused with preoperative imaging. Virtual reality technology and 3‑dimensional printing are currently under investigation to improve the intraoperative anatomical orientation of LLS.

CONCLUSION

The current literature reveals significant advantages for LLS so that this procedure should be further developed in Germany in order to offer this technique to as many patients as possible.

Registration of Untracked 2D Laparoscopic Ultrasound to CT Images of the Liver Using Multi-Labelled Content-Based Image Retrieval

Laparoscopic Ultrasound (LUS) is recommended as a standard-of-care when performing laparoscopic liver resections as it images sub-surface structures such as tumours and major vessels. Given that LUS probes are difficult to handle and some tumours are iso-echoic, registration of LUS images to a pre-operative CT has been proposed as an image-guidance method. This registration problem is particularly challenging due to the small field of view of LUS, and usually depends on both a manual initialisation and tracking to compose a volume, hindering clinical translation. In this paper, we extend a proposed registration approach using Content-Based Image Retrieval (CBIR), removing the requirement for tracking or manual initialisation. Pre-operatively, a set of possible LUS planes is simulated from CT and a descriptor generated for each image. Then, a Bayesian framework is employed to estimate the most likely sequence of CT simulations that matches a series of LUS images. We extend our CBIR formulation to use multiple labelled objects and constrain the registration by separating liver vessels into portal vein and hepatic vein branches. The value of this new labeled approach is demonstrated in retrospective data from 5 patients. Results show that, by including a series of 5 untracked images in time, a single LUS image can be registered with accuracies ranging from 5.7 to 16.4 mm with a success rate of 78%. Initialisation of the LUS to CT registration with the proposed framework could potentially enable the clinical translation of these image fusion techniques.

Intraoperative Correction of Liver Deformation Using Sparse Surface and Vascular Features via Linearized Iterative Boundary Reconstruction

During image guided liver surgery, soft tissue deformation can cause considerable error when attempting to achieve accurate localization of the surgical anatomy through image-to-physical registration. In this paper, a linearized iterative boundary reconstruction technique is proposed to account for these deformations. The approach leverages a superposed formulation of boundary conditions to rapidly and accurately estimate the deformation applied to a preoperative model of the organ given sparse intraoperative data of surface and subsurface features. With this method, tracked intraoperative ultrasound (iUS) is investigated as a potential data source for augmenting registration accuracy beyond the capacity of conventional organ surface registration. In an expansive simulated dataset, features including vessel contours, vessel centerlines, and the posterior liver surface are extracted from iUS planes. Registration accuracy is compared across increasing data density to establish how iUS can be best employed to improve target registration error (TRE). From a baseline average TRE of 11.4 ± 2.2 mm using sparse surface data only, incorporating additional sparse features from three iUS planes improved average TRE to 6.4 ± 1.0 mm. Furthermore, increasing the sparse coverage to 16 tracked iUS planes improved average TRE to 3.9 ± 0.7 mm, exceeding the accuracy of registration based on complete surface data available with more cumbersome intraoperative CT without contrast. Additionally, the approach was applied to three clinical cases where on average error improved 67% over rigid registration and 56% over deformable surface registration when incorporating additional features from one independent tracked iUS plane.

Tissue Structure Updating for In Situ Augmented Reality Navigation Using Calibrated Ultrasound and Two-Level Surface Warping

OBJECTIVE

In minimally invasive surgery (MIS), in situ augmented reality (AR) navigation systems are usually implemented using a glasses-free 3D display to represent the preoperative tissue structure, and can provide intuitive see-through guidance information. However, due to changes in intraoperative tissue, the preoperative tissue structure is not able to exactly correspond to reality, which influences the precision of in situ AR navigation. To solve this problem, we propose a method to update the tissue structure for in situ AR navigation in such way to reflect changes in intraoperative tissue.

METHODS

The proposed method to update the tissue structure is based on the calibrated ultrasound and two-level surface warping technologies. Firstly, the particle filter-based calibration is implemented to perform ultrasound calibration and obtain intraoperative position of anatomical points. Secondly, intraoperative positions of anatomical points are inputted in the two-level surface warping method to update the preoperative tissue structure. Finally, the glasses-free real 3-D display of the updated tissue structure is finished, and is superimposed onto a patient by a translucent mirror for in situ AR navigation.

RESULTS

we validated the proposed method by simulating liver tissue intervention, and achieved the tissue updating accuracy of 92.86%. Furthermore, the targeting error of AR navigation based on the proposed method was also evaluated through minimally invasive liver surgery, and the acquired mean targeting error was 1.92 mm.

CONCLUSION

The results demonstrate that the proposed AR navigation method is effective.

SIGNIFICANCE

The proposed method can facilitate MIS, as it provides accurate 3D navigation.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.

Deformable registration of 3D ultrasound volumes using automatic landmark generation

US image registration is an important task e.g. in Computer Aided Surgery. Due to tissue deformation occurring between pre-operative and interventional images often deformable registration is necessary. We present a registration method focused on surface structures (i.e. saliencies) of soft tissues like organ capsules or vessels. The main concept follows the idea of representative landmarks (so called leading points). These landmarks represent saliencies in each image in a certain region of interest. The determination of deformation was based on a geometric model assuming that saliencies could locally be described by planes. These planes were calculated from the landmarks using two dimensional linear regression. Once corresponding regions in both images were found, a displacement vector field representing the local deformation was computed. Finally, the deformed image was warped to match the pre-operative image. For error calculation we used a phantom representing the urinary bladder and the prostate. The phantom could be deformed to mimic tissue deformation. Error calculation was done using corresponding landmarks in both images. The resulting target registration error of this procedure amounted to 1.63 mm. With respect to patient data a full deformable registration was performed on two 3D-US images of the abdomen. The resulting mean distance error was 2.10 ± 0.66 mm compared to an error of 2.75 ± 0.57 mm from a simple rigid registration. A two-sided paired t-test showed a p-value < 0.001. We conclude that the method improves the results of the rigid registration considerably. Provided an appropriate choice of the filter there are many possible fields of applications.