1
|
Zeng Q, Yan R, Zhang L, Yu X, Wu Y, Zheng R, Xu E, Li K. Intelligent automatic registration: is it feasible and efficient for application of ultrasound fusion imaging in liver? Abdom Radiol (NY) 2025; 50:2512-2521. [PMID: 39613872 DOI: 10.1007/s00261-024-04724-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2024] [Revised: 11/24/2024] [Accepted: 11/25/2024] [Indexed: 12/01/2024]
Abstract
PURPOSE To evaluate the feasibility and efficiency between the two intelligent auto-registrations (based on hepatic vessels or based on liver surface) and manual registration for US-CT/MR fusion imaging of liver tumours. METHODS From May 2017 to December 2017, 30 patients with 30 liver tumours were enrolled in this prospectively study. Two intelligent auto-registrations (based on hepatic vessels or based on liver surface) and manual registration were randomly performed, the registration success rate and efficiency were compared. RESULTS In terms of success rate, auto-registrations based on the hepatic vessels (80%) was lower than auto-registration base on liver surface and manual registration (96.67%), but with no statistical difference (P = 0.125). In comparison of the registration efficiency, the efficiency of the auto-registration based on the hepatic vessels was superior to auto-registration based on liver surface and manual registration (P < 0.05). The one-step success rate of auto-registration based on the hepatic vessels (53.33%, 16/30) was higher than that of other two registrations (P < 0.05). Stratified analysis showed that, for the lesion with display of hepatic vessels in grade 3, the success rate of auto-registration based on vessels (0%) was lower than that of auto-registration based on liver surface and manual registration (100%) (P = 0.031). CONCLUSION Intelligent auto-registration based on hepatic vessels is a feasible and efficient registration method for US-CT/MR fusion imaging of liver tumours for the patients with clear hepatic vessels. The auto-registration based on liver surface and manual registration can be an effective supplement for cases with poor hepatic vessels display.
Collapse
Affiliation(s)
- Qingjing Zeng
- The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Ronghua Yan
- Peking University Shenzhen Hospital, Shenzhen, China
| | - Lanxia Zhang
- The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Xuan Yu
- The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Yuxuan Wu
- The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Rongqin Zheng
- The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China
| | - Erjiao Xu
- The Eighth Affiliated Hospital of Sun Yat-Sen University, Shenzhen, China.
| | - Kai Li
- The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China.
| |
Collapse
|
2
|
Saeed SU, Ramalhinho J, Montaña-Brown N, Bonmati E, Pereira SP, Davidson B, Clarkson MJ, Hu Y. Guided ultrasound acquisition for nonrigid image registration using reinforcement learning. Med Image Anal 2025; 102:103555. [PMID: 40168873 DOI: 10.1016/j.media.2025.103555] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 06/26/2024] [Accepted: 03/14/2025] [Indexed: 04/03/2025]
Abstract
We propose a guided registration method for spatially aligning a fixed preoperative image and untracked ultrasound image slices. We exploit the unique interactive and spatially heterogeneous nature of this application to develop a registration algorithm that interactively suggests and acquires ultrasound images at optimised locations (with respect to registration performance). Our framework is based on two trainable functions: (1) a deep hyper-network-based registration function, which is generalisable over varying location and deformation, and adaptable at test-time; (2) a reinforcement learning function for producing test-time estimates of image acquisition locations and adapted deformation regularisation (the latter is required due to varying acquisition locations). We evaluate our proposed method with real preoperative patient data, and simulated intraoperative data with variable field-of-view. In addition to simulation of intraoperative data, we simulate global alignment based on previous work for efficient training, and investigate probe-level guidance towards an improved deformable registration. The evaluation in a simulated environment shows statistically significant improvements in overall registration performance across a variety of metrics for our proposed method, compared to registration without acquisition guidance or adaptable deformation regularisation, and to commonly used classical iterative methods and learning-based registration. For the first time, efficacy of proactive image acquisition is demonstrated in a simulated surgical interventional registration, in contrast to most existing work addressing registration post-data-acquisition, one of the reasons we argue may have led to previously under-constrained nonrigid registration in such applications. Code: https://github.com/s-sd/rl_guided_registration.
Collapse
Affiliation(s)
- Shaheer U Saeed
- UCL Hawkes Institute, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK.
| | - João Ramalhinho
- UCL Hawkes Institute, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Nina Montaña-Brown
- UCL Hawkes Institute, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Ester Bonmati
- UCL Hawkes Institute, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK; School of Computer Science and Engineering, University of Westminster, London, UK
| | - Stephen P Pereira
- Institute for Liver and Digestive Health, University College London, London, UK
| | - Brian Davidson
- Division of Surgery & Interventional Science, University College London, London, UK
| | - Matthew J Clarkson
- UCL Hawkes Institute, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| | - Yipeng Hu
- UCL Hawkes Institute, and Department of Medical Physics & Biomedical Engineering, University College London, London, UK
| |
Collapse
|
3
|
Tu P, Hu P, Wang J, Chen X. From Coarse to Fine: Non-Rigid Sparse-Dense Registration for Deformation-Aware Liver Surgical Navigation. IEEE Trans Biomed Eng 2024; 71:2663-2677. [PMID: 38683702 DOI: 10.1109/tbme.2024.3386704] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/02/2024]
Abstract
OBJECTIVE Intraoperative liver deformation poses a considerable challenge during liver surgery, causing significant errors in image-guided surgical navigation systems. This study addresses a critical non-rigid registration problem in liver surgery: the alignment of intrahepatic vascular trees. The goal is to deform the complete vascular shape extracted from preoperative Computed Tomography (CT) volume, aligning it with sparse vascular contour points obtained from intraoperative ultrasound (iUS) images. Challenges arise due to the intricate nature of slender vascular branches, causing existing methods to struggle with accuracy and vascular self-intersection. METHODS We present a novel non-rigid sparse-dense registration pipeline structured in a coarse-to-fine fashion. In the initial coarse registration stage, we introduce a parametrization deformation graph and a Welsch function-based error metric to enhance convergence and robustness of non-rigid registration. For the fine registration stage, we propose an automatic curvature-based algorithm to detect and eliminate overlapping regions. Subsequently, we generate the complete vascular shape using posterior computation of a Gaussian Process Shape Model. RESULTS Experimental results using simulated data demonstrate the accuracy and robustness of our proposed method. Evaluation results on the target registration error of tumors highlight the clinical significance of our method in tumor location computation. Comparative analysis against related methods reveals superior accuracy and competitive efficiency of our approach. Moreover, Ex vivo swine liver experiments and clinical experiments were conducted to evaluate the method's performance. CONCLUSION The experimental results emphasize the accurate and robust performance of our proposed method. SIGNIFICANCE Our proposed non-rigid registration method holds significant application potential in clinical practice.
Collapse
|
4
|
Ramalhinho J, Koo B, Montaña-Brown N, Saeed SU, Bonmati E, Gurusamy K, Pereira SP, Davidson B, Hu Y, Clarkson MJ. Deep hashing for global registration of untracked 2D laparoscopic ultrasound to CT. Int J Comput Assist Radiol Surg 2022; 17:1461-1468. [PMID: 35366130 PMCID: PMC9307559 DOI: 10.1007/s11548-022-02605-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2022] [Accepted: 03/09/2022] [Indexed: 11/30/2022]
Abstract
PURPOSE The registration of Laparoscopic Ultrasound (LUS) to CT can enhance the safety of laparoscopic liver surgery by providing the surgeon with awareness on the relative positioning between critical vessels and a tumour. In an effort to provide a translatable solution for this poorly constrained problem, Content-based Image Retrieval (CBIR) based on vessel information has been suggested as a method for obtaining a global coarse registration without using tracking information. However, the performance of these frameworks is limited by the use of non-generalisable handcrafted vessel features. METHODS We propose the use of a Deep Hashing (DH) network to directly convert vessel images from both LUS and CT into fixed size hash codes. During training, these codes are learnt from a patient-specific CT scan by supplying the network with triplets of vessel images which include both a registered and a mis-registered pair. Once hash codes have been learnt, they can be used to perform registration with CBIR methods. RESULTS We test a CBIR pipeline on 11 sequences of untracked LUS distributed across 5 clinical cases. Compared to a handcrafted feature approach, our model improves the registration success rate significantly from 48% to 61%, considering a 20 mm error as the threshold for a successful coarse registration. CONCLUSIONS We present the first DH framework for interventional multi-modal registration tasks. The presented approach is easily generalisable to other registration problems, does not require annotated data for training, and may promote the translation of these techniques.
Collapse
Affiliation(s)
- João Ramalhinho
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK.
| | - Bongjin Koo
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK
| | - Nina Montaña-Brown
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK
| | - Shaheer U Saeed
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK
| | - Ester Bonmati
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK
| | | | | | - Brian Davidson
- Division of Surgery and Interventional Science, UCL, London, UK
| | - Yipeng Hu
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK
| | - Matthew J Clarkson
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences and Centre for Medical Image Computing, UCL, London, UK
| |
Collapse
|
5
|
Schneider C, Allam M, Stoyanov D, Hawkes DJ, Gurusamy K, Davidson BR. Performance of image guided navigation in laparoscopic liver surgery - A systematic review. Surg Oncol 2021; 38:101637. [PMID: 34358880 DOI: 10.1016/j.suronc.2021.101637] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2021] [Revised: 07/04/2021] [Accepted: 07/24/2021] [Indexed: 02/07/2023]
Abstract
BACKGROUND Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations. METHODS Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format. RESULTS Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8-15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes. CONCLUSIONS Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard.
Collapse
Affiliation(s)
- C Schneider
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK.
| | - M Allam
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK; General surgery Department, Tanta University, Egypt
| | - D Stoyanov
- Department of Computer Science, University College London, London, UK; Centre for Medical Image Computing (CMIC), University College London, London, UK
| | - D J Hawkes
- Centre for Medical Image Computing (CMIC), University College London, London, UK; Wellcome / EPSRC Centre for Surgical and Interventional Sciences (WEISS), University College London, London, UK
| | - K Gurusamy
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK
| | - B R Davidson
- Department of Surgical Biotechnology, University College London, Pond Street, NW3 2QG, London, UK
| |
Collapse
|
6
|
Xiang K, Jiang B, Shang D. The overview of the deep learning integrated into the medical imaging of liver: a review. Hepatol Int 2021; 15:868-880. [PMID: 34264509 DOI: 10.1007/s12072-021-10229-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Accepted: 06/24/2021] [Indexed: 12/13/2022]
Abstract
Deep learning (DL) is a recently developed artificial intelligent method that can be integrated into numerous fields. For the imaging diagnosis of liver disease, several remarkable outcomes have been achieved with the application of DL currently. This advanced algorithm takes part in various sections of imaging processing such as liver segmentation, lesion delineation, disease classification, process optimization, etc. The DL optimized imaging diagnosis shows a broad prospect instead of the pathological biopsy for the advantages of convenience, safety, and inexpensiveness. In this paper, we reviewed the published representative DL-related hepatic imaging works, described the general situation of this new-rising technology in medical liver imaging and explored the future direction of DL development.
Collapse
Affiliation(s)
- Kailai Xiang
- Department of General Surgery, First Affiliated Hospital of Dalian Medical University, Dalian, 116011, Liaoning, China.,Clinical Laboratory of Integrative Medicine, First Affiliated Hospital of Dalian Medical University, Dalian, 116011, Liaoning, China
| | - Baihui Jiang
- Department of Ophthalmology, First Affiliated Hospital of Dalian Medical University, Dalian, 116011, Liaoning, China
| | - Dong Shang
- Department of General Surgery, First Affiliated Hospital of Dalian Medical University, Dalian, 116011, Liaoning, China. .,Clinical Laboratory of Integrative Medicine, First Affiliated Hospital of Dalian Medical University, Dalian, 116011, Liaoning, China.
| |
Collapse
|
7
|
Heinrich S, Seehofer D, Corvinus F, Tripke V, Huber T, Hüttl F, Penzkofer L, Mittler J, Abu Hilal M, Lang H. [Advantages and future perspectives of laparoscopic liver surgery]. Chirurg 2021; 92:542-549. [PMID: 32995902 DOI: 10.1007/s00104-020-01288-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
BACKGROUND Laparoscopic liver surgery (LLS) is increasingly utilized worldwide due to several potential advantages over open liver surgery. OBJECTIVE Analysis and presentation of the advantages and disadvantages of LLS in comparison to open surgery. MATERIAL AND METHODS Analysis of clinically relevant factors of minimally invasive liver surgery in comparison to open liver surgery in the current literature. RESULTS In addition to obvious cosmetic advantages, the current literature shows advantages regarding length of hospital stay and quality of life after LLC. In contrast to major liver resections, parenchyma-preserving resections often appear cost-neutral due a shorter postoperative hospital stay compared to conventional liver resections. In addition to particular personnel requirements, LLS also has technical prerequisites, such as a dedicated intraoperative ultrasound system. Furthermore, contrast-enhanced laparoscopic examinations are possible and ultrasound information can be fused with preoperative imaging. Virtual reality technology and 3‑dimensional printing are currently under investigation to improve the intraoperative anatomical orientation of LLS. CONCLUSION The current literature reveals significant advantages for LLS so that this procedure should be further developed in Germany in order to offer this technique to as many patients as possible.
Collapse
Affiliation(s)
- Stefan Heinrich
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Daniel Seehofer
- Klinik u. Poliklinik für Viszeral‑, Transplantations‑, Thorax- und Gefäßchirurgie, Universitätsklinikum Leipzig, Leipzig, Deutschland
| | - Florian Corvinus
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Verena Tripke
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Tobias Huber
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Florentine Hüttl
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Lea Penzkofer
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Jens Mittler
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland
| | - Mohammad Abu Hilal
- Department of Surgery, Fondazione Poliambulanza Istituto Ospedaliero Multispecialistico, Brescia, Italien
| | - Hauke Lang
- Klinik f. Allgemein‑, Viszeral- u. Transplantationschirurgie, Universitätsmedizin Mainz, Langenbeckstraße 1, 55131, Mainz, Deutschland.
| |
Collapse
|
8
|
Ramalhinho J, Tregidgo HFJ, Gurusamy K, Hawkes DJ, Davidson B, Clarkson MJ. Registration of Untracked 2D Laparoscopic Ultrasound to CT Images of the Liver Using Multi-Labelled Content-Based Image Retrieval. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:1042-1054. [PMID: 33326379 DOI: 10.1109/tmi.2020.3045348] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Laparoscopic Ultrasound (LUS) is recommended as a standard-of-care when performing laparoscopic liver resections as it images sub-surface structures such as tumours and major vessels. Given that LUS probes are difficult to handle and some tumours are iso-echoic, registration of LUS images to a pre-operative CT has been proposed as an image-guidance method. This registration problem is particularly challenging due to the small field of view of LUS, and usually depends on both a manual initialisation and tracking to compose a volume, hindering clinical translation. In this paper, we extend a proposed registration approach using Content-Based Image Retrieval (CBIR), removing the requirement for tracking or manual initialisation. Pre-operatively, a set of possible LUS planes is simulated from CT and a descriptor generated for each image. Then, a Bayesian framework is employed to estimate the most likely sequence of CT simulations that matches a series of LUS images. We extend our CBIR formulation to use multiple labelled objects and constrain the registration by separating liver vessels into portal vein and hepatic vein branches. The value of this new labeled approach is demonstrated in retrospective data from 5 patients. Results show that, by including a series of 5 untracked images in time, a single LUS image can be registered with accuracies ranging from 5.7 to 16.4 mm with a success rate of 78%. Initialisation of the LUS to CT registration with the proposed framework could potentially enable the clinical translation of these image fusion techniques.
Collapse
|
9
|
Heiselman JS, Jarnagin WR, Miga MI. Intraoperative Correction of Liver Deformation Using Sparse Surface and Vascular Features via Linearized Iterative Boundary Reconstruction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2020; 39:2223-2234. [PMID: 31976882 PMCID: PMC7314378 DOI: 10.1109/tmi.2020.2967322] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
During image guided liver surgery, soft tissue deformation can cause considerable error when attempting to achieve accurate localization of the surgical anatomy through image-to-physical registration. In this paper, a linearized iterative boundary reconstruction technique is proposed to account for these deformations. The approach leverages a superposed formulation of boundary conditions to rapidly and accurately estimate the deformation applied to a preoperative model of the organ given sparse intraoperative data of surface and subsurface features. With this method, tracked intraoperative ultrasound (iUS) is investigated as a potential data source for augmenting registration accuracy beyond the capacity of conventional organ surface registration. In an expansive simulated dataset, features including vessel contours, vessel centerlines, and the posterior liver surface are extracted from iUS planes. Registration accuracy is compared across increasing data density to establish how iUS can be best employed to improve target registration error (TRE). From a baseline average TRE of 11.4 ± 2.2 mm using sparse surface data only, incorporating additional sparse features from three iUS planes improved average TRE to 6.4 ± 1.0 mm. Furthermore, increasing the sparse coverage to 16 tracked iUS planes improved average TRE to 3.9 ± 0.7 mm, exceeding the accuracy of registration based on complete surface data available with more cumbersome intraoperative CT without contrast. Additionally, the approach was applied to three clinical cases where on average error improved 67% over rigid registration and 56% over deformable surface registration when incorporating additional features from one independent tracked iUS plane.
Collapse
Affiliation(s)
| | - William R. Jarnagin
- Department of Surgery at Memorial Sloan Kettering Cancer Center, New York, NY 10065 USA
| | - Michael I. Miga
- Department of Biomedical Engineering at Vanderbilt University, Nashville, TN 37235 USA
| |
Collapse
|
10
|
Chen F, Cui X, Liu J, Han B, Zhang X, Zhang D, Liao H. Tissue Structure Updating for In Situ Augmented Reality Navigation Using Calibrated Ultrasound and Two-Level Surface Warping. IEEE Trans Biomed Eng 2020; 67:3211-3222. [PMID: 32175853 DOI: 10.1109/tbme.2020.2979535] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
OBJECTIVE In minimally invasive surgery (MIS), in situ augmented reality (AR) navigation systems are usually implemented using a glasses-free 3D display to represent the preoperative tissue structure, and can provide intuitive see-through guidance information. However, due to changes in intraoperative tissue, the preoperative tissue structure is not able to exactly correspond to reality, which influences the precision of in situ AR navigation. To solve this problem, we propose a method to update the tissue structure for in situ AR navigation in such way to reflect changes in intraoperative tissue. METHODS The proposed method to update the tissue structure is based on the calibrated ultrasound and two-level surface warping technologies. Firstly, the particle filter-based calibration is implemented to perform ultrasound calibration and obtain intraoperative position of anatomical points. Secondly, intraoperative positions of anatomical points are inputted in the two-level surface warping method to update the preoperative tissue structure. Finally, the glasses-free real 3-D display of the updated tissue structure is finished, and is superimposed onto a patient by a translucent mirror for in situ AR navigation. RESULTS we validated the proposed method by simulating liver tissue intervention, and achieved the tissue updating accuracy of 92.86%. Furthermore, the targeting error of AR navigation based on the proposed method was also evaluated through minimally invasive liver surgery, and the acquired mean targeting error was 1.92 mm. CONCLUSION The results demonstrate that the proposed AR navigation method is effective. SIGNIFICANCE The proposed method can facilitate MIS, as it provides accurate 3D navigation.
Collapse
|
11
|
Vercauteren T, Unberath M, Padoy N, Navab N. CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions. PROCEEDINGS OF THE IEEE. INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS 2020; 108:198-214. [PMID: 31920208 PMCID: PMC6952279 DOI: 10.1109/jproc.2019.2946993] [Citation(s) in RCA: 55] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/01/2019] [Revised: 09/12/2019] [Accepted: 10/04/2019] [Indexed: 05/10/2023]
Abstract
Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.
Collapse
Affiliation(s)
- Tom Vercauteren
- School of Biomedical Engineering & Imaging SciencesKing’s College LondonLondonWC2R 2LSU.K.
| | - Mathias Unberath
- Department of Computer ScienceJohns Hopkins UniversityBaltimoreMD21218USA
| | - Nicolas Padoy
- ICube institute, CNRS, IHU Strasbourg, University of Strasbourg67081StrasbourgFrance
| | - Nassir Navab
- Fakultät für InformatikTechnische Universität München80333MunichGermany
| |
Collapse
|
12
|
Figl M, Hoffmann R, Kaar M, Hummel J. Deformable registration of 3D ultrasound volumes using automatic landmark generation. PLoS One 2019; 14:e0213004. [PMID: 30875379 PMCID: PMC6420033 DOI: 10.1371/journal.pone.0213004] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2018] [Accepted: 02/13/2019] [Indexed: 11/18/2022] Open
Abstract
US image registration is an important task e.g. in Computer Aided Surgery. Due to tissue deformation occurring between pre-operative and interventional images often deformable registration is necessary. We present a registration method focused on surface structures (i.e. saliencies) of soft tissues like organ capsules or vessels. The main concept follows the idea of representative landmarks (so called leading points). These landmarks represent saliencies in each image in a certain region of interest. The determination of deformation was based on a geometric model assuming that saliencies could locally be described by planes. These planes were calculated from the landmarks using two dimensional linear regression. Once corresponding regions in both images were found, a displacement vector field representing the local deformation was computed. Finally, the deformed image was warped to match the pre-operative image. For error calculation we used a phantom representing the urinary bladder and the prostate. The phantom could be deformed to mimic tissue deformation. Error calculation was done using corresponding landmarks in both images. The resulting target registration error of this procedure amounted to 1.63 mm. With respect to patient data a full deformable registration was performed on two 3D-US images of the abdomen. The resulting mean distance error was 2.10 ± 0.66 mm compared to an error of 2.75 ± 0.57 mm from a simple rigid registration. A two-sided paired t-test showed a p-value < 0.001. We conclude that the method improves the results of the rigid registration considerably. Provided an appropriate choice of the filter there are many possible fields of applications.
Collapse
Affiliation(s)
- Michael Figl
- Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| | - Rainer Hoffmann
- Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| | - Marcus Kaar
- Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| | - Johann Hummel
- Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
- * E-mail:
| |
Collapse
|