Augmented reality in laparoscopic liver resection evaluated on an ex-vivo animal model with pseudo-tumours

A new technology for a novel clinical approach in a dog with a complex vascular anomaly: the "extended reality"

Extended reality includes both virtual and augmented realities. In virtual reality objects are rendered in an artificial environment where the user can move and interact with a head mounted display. In augmented reality virtual objects are superimposed to real environment enriching it via a head mounted display. In human medicine these technologies have been already used for educational surgical purposes, but remain relatively unknown in veterinary medicine. We report a case of a 1-year-old, female, French bulldog presented for exercise intolerance and dyspnea. Echocardiography showed signs of left ventricular enlargement with reduced fractional shortening and turbulent flow distal to the pulmonary artery bifurcation. Computed tomography revealed a complex vascular network comprising the descending aorta and left pulmonary artery resembling a patent ductus arteriosus. Virtual reality was used for the surgical planning and a left thoracotomy was performed to close the abnormal vessel at the level of the entrance in the left pulmonary artery with augmented reality assistance. No complications were reported during or after the surgery and the dog completely recovered. Echocardiographic findings 3 days, 1 month and 18 months after the surgery demonstrated absence of residual flow and improving ventricular dimensions. To our knowledge this report documents the first use of extended reality for the visualization, planning and execution of the surgical correction of a complex vascular defect in veterinary medicine.

Video-Based Soft Tissue Deformation Tracking for Laparoscopic Augmented Reality-Based Navigation in Kidney Surgery

Minimally invasive surgery (MIS) remains technically demanding due to the difficulty of tracking hidden critical structures within the moving anatomy of the patient. In this study, we propose a soft tissue deformation tracking augmented reality (AR) navigation pipeline for laparoscopic surgery of the kidneys. The proposed navigation pipeline addresses two main sub-problems: the initial registration and deformation tracking. Our method utilizes preoperative MR or CT data and binocular laparoscopes without any additional interventional hardware. The initial registration is resolved through a probabilistic rigid registration algorithm and elastic compensation based on dense point cloud reconstruction. For deformation tracking, the sparse feature point displacement vector field continuously provides temporal boundary conditions for the biomechanical model. To enhance the accuracy of the displacement vector field, a novel feature points selection strategy based on deep learning is proposed. Moreover, an ex-vivo experimental method for internal structures error assessment is presented. The ex-vivo experiments indicate an external surface reprojection error of 4.07 ± 2.17 mm and a maximum mean absolutely error for internal structures of 2.98 mm. In-vivo experiments indicate mean absolutely error of 3.28 ± 0.40 mm and 1.90 ± 0.24 mm, respectively. The combined qualitative and quantitative findings indicated the potential of our AR-assisted navigation system in improving the clinical application of laparoscopic kidney surgery.

Augmented Reality Implementation in Minimally Invasive Surgery for Future Application in Pulmonary Surgery: A Systematic Review

OBJECTIVE

This systematic review investigates of Augmented Reality (AR) systems used in minimally invasive surgery of deformable organs, focusing on initial registration, dynamic tracking, and visualization. The objective is to acquire a comprehensive understanding of the current knowledge, applications, and challenges associated with current AR-techniques, aiming to leverage these insights for developing a dedicated AR pulmonary Video or Robotic Assisted Thoracic Surgery (VATS/RATS) workflow.

METHODS

A systematic search was conducted within Embase, Medline (Ovid) and Web of Science on April 16, 2024, following the Preferred Reporting items for Systematic Reviews and Meta-Analyses (PRISMA). The search focused on intraoperative AR applications and intraoperative navigational purposes for deformable organs. Quality assessment was performed and studies were categorized according to initial registration and dynamic tracking methods.

RESULTS

33 articles were included, of which one involved pulmonary surgery. Studies used both manual and (semi-) automatic registration methods, established through anatomical landmark-based, fiducial-based, or surface-based techniques. Diverse outcome measures were considered, including surgical outcomes and registration accuracy. The majority of studies that reached an registration accuracy below 5 mm applied surface-based registration.

CONCLUSIONS

AR can potentially aid surgeons with real-time navigation and decision making during anatomically complex minimally invasive procedures. Future research for pulmonary applications should focus on exploring surface-based registration methods, considering their non-invasive, marker-less nature, and promising accuracy. Additionally, vascular-labeling-based methods are worth exploring, given the importance and relative stability of broncho-vascular anatomy in pulmonary VATS/RATS. Assessing clinical feasibility of these approaches is crucial, particularly concerning registration accuracy and potential impact on surgical outcomes.

Nomogram for predicting early recurrence of hepatocellular carcinoma with narrow resection margin

PURPOSE

Narrow resection margin hepatocellular carcinoma (NRM-HCC) has a high incidence of early recurrence. Our study was designed to identify prognostic factors in patients with NRM-HCC, establish and validate a nomogram model to predict early recurrence of NRM-HCC patients.

METHODS

We retrospectively analyzed data from 2957 NRM-HCC patients who underwent radical hepatectomy at three medical centers between December 2009 and January 2015. Patients were randomly assigned to a training cohort (n = 2069) and a validation cohort (n = 888). Using univariate and multivariate COX regression to determine early relapse factors in NRM-HCC patients, and used these factors to construct a nomogram. The accuracy of the prediction was evaluated using the C-index, receiver operating characteristic (ROC) and calibration curve. Decision curve analysis (DCA) assessed the predictive value of the models. Finally, the recurrence-free survival of different risks was analyzed using Kaplan-Meier (K-M) method.

RESULTS

The nomogram of NRM model contains alpha-fetoprotein (AFP), alkaline phosphatase (ALP), tumor size, tumor number, microvascular invasion (MVI), tumor capsular, and satellite nodules. The model shows good discrimination with C-indexes of 0.71 (95% CI: 0.69-0.72) and 0.72 (95% CI: 0.70-0.75) in the train cohort and test cohort respectively. Decision curve analysis demonstrated that the model is clinically useful and the calibration of our model was favorable. Our model stratified patients into two different risk groups, which exhibited significantly different early recurrence. The web-based tools are convenient for clinical practice.

CONCLUSIONS

NRM model demonstrated favorable performance in predicting early recurrence in NRM-HCC patients. This novel model will be helpful to guide postoperative follow-up and adjuvant therapy.

Navigating the Future of 3D Laparoscopic Liver Surgeries: Visualization of Internal Anatomy on Laparoscopic Images With Augmented Reality

INTRODUCTION

Liver tumor resection requires precise localization of tumors and blood vessels. Despite advancements in 3-dimensional (3D) visualization for laparoscopic surgeries, challenges persist. We developed and evaluated an augmented reality (AR) system that overlays preoperative 3D models onto laparoscopic images, offering crucial support for 3D visualization during laparoscopic liver surgeries.

METHODS

Anatomic liver structures from preoperative computed tomography scans were segmented using open-source software including 3D Slicer and Maya 2022 for 3D model editing. A registration system was created with 3D visualization software utilizing a stereo registration input system to overlay the virtual liver onto laparoscopic images during surgical procedures. A controller was customized using a modified keyboard to facilitate manual alignment of the virtual liver with the laparoscopic image. The AR system was evaluated by 3 experienced surgeons who performed manual registration for a total of 27 images from 7 clinical cases. The evaluation criteria included registration time; measured in minutes, and accuracy; measured using the Dice similarity coefficient.

RESULTS

The overall mean registration time was 2.4±1.7 minutes (range: 0.3 to 9.5 min), and the overall mean registration accuracy was 93.8%±4.9% (range: 80.9% to 99.7%).

CONCLUSION

Our validated AR system has the potential to effectively enable the prediction of internal hepatic anatomic structures during 3D laparoscopic liver resection, and may enhance 3D visualization for select laparoscopic liver surgeries.

Augmented Reality Guided Laparoscopic Liver Resection: A Phantom Study With Intraparenchymal Tumors

INTRODUCTION

Augmented reality (AR) in laparoscopic liver resection (LLR) can improve intrahepatic navigation by creating a virtual liver transparency. Our team has recently developed Hepataug, an AR software that projects the invisible intrahepatic tumors onto the laparoscopic images and allows the surgeon to localize them precisely. However, the accuracy of registration according to the location and size of the tumors, as well as the influence of the projection axis, have never been measured. The aim of this work was to measure the three-dimensional (3D) tumor prediction error of Hepataug.

METHODS

Eight 3D virtual livers were created from the computed tomography scan of a healthy human liver. Reference markers with known coordinates were virtually placed on the anterior surface. The virtual livers were then deformed and 3D printed, forming 3D liver phantoms. After placing each 3D phantom inside a pelvitrainer, registration allowed Hepataug to project virtual tumors along two axes: the laparoscope axis and the operator port axis. The surgeons had to point the center of eight virtual tumors per liver with a pointing tool whose coordinates were precisely calculated.

RESULTS

We obtained 128 pointing experiments. The average pointing error was 29.4 ± 17.1 mm and 9.2 ± 5.1 mm for the laparoscope and operator port axes respectively (P = 0.001). The pointing errors tended to increase with tumor depth (correlation coefficients greater than 0.5 with P < 0.001). There was no significant dependency of the pointing error on the tumor size for both projection axes.

CONCLUSIONS

Tumor visualization by projection toward the operating port improves the accuracy of AR guidance and partially solves the problem of the two-dimensional visual interface of monocular laparoscopy. Despite a lower precision of AR for tumors located in the posterior part of the liver, it could allow the surgeons to access these lesions without completely mobilizing the liver, hence decreasing the surgical trauma.

Augmented reality in liver surgery

INTRODUCTION

During an operation, augmented reality (AR) enables surgeons to enrich their vision of the operating field by means of digital imagery, particularly as regards tumors and anatomical structures. While in some specialties, this type of technology is routinely ustilized, in liver surgery due to the complexity of modeling organ deformities in real time, its applications remain limited. At present, numerous teams are attempting to find a solution applicable to current practice, the objective being to overcome difficulties of intraoperative navigation in an opaque organ.

OBJECTIVE

To identify, itemize and analyze series reporting AR techniques tested in liver surgery, the objectives being to establish a state of the art and to provide indications of perspectives for the future.

METHODS

In compliance with the PRISMA guidelines and availing ourselves of the PubMed, Embase and Cochrane databases, we identified English-language articles published between January 2020 and January 2022 corresponding to the following keywords: augmented reality, hepatic surgery, liver and hepatectomy.

RESULTS

Initially, 102 titles, studies and summaries were preselected. Twenty-eight corresponding to the inclusion criteria were included, reporting on 183patients operated with the help of AR by laparotomy (n=31) or laparoscopy (n=152). Several techniques of acquisition and visualization were reported. Anatomical precision was the main assessment criterion in 19 articles, with values ranging from 3mm to 14mm, followed by time of acquisition and clinical feasibility.

CONCLUSION

While several AR technologies are presently being developed, due to insufficient anatomical precision their clinical applications have remained limited. That much said, numerous teams are currently working toward their optimization, and it is highly likely that in the short term, the application of AR in liver surgery will have become more frequent and effective. As for its clinical impact, notably in oncology, it remains to be assessed.

Phantom study on surgical performance in augmented reality laparoscopy

Purpose

Only a few studies have evaluated Augmented Reality (AR) in in vivo simulations compared to traditional laparoscopy; further research is especially needed regarding the most effective AR visualization technique. This pilot study aims to determine, under controlled conditions on a 3D-printed phantom, whether an AR laparoscope improves surgical outcomes over conventional laparoscopy without augmentation.

Methods

We selected six surgical residents at a similar level of training and had them perform a laparoscopic task. The participants repeated the experiment three times, using different 3D phantoms and visualizations: Floating AR, Occlusion AR, and without any AR visualization (Control). Surgical performance was determined using objective measurements. Subjective measures, such as task load and potential application areas, were collected with questionnaires.

Results

Differences in operative time, total touching time, and SurgTLX scores showed no statistical significance ($$p>0.05$$ p > 0.05 ). However, when assessing the invasiveness of the simulated intervention, the comparison revealed a statistically significant difference ($$p=0.009$$ p = 0.009 ). Participants felt AR could be useful for various surgeries, especially for liver, sigmoid, and pancreatic resections (100%). Almost all participants agreed that AR could potentially lead to improved surgical parameters, such as operative time (83%), complication rate (83%), and identifying risk structures (83%).

Conclusion

According to our results, AR may have great potential in visceral surgery and based on the objective measures of the study, may improve surgeons' performance in terms of an atraumatic approach. In this pilot study, participants consistently took more time to complete the task, had more contact with the vascular tree, were significantly more invasive, and scored higher on the SurgTLX survey than with AR.

State of the Art and Future Prospects of Virtual and Augmented Reality in Veterinary Medicine: A Systematic Review

Virtual reality and augmented reality are new but rapidly expanding topics in medicine. In virtual reality, users are immersed in a three-dimensional environment, whereas in augmented reality, computer-generated images are superimposed on the real world. Despite advances in human medicine, the number of published articles in veterinary medicine is low. These cutting-edge technologies can be used in combination with existing methods in veterinary medicine to achieve diagnostic/therapeutic and educational goals. The purpose of our review was to evaluate studies for their use of virtual reality and augmented reality in veterinary medicine, as well as human medicine with animal trials, to report results and the state of the art. We collected all of the articles we included in our review by screening the Scopus, PubMed, and Web of Science databases. Of the 24 included studies, 11 and 13 articles belonged to virtual reality and augmented reality, respectively. Based on these articles, we determined that using these technologies has a positive impact on the scientific output of students and residents, can reduce training costs, and can be used in training/educational programs. Furthermore, using these tools can promote ethical standards. We reported the absence of standard operation protocols and equipment costs as study limitations.

The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

This is a review focused on advances and current limitations of computer vision (CV) and how CV can help us obtain to more autonomous actions in surgery. It is a follow-up article to one that we previously published in Sensors entitled, "Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?" As opposed to that article that also discussed issues of machine learning, deep learning and natural language processing, this review will delve deeper into the field of CV. Additionally, non-visual forms of data that can aid computerized robots in the performance of more autonomous actions, such as instrument priors and audio haptics, will also be highlighted. Furthermore, the current existential crisis for surgeons, endoscopists and interventional radiologists regarding more autonomy during procedures will be discussed. In summary, this paper will discuss how to harness the power of CV to keep doctors who do interventions in the loop.

Augmented Reality-Based Visual Cue for Guiding Central Catheter Insertion in Pediatric Oncologic Patients

BACKGROUND

Pediatric hemato-oncologic patients require central catheters for chemotherapy, and the junction of the superior vena cava and right atrium is considered the ideal location for catheter tips. Skin landmarks or fluoroscopic supports have been applied to identify the cavoatrial junction; however, none has been recognized as the gold standard. Therefore, we aim to develop a safe and accurate technique using augmented reality technology for the location of the cavoatrial junction in pediatric hemato-oncologic patients.

METHODS

Fifteen oncology patients who underwent chest computed tomography were enrolled for Hickman catheter or chemoport insertion. With the aid of augmented reality technology, three-dimensional models of the internal jugular veins, external jugular veins, subclavian veins, superior vena cava, and right atrium were constructed. On inserting the central vein catheters, the cavoatrial junction identified using the three-dimensional models were marked on the body surface, the tip was positioned at the corresponding location, and the actual insertion location was confirmed using a portable x-ray machine. The proposed method was evaluated by comparing the distance from the cavoatrial junction to the augmented reality location with that to the conventional location on x-ray.

RESULTS

The mean distance between the cavoatrial junction and augmented reality location on x-ray was 1.2 cm, which was significantly shorter than that between the cavoatrial junction and conventional location (1.9 cm; P = 0.027).

CONCLUSIONS

Central catheter insertion using augmented reality technology is more safe and accurate than that using conventional methods and can be performed at no additional cost in oncology patients.