Augmented Reality and 3-Dimensional Printing Technologies for Guiding Complex Thoracoscopic Surgery

Intraoperative support using mixed reality holograms for hilar and mediastinal lymph node dissection

BACKGROUND

Accurate lymph node (LN) dissection is crucial in the surgical management of lung cancer. However, studies addressing the challenges of anatomically precise LN dissection are limited. This study aimed to evaluate the utility and safety of mixed reality (MR) holograms as an intraoperative support tool for hilar and mediastinal lymph node dissection (HMLND).

METHODS

Polygonal (stereolithographic) files of the thoracic cavity were created from SYNAPSE VINCENT images and uploaded into Holoeyes MD virtual reality software. The three-dimensional (3D) holograms generated from these images were displayed on head-mounted displays (HoloLens2) worn by the surgeons and their assistants during HMLND. Intraoperative hologram support (IHS) safety was assessed in 10 patients who underwent HMLND with anatomical lung resection. Additionally, the utility of the intraoperative hologram was evaluated through a questionnaire completed by four thoracic surgeons experienced in hologram manipulation.

RESULTS

IHS was used for a median duration of 5 min (interquartile range: 4-6). No 90-day postoperative complications were observed. Surgeons unanimously agreed that the holograms accurately represented the vascular and bronchial structures of the hilar and mediastinal regions. Additionally, none of the surgeons disagreed that the holograms provided greater benefit intraoperatively compared to preoperatively. In particular, IHS was found effective for the dissection of non-adjacent interlobar LNs post-segmentectomy and LN #4L.

CONCLUSIONS

IHS improved the surgical understanding of thoracic anatomy during HMLND, potentially leading to more precise and reliable LN dissection.

Clinical observation of three-dimensional reconstruction in thoracoscopic segmental pneumonectomy

BACKGROUND

Accurately identifying the branches of pulmonary segmental vessels and bronchi, as well as adjacent structures, and determining the spatial location of lesions within pulmonary segments, are major challenges for thoracic surgeons. The application of three-dimensional reconstruction technology holds promise in addressing this issue.

OBJECTIVE

To evaluate the clinical value of three-dimensional reconstruction in thoracoscopic segmental surgery.

METHODS

Seventy-seven patients who underwent thoracoscopic segmental surgery combined with three-dimensional reconstruction at our hospital from January 1, 2020, to August 31, 2023, were retrospectively analyzed. Preoperative chest enhanced CT scans were conducted, and MIMICS software aided in reconstructing DICOM format original data for patients with pulmonary nodules to facilitate intraoperative nodule localization. Accurate segmental pneumonectomy was performed by comparing preoperative anatomical identification of target segmental arteries, veins, and bronchi, with surgical details and postoperative outcomes recorded, including intraoperative pulmonary resection distribution, operation time, blood loss, chest tube drainage, extubation time, hospital stay, and complications.

RESULTS

Following preoperative three-dimensional reconstruction, successful segmental lung surgeries were performed, predominantly with single segmental resection (92.2%), and a minority with combined segmentectomy (7.8%). Median operation time was 130225 minutes, with intraoperative blood loss at 70100 mL and postoperative chest tube drainage at 347 mL (159690 mL). Median extubation time and hospital stay were 4 days and 7 days, respectively. Complications within the 3-month follow-up affected 11.7% of cases, including persistent pulmonary leakage (7.1%), pulmonary infection (4.3%), atelectasis (4.3%), and pleural effusion (1.4%), with no fatalities.

CONCLUSION

Preoperative 3D reconstruction can help the operator to perform safe, efficient and accurate thoracoscopic segmental pneumonectomy, which is worth popularizing in clinic.

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

Augmented Reality Implementation in Minimally Invasive Surgery for Future Application in Pulmonary Surgery: A Systematic Review

OBJECTIVE

This systematic review investigates of Augmented Reality (AR) systems used in minimally invasive surgery of deformable organs, focusing on initial registration, dynamic tracking, and visualization. The objective is to acquire a comprehensive understanding of the current knowledge, applications, and challenges associated with current AR-techniques, aiming to leverage these insights for developing a dedicated AR pulmonary Video or Robotic Assisted Thoracic Surgery (VATS/RATS) workflow.

METHODS

A systematic search was conducted within Embase, Medline (Ovid) and Web of Science on April 16, 2024, following the Preferred Reporting items for Systematic Reviews and Meta-Analyses (PRISMA). The search focused on intraoperative AR applications and intraoperative navigational purposes for deformable organs. Quality assessment was performed and studies were categorized according to initial registration and dynamic tracking methods.

RESULTS

33 articles were included, of which one involved pulmonary surgery. Studies used both manual and (semi-) automatic registration methods, established through anatomical landmark-based, fiducial-based, or surface-based techniques. Diverse outcome measures were considered, including surgical outcomes and registration accuracy. The majority of studies that reached an registration accuracy below 5 mm applied surface-based registration.

CONCLUSIONS

AR can potentially aid surgeons with real-time navigation and decision making during anatomically complex minimally invasive procedures. Future research for pulmonary applications should focus on exploring surface-based registration methods, considering their non-invasive, marker-less nature, and promising accuracy. Additionally, vascular-labeling-based methods are worth exploring, given the importance and relative stability of broncho-vascular anatomy in pulmonary VATS/RATS. Assessing clinical feasibility of these approaches is crucial, particularly concerning registration accuracy and potential impact on surgical outcomes.

More than meets the eye: Augmented reality in surgical oncology

BACKGROUND AND OBJECTIVES

In the field of surgical oncology, there has been a desire for innovative techniques to improve tumor visualization, resection, and patient outcomes. Augmented reality (AR) technology superimposes digital content onto the real-world environment, enhancing the user's experience by blending digital and physical elements. A thorough examination of AR technology in surgical oncology has yet to be performed.

METHODS

A scoping review of intraoperative AR in surgical oncology was conducted according to the guidelines and recommendations of The Preferred Reporting Items for Systematic Review and Meta-analyzes Extension for Scoping Reviews (PRISMA-ScR) framework. All original articles examining the use of intraoperative AR during surgical management of cancer were included. Exclusion criteria included virtual reality applications only, preoperative use only, fluorescence, AR not specific to surgical oncology, and study design (reviews, commentaries, abstracts).

RESULTS

A total of 2735 articles were identified of which 83 were included. Most studies (52) were performed on animals or phantom models, while the remaining included patients. A total of 1112 intraoperative AR surgical cases were performed across the studies. The most common anatomic site was brain (20 articles), followed by liver (16), renal (9), and head and neck (8). AR was most often used for intraoperative navigation or anatomic visualization of tumors or critical structures but was also used to identify osteotomy or craniotomy planes.

CONCLUSIONS

AR technology has been applied across the field of surgical oncology to aid in localization and resection of tumors.

[Modern individualized diagnostics and treatment of non-small cell lung cancer]

Approximately one half of patients with non-small cell lung cancer (NSCLC) are diagnosed at resectable tumor stages (I-IIIA), which can potentially be curatively treated. In the early tumor stages (tumor diameter ≤2 cm) sublobar resection (segmentectomy or atypical wedge resection) leads to a 5‑year long-term survival comparable to lobectomy. The use of immunotherapy, especially within the framework of neoadjuvant treatment, is anticipated to change the surgical treatment of NSCLC in the future. With the introduction of lung cancer screening for certain risk groups in Germany planned for 2024, lung tumors can be expected to be diagnosed at earlier stages and more frequently curatively treated. This article provides an overview of the potential impact of lung cancer screening, modern minimally invasive surgical techniques and neoadjuvant treatment concepts for the surgical treatment of NSCLC.

The future of artificial intelligence in thoracic surgery for non-small cell lung cancer treatment a narrative review

OBJECTIVES

To present a comprehensive review of the current state of artificial intelligence (AI) applications in lung cancer management, spanning the preoperative, intraoperative, and postoperative phases.

METHODS

A review of the literature was conducted using PubMed, EMBASE and Cochrane, including relevant studies between 2002 and 2023 to identify the latest research on artificial intelligence and lung cancer.

CONCLUSION

While AI holds promise in managing lung cancer, challenges exist. In the preoperative phase, AI can improve diagnostics and predict biomarkers, particularly in cases with limited biopsy materials. During surgery, AI provides real-time guidance. Postoperatively, AI assists in pathology assessment and predictive modeling. Challenges include interpretability issues, training limitations affecting model use and AI's ineffectiveness beyond classification. Overfitting and global generalization, along with high computational costs and ethical frameworks, pose hurdles. Addressing these challenges requires a careful approach, considering ethical, technical, and regulatory factors. Rigorous analysis, external validation, and a robust regulatory framework are crucial for responsible AI implementation in lung surgery, reflecting the evolving synergy between human expertise and technology.

Clinical application of a three-dimensional-printed model in the treatment of intracranial and extracranial communicating tumors: a pilot study

BACKGROUND

Surgical management for intracranial and extracranial communicating tumors is difficult due to the complex anatomical structures. Therefore, assisting methods are urgently needed. Accordingly, this study aimed to investigate the utility of a three-dimensional (3D)-printed model in the treatment of intracranial and extracranial communicating tumors as well as its applicability in surgical planning and resident education.

METHODS

Individualized 3D-printed models were created for eight patients with intracranial and extracranial communicating tumors. Based on these 3D-printed models, a comprehensive surgical plan was made for each patient, after which the patients underwent surgery. The clinicopathological data of patients were collected and retrospectively analyzed to determine surgical outcomes. To examine the educational capability of the 3D-printed models, specialists and resident doctors were invited to review three of these cases and then rate the clinical utility of the models using a questionnaire.

RESULTS

The 3D-printed models accurately replicated anatomical structures, including the tumor, surrounding structures, and the skull. Based on these models, customized surgical approaches, including the orbitozygomatic approach and transcervical approach, were designed for the patients. Although parameters such as operation time and blood loss varied among the patients, satisfactory surgical outcomes were achieved, with only one patient developing a postoperative complication. Regarding the educational applicability of the 3D-printed model, the mean agreement for all eight questionnaire items was above six (seven being complete agreement). Moreover, no significant difference was noted in the agreement scores between specialists and residents.

CONCLUSION

The results revealed that 3D-printed models have good structural accuracy and are potentially beneficial in developing surgical approaches and educating residents. Further research is needed to test the true applicability of these models in the treatment of intracranial and extracranial communicating tumors.

What's new in minimally invasive thoracic surgery? Clinical application of augmented reality and learning opportunities in surgical simulation

Lung cancer represents the most lethal cancer worldwide. Surgery is the treatment of choice for early-stage non-small cell lung cancer, with an overall survival that can reach 90% at 5 years, but its detection is difficult to achieve due to the lack of symptoms. Screening programs are crucial to identify small cancer. Minimally invasive surgery has modified the therapeutical approach of these tumors, becoming the standard of care, with an important clinical yield in terms of reduction of postoperative pain and length of hospital stay. The aim of this mini-review is to explore and describe two important and innovative aspects in the context of "growing opportunities in minimally invasive thoracic surgery": the clinical application of augmented reality and its advantages for patient and surgeon, and the pedagogical issue through simulation-based training.

The Role of Sublobar Resection for the Surgical Treatment of Non-Small Cell Lung Cancer

Lung cancer is the most common cancer killer in the world. The standard of care for surgical treatment of non-small cell lung cancer has been lobectomy. Recent studies have identified that sublobar resection has non-inferior survival rates compared to lobectomy, however. Sublobar resection may increase the number of patients who can tolerate surgery and reduce postoperative pulmonary decline. Sublobar resection appears to have equivalent results to surgery in patients with small, peripheral tumors and no lymph node disease. As the utilization of segmentectomy increases, there may be some centers that perform this operation more than other centers. Care must be taken to ensure that all patients have access to this modality. Future investigations should focus on examining the outcomes from segmentectomy as it is applied more widely. When employed on a broad scale, morbidity and survival rates should be monitored. As segmentectomy is performed more frequently, patients may experience improved postoperative quality of life while maintaining the same oncologic benefit.

A scoping review of deep learning in cancer nursing combined with augmented reality: The era of intelligent nursing is coming

Artificial intelligence has been developing greatly in the field of medicine. As a new research hotspot of artificial intelligence, deep learning (DL) has been widely applied in the fields of cancer risk assessment, symptom recognition, and cancer detection. Therefore, nursing care issues in terms of consuming time and energy, lower accuracy, and lower efficiency can be solved with applying DL in caring cancer patients. In addition, augmented reality (AR) has great navigation potential through combining computer-generated virtual elements with the real world. Thus, DL ​+ ​AR may facilitate patients with cancer to possess a brand-new model of nursing care that is more intelligent, mobile, and adapted to the information age, compared to traditional nursing. With the advent of the era of intelligent nursing, future nursing models can not only learn from the DL ​+ ​AR model to meet the needs of patients with cancer but also reduce nursing workload, save healthcare resources, and improve work efficiency, the quality of nursing care, as well as the quality of life for cancer patients.

Single-port robotic surgery for mediastinal tumors using the da vinci SP system: Initial experience

PURPOSE

Studies of single-port robot-assisted thoracic surgery (RATS) using the da Vinci SP system, which uses a smaller surgical incision than the conventional multiport robot, have yet to be reported because of its smaller operating range. We report our initial experience using the da Vinci SP system in thoracic surgery for the resection of mediastinal tumors that requires a smaller workspace.

DESCRIPTION

Two patients diagnosed with superior mediastinal tumors underwent RATS performed with the da Vinci SP surgical system in January 2022. We used three-dimensional reconstruction to preoperatively determine the surgical incision. This is the first report of single-port RATS using the SP system in China.

EVALUATION

R0 resection was achieved in both operations without complications. Operation times and bleeding volumes were similar to the use of multiport RATS. No perioperative complications occurred.

CONCLUSIONS

The da Vinci SP system can be used for the resection of superior mediastinal tumors. Case selection and preoperative planning should be performed prior to these surgeries.

State of the Art in Lung Nodule Localization

Lung nodule and ground-glass opacity localization for diagnostic and therapeutic purposes is often a challenge for thoracic surgeons. While there are several adjuncts and techniques in the surgeon's armamentarium that can be helpful, accurate localization persists as a problem without a perfect solution. The last several decades have seen tremendous improvement in our ability to perform major operations with minimally invasive procedures and resulting lower morbidity. However, technological advances have not been as widely realized for lung nodule localization to complement minimally invasive surgery. This review describes the latest advances in lung nodule localization technology while also demonstrating that more efforts in this area are needed.

Application Research of Three-Dimensional Printing Technology and Three-Dimensional Computed Tomography in Segmentectomy

Background

To compare the application of the emerging 3D printing technology and 3D-CT in segmentectomy. And to explore the advantages of 3D printing technology in thoracoscopic segmentectomy.

Methods

We collected the clinical data of 118 patients undergoing thoracoscopic segmentectomy from January 2019 to April 2021 at the Thoracic Surgery Department, the Dianjiang People's Hospital of Chongqing and Southwest Hospital. Among them, 61 patients were in the 3D printing group and 57 patients were in the 3D-CT group respectively. The perioperative data of these two groups of patients were analyzed respectively.

Results

There were no significant differences between the two groups in age, gender, tumor diameter, pathology, the preoperative complications of diabetes and heart disease. However, the patients with the complications of hypertension in the 3D printing group are significantly more than the 3D-CT group (P = 0.003). Compared with the 3D-CT group, patients in the 3D printing group had significantly shorter operation time (162.7 ± 47.0 vs. 190.3 ± 56.9 min, P = 0.006), less intraoperative fluid input (1,158.5 ± 290.2 vs. 1,433.2 ± 653.3, P = 0.013), and less total intraoperative fluid output, including intraoperative blood loss, urine excretion, and other fluid loss. In addition, there were no statistically significant differences in intraoperative blood loss, 24 h pleural fluid volume, 48 h pleural fluid volume, postoperative chest tube duration, postoperative hospital stay and complications between the two groups of patients (P > 0.05).

Conclusions

In thoracoscopic segmentectomy, the application of 3D printing technology shortens the operation time, reduces intraoperative fluid input and output, guides the operation more safely and effectively, and has better clinical application value.

Artificial intelligence assisted display in thoracic surgery: development and possibilities

In this golden age of rapid development of artificial intelligence (AI), researchers and surgeons realized that AI could contribute to healthcare in all aspects, especially in surgery. The popularity of low-dose computed tomography (LDCT) and the improvement of the video-assisted thoracoscopic surgery (VATS) not only bring opportunities for thoracic surgery but also bring challenges on the way forward. Preoperatively localizing lung nodules precisely, intraoperatively identifying anatomical structures accurately, and avoiding complications requires a visual display of individuals’ specific anatomy for surgical simulation and assistance. With the advance of AI-assisted display technologies, including 3D reconstruction/3D printing, virtual reality (VR), augmented reality (AR), and mixed reality (MR), computer tomography (CT) imaging in thoracic surgery has been fully utilized for transforming 2D images to 3D model, which facilitates surgical teaching, planning, and simulation. AI-assisted display based on surgical videos is a new surgical application, which is still in its infancy. Notably, it has potential applications in thoracic surgery education, surgical quality evaluation, intraoperative assistance, and postoperative analysis. In this review, we illustrated the current AI-assisted display applications based on CT in thoracic surgery; focused on the emerging AI applications in thoracic surgery based on surgical videos by reviewing its relevant researches in other surgical fields and anticipate its potential development in thoracic surgery.

Augmented reality-assisted localization of solitary pulmonary nodules for precise sublobar lung resection: a preliminary study using an animal model

Background

Accurate localization of early lung cancer, manifested as solitary pulmonary nodules (SPNs) on computed tomography (CT), is critical in sublobar lung resection. The AR-assisted localization of SPNs was evaluated using a pig animal model.

Methods

A Microsoft HoloLens AR system was used. First, a plastic thoracic model was used for the pilot study. Three female 12 months 45 kg Danish Landrace Pigs were then used for the animal study. Thirty natural pulmonary structures, such as lymphonodus and bifurcated bronchioles or bronchial vessels, were chosen as simulated SPNs. The average angle between the actual puncturing needle and the expected path, the average distance between the puncture point and the plan point, and the difference between the actual puncturing depth and expected depth were recorded, and the accuracy rate was calculated.

Results

The point selected in the plastic thoracic model could be hit accurately with the assistance from the AR system in the pilot study. Moreover, the average angle between the actual puncturing needle and the expected path was 14.52°±6.04°. Meanwhile, the average distance between the puncture point and the expected point was 8.74±5.07 mm, and the difference between the actual and expected depths was 9.42±7.95 mm. Puncturing within a 1 cm³ area around the SPN using a hook-wire was considered a successful hit. The puncture accuracy was calculated. The average hit rate within a spherical area with a diameter of 1 cm range was 76.67%, and within a diameter of 2 cm range was 100%.

Conclusions

The HoloLens AR-assisted localization of SPNs may become a promising technique to improve the surgical treatment of early-stage lung cancer. Here, we evaluated its feasibility in an animal model. Nevertheless, its safety and effectiveness require further investigation in clinical trials.