Extended, virtual and augmented reality in thoracic surgery: a systematic review

Surgical planning in virtual reality: a systematic review

Purpose

Virtual reality (VR) technology has emerged as a promising tool for physicians, offering the ability to assess anatomical data in 3D with visuospatial interaction qualities. The last decade has witnessed a remarkable increase in the number of studies focusing on the application of VR to assess patient-specific image data. This systematic review aims to provide an up-to-date overview of the latest research on VR in the field of surgical planning.

Approach

A comprehensive literature search was conducted based on the preferred reporting items for systematic reviews and meta-analyses covering the period from April 1, 2021 to May 10, 2023. It includes research articles reporting on preoperative surgical planning using patient-specific medical images in virtual reality using head-mounted displays. The review summarizes the current state of research in this field, identifying key findings, technologies, study designs, methods, and potential directions for future research.

Results

The selected studies show a positive impact on surgical decision-making and anatomy understanding compared to other visualization modalities. A substantial number of studies are reporting anecdotal evidence and case-specific outcomes. Notably, surgical planning using VR led to more frequent changes in surgical plans compared to planning with other visualization methods when surgeons reassessed their initial plans. VR demonstrated benefits in reducing planning time and improving spatial localization of pathologies.

Conclusions

Results show that the application of VR for surgical planning is still in an experimental stage but is gradually advancing toward clinical use. The diverse study designs, methodologies, and varying reporting hinder a comprehensive analysis. Some findings lack statistical evidence and rely on subjective assumptions. To strengthen evaluation, future research should focus on refining study designs, improving technical reporting, defining visual and technical proficiency requirements, and enhancing VR software usability and design. Addressing these areas could pave the way for an effective implementation of VR in clinical settings.

Extended reality navigation for pain procedures: a narrative review

BACKGROUND

Extended reality (XR) technology, encompassing virtual reality, augmented reality, and mixed reality, has been widely studied for procedural navigation in surgical specialties. Similar to how ultrasound transformed regional anesthesia, XR has the potential to reshape how anesthesiologists and pain physicians perform procedures to relieve pain.

OBJECTIVE

This narrative review examines the clinical benefits of XR for navigation in various pain procedures. It defines key terms and concepts related to XR technology and explores characteristics of procedures that are most amenable to XR-based navigation. Finally, it suggests best practices for developing XR navigation systems and discusses the role of emerging technology in the future of XR in regional anesthesia and pain medicine.

EVIDENCE REVIEW

A search was performed across PubMed, Embase, and Cochrane Central Register of Controlled Trials for primary literature investigating the clinical benefits of XR navigation for pain procedures.

FINDINGS

Thirteen studies using XR for procedural navigation are included. The evidence includes randomized controlled trials, retrospective studies, and case series.

CONCLUSIONS

Early randomized controlled trials show potential for XR to improve procedural efficiency, but more comprehensive research is needed to determine if there are significant clinical benefits. Case reports demonstrate XR's utility in generating patient-specific navigation plans when difficult anatomy is encountered. Procedures that facilitate the generation and registration of XR images are most conducive to XR navigation, whereas those that rely on frequent re-imaging will continue to depend on traditional modes of navigation.

Mitral valve surgery assisted by virtual and augmented reality: Cardiac surgery at the front of innovation

BACKGROUND

Given the variety in mitral valve (MV) pathology and associated surgical techniques, extended reality (XR) holds great potential to assist MV surgeons. This review aims to systematically evaluate the currently available evidence investigating the use of XR and associated technologies in MV surgery.

METHODS

A systematic database search was conducted of original articles and case reports that explored the use of XR and MV surgery in EMBASE, MEDLINE, Cochrane database and Google Scholar, from inception to February 2022.

RESULTS

Our search yielded 171 articles, of which 15 studies were included in this review, featuring 328 patients. Two main areas of application were identified: (i) pre-operative planning and (ii) predicting post-operative outcomes. The articles reporting outcomes relating to pre-operative planning were further categorised as exploring themes relevant to (i) mitral annular assessment; (ii) training; (iii) evaluation of surgical technique; (iv) surgical approach or plan and (v) selecting ring size or type. Preoperatively, XR has been shown to evaluate mitral annular pathology more accurately than echocardiography, informing the surgeon about the optimal surgical technique, approach and plan for a particular patient's MV pathology. Furthermore, XR could simulate and aid ring size/type selection for MV annuloplasty, creating a personalized surgical plan. Additionally, XR could estimate the postoperative MV biomechanical and physiological characteristics, predicting and pre-empting post-operative complications.

CONCLUSION

XR demonstrated promising applications for assisting MV surgery, enhancing outcomes and patient-centred care, nevertheless, there remain the need for randomized studies to ascertain its feasibility, safety, and validity in clinical practice.

Cardiovascular Computed Tomography in the Diagnosis of Cardiovascular Disease: Beyond Lumen Assessment

Cardiovascular CT is being widely used in the diagnosis of cardiovascular disease due to the rapid technological advancements in CT scanning techniques. These advancements include the development of multi-slice CT, from early generation to the latest models, which has the capability of acquiring images with high spatial and temporal resolution. The recent emergence of photon-counting CT has further enhanced CT performance in clinical applications, providing improved spatial and contrast resolution. CT-derived fractional flow reserve is superior to standard CT-based anatomical assessment for the detection of lesion-specific myocardial ischemia. CT-derived 3D-printed patient-specific models are also superior to standard CT, offering advantages in terms of educational value, surgical planning, and the simulation of cardiovascular disease treatment, as well as enhancing doctor-patient communication. Three-dimensional visualization tools including virtual reality, augmented reality, and mixed reality are further advancing the clinical value of cardiovascular CT in cardiovascular disease. With the widespread use of artificial intelligence, machine learning, and deep learning in cardiovascular disease, the diagnostic performance of cardiovascular CT has significantly improved, with promising results being presented in terms of both disease diagnosis and prediction. This review article provides an overview of the applications of cardiovascular CT, covering its performance from the perspective of its diagnostic value based on traditional lumen assessment to the identification of vulnerable lesions for the prediction of disease outcomes with the use of these advanced technologies. The limitations and future prospects of these technologies are also discussed.

What's new in minimally invasive thoracic surgery? Clinical application of augmented reality and learning opportunities in surgical simulation

Lung cancer represents the most lethal cancer worldwide. Surgery is the treatment of choice for early-stage non-small cell lung cancer, with an overall survival that can reach 90% at 5 years, but its detection is difficult to achieve due to the lack of symptoms. Screening programs are crucial to identify small cancer. Minimally invasive surgery has modified the therapeutical approach of these tumors, becoming the standard of care, with an important clinical yield in terms of reduction of postoperative pain and length of hospital stay. The aim of this mini-review is to explore and describe two important and innovative aspects in the context of "growing opportunities in minimally invasive thoracic surgery": the clinical application of augmented reality and its advantages for patient and surgeon, and the pedagogical issue through simulation-based training.

ARIAS: An AR-based interactive advertising system

In this paper, we present an interactive advertising system based on augmented reality(AR) called ARIAS, which is manipulated with gestures for displaying advertising videos. Two-dimensional markers are defined in the system. The system captures the frame data through the camera in real time, uses OpenCV library to identify the predefined markers, and calculates the pose of markers captured by the camera. With OpenGL library, a virtual cubic model is created at the position of the marker, and thus videos or images are displayed on the six faces of the cube. The virtual cube, together with the original frame data collected by the camera, is displayed in the interactive window to achieve the augmented reality effect. Customers are accessible to various advertising content by observing the marker from different positions. The system, meanwhile, supports gesture operation in order to make the customers pay attention to the content they are interested in with one hand. The MediaPipe Hand framework is used to extract the landmarks of hands, based on which, a series of gestures are designed for interactive operation. The efficiency and accuracy of the system are tested and analyzed with the result, indicating that the system has high reliability and good interactiveness. This system is open at https://github.com/wanzhuxie/ARIAS/tree/PLOS-ONE.

Gaze behavior is related to objective technical skills assessment during virtual reality simulator-based surgical training: a proof of concept

PURPOSE

Simulation-based training allows surgical skills to be learned safely. Most virtual reality-based surgical simulators address technical skills without considering non-technical skills, such as gaze use. In this study, we investigated surgeons' visual behavior during virtual reality-based surgical training where visual guidance is provided. Our hypothesis was that the gaze distribution in the environment is correlated with the simulator's technical skills assessment.

METHODS

We recorded 25 surgical training sessions on an arthroscopic simulator. Trainees were equipped with a head-mounted eye-tracking device. A U-net was trained on two sessions to segment three simulator-specific areas of interest (AoI) and the background, to quantify gaze distribution. We tested whether the percentage of gazes in those areas was correlated with the simulator's scores.

RESULTS

The neural network was able to segment all AoI with a mean Intersection over Union superior to 94% for each area. The gaze percentage in the AoI differed among trainees. Despite several sources of data loss, we found significant correlations between gaze position and the simulator scores. For instance, trainees obtained better procedural scores when their gaze focused on the virtual assistance (Spearman correlation test, N = 7, r = 0.800, p = 0.031).

CONCLUSION

Our findings suggest that visual behavior should be quantified for assessing surgical expertise in simulation-based training environments, especially when visual guidance is provided. Ultimately visual behavior could be used to quantitatively assess surgeons' learning curve and expertise while training on VR simulators, in a way that complements existing metrics.

Simulation-based training in cardiac surgery: a systematic review

OBJECTIVES

The increase in the complexity of operations, the rising quest for improved outcomes and the scrutiny of surgical practice and its associated complications have led to a decreased educational value of in-patient surgical training within cardiac surgery. Simulation-based training has emerged as an adjunct to the apprenticeship model. In the following review, we aimed to evaluate the currently available evidence regarding simulation-based training in cardiac surgery.

METHODS

A systematic database search was conducted as per PRISMA guidelines, of original articles that explored the use of simulation-based training in adult cardiac surgery programs in EMBASE, MEDLINE, Cochrane database and Google Scholar, from inception to 2022. Data extraction covered the study characteristics, simulation modality, main methodology and main outcomes.

RESULTS

Our search yielded 341 articles, of which 28 studies were included in this review. Three main areas of focus were identified: (i) validity testing of the models; (ii) impact on surgeons' skills; and (iii) impact on clinical practice. Fouteen studies reported animal-based models and 14 reported on non-tissue-based models covering a wide spectrum of surgical operations. The results of the included studies suggest that validity assessment is scarce within the field, being carried out for only 4 of the models. Nonetheless, all studies reported improvement in trainees' confidence, clinical knowledge and surgical skills (including accuracy, speed, dexterity) of trainees both at senior and junior levels. The direct clinical impact included initiation of minimally invasive programmes and improved board exam pass rates, and creating positive behavioural changes to minimize further cardiovascular risk.

CONCLUSIONS

Surgical simulation has been shown to provide substantial benefits to trainees. Further evidence is needed to explore its direct impact on clinical practice.

Using mixed reality technique combines multimodal imaging signatures to adjuvant glioma photodynamic therapy

Background

Photodynamic therapy (PDT) promotes significant tumor regression and extends the lifetime of patients. The actual operation of PDT often relies on the subjective judgment of experienced neurosurgeons. Patients can benefit more from precisely targeting PDT's key operating zones.

Methods

We used magnetic resonance imaging scans and created 3D digital models of patient anatomy. Multiple images are aligned and merged in STL format. Neurosurgeons use HoloLens to import reconstructions and assist in PDT execution. Also, immunohistochemistry was used to explore the association of hyperperfusion sites in PDT of glioma with patient survival.

Results

We constructed satisfactory 3D visualization of glioma models and accurately localized the hyperperfused areas of the tumor. Tumor tissue taken in these areas was rich in CD31, VEGFA and EGFR that were associated with poor prognosis in glioma patients. We report the first study using MR technology combined with PDT in the treatment of glioma. Based on this model, neurosurgeons can focus PDT on the hyperperfused area of the glioma. A direct benefit was expected for the patients in this treatment.

Conclusion

Using the Mixed Reality technique combines multimodal imaging signatures to adjuvant glioma PDT can better exploit the vascular sealing effect of PDT on glioma.

Augmented reality-based affective training for improving care communication skill and empathy

It is important for caregivers of people with dementia (PwD) to have good patient communication skills as it has been known to reduce the behavioral and psychological symptoms of dementia (BPSD) of PwD as well as caregiver burnout. However, acquiring such skills often requires one-on-one affective training, which can be costly. In this study, we propose affective training using augmented reality (AR) for supporting the acquisition of such skills. The system uses see-through AR glasses and a nursing training doll to train the user in both practical nursing skills and affective skills such as eye contact and patient communication. The experiment was conducted with 38 nursing students. The participants were assigned to either the Doll group, which only used a doll for training, or the AR group, which used both a doll and the AR system. The results showed that eye contact significantly increased and the face-to-face distance and angle decreased in the AR group, while the Doll group had no significant difference. In addition, the empathy score of the AR group significantly increased after the training. Upon analyzing the correlation between personality and changes of physical skills, we found a significant positive correlation between the improvement rate of eye contact and extraversion in the AR group. These results demonstrated that affective training using AR is effective for improving caregivers' physical skills and their empathy for their patients. We believe that this system will be beneficial not only for dementia caregivers but for anyone looking to improve their general communication skills.

Exploring the Potential of Three-Dimensional Imaging, Printing, and Modeling in Pediatric Surgical Oncology: A New Era of Precision Surgery

Pediatric surgical oncology is a technically challenging field that relies on CT and MRI as the primary imaging tools for surgical planning. However, recent advances in 3D reconstructions, including Cinematic Rendering, Volume Rendering, 3D modeling, Virtual Reality, Augmented Reality, and 3D printing, are increasingly being used to plan complex cases bringing new insights into pediatric tumors to guide therapeutic decisions and prognosis in different pediatric surgical oncology areas and locations including thoracic, brain, urology, and abdominal surgery. Despite this, challenges to their adoption remain, especially in soft tissue-based specialties such as pediatric surgical oncology. This work explores the main innovative imaging reconstruction techniques, 3D modeling technologies (CAD, VR, AR), and 3D printing applications through the analysis of three real cases of the most common and surgically challenging pediatric tumors: abdominal neuroblastoma, thoracic inlet neuroblastoma, and a bilateral Wilms tumor candidate for nephron-sparing surgery. The results demonstrate that these new imaging and modeling techniques offer a promising alternative for planning complex pediatric oncological cases. A comprehensive analysis of the advantages and limitations of each technique has been carried out to assist in choosing the optimal approach.

Leading Transformation in Medical Education Through Extended Reality

Extended reality (XR) has exponentially developed over the past decades to incorporate technology whereby users can visualise, explore, and interact with 3-dimensional-generated computer environments, and superimpose virtual reality (VR) onto real-world environments, thus displaying information and data on various levels of the reality-virtuality continuum. In the context of medicine, VR tools allow for anatomical assessment and diagnosis, surgical training through lifelike procedural simulations, planning of surgeries and biopsies, intraprocedural guidance, and medical education. The following chapter aims to provide an overview of the currently available evidence and perspectives on the application of XR within medical education. It will focus on undergraduate and postgraduate teaching, medical education within Low-Middle Income Countries, key practical steps in implementing a successful XR programme, and the limitations and future of extended reality within medical education.

State of the Art and Future Prospects of Virtual and Augmented Reality in Veterinary Medicine: A Systematic Review

Virtual reality and augmented reality are new but rapidly expanding topics in medicine. In virtual reality, users are immersed in a three-dimensional environment, whereas in augmented reality, computer-generated images are superimposed on the real world. Despite advances in human medicine, the number of published articles in veterinary medicine is low. These cutting-edge technologies can be used in combination with existing methods in veterinary medicine to achieve diagnostic/therapeutic and educational goals. The purpose of our review was to evaluate studies for their use of virtual reality and augmented reality in veterinary medicine, as well as human medicine with animal trials, to report results and the state of the art. We collected all of the articles we included in our review by screening the Scopus, PubMed, and Web of Science databases. Of the 24 included studies, 11 and 13 articles belonged to virtual reality and augmented reality, respectively. Based on these articles, we determined that using these technologies has a positive impact on the scientific output of students and residents, can reduce training costs, and can be used in training/educational programs. Furthermore, using these tools can promote ethical standards. We reported the absence of standard operation protocols and equipment costs as study limitations.

Congenital Heart Surgery Skill Training Using Simulation Models: Not an Option but a Necessity

Congenital heart surgery (CHS) is technically demanding, and its training is extremely complex and challenging. Training of the surgeon's technical skills has relied on a preceptorship format in which the trainees are gradually exposed to patients in the operating room under the close tutelage of senior staff surgeons. Training in the operating room is an inefficient process and the concept of a learning curve is no longer acceptable in terms of patient outcomes. The benefits of surgical simulation in training of congenital heart surgeons are well known and appreciated. However, adequate surgical simulation models and equipment for training have been scarce until the recent development of three-dimensionally (3D) printed models. Using comprehensive 3D printing and silicone-molding techniques, realistic simulation training models for most congenital heart surgical procedures have been produced. Newly developed silicone-molded models allow efficient CHS training in a stress-free environment with instantaneous feedback from the proctors and avoids risk to patients. The time has arrived when all congenital heart surgeons should consider surgical simulation training before progressing to real-life operating in a similar fashion to the aviation industry where all pilots are required to complete simulation training before flying a real aircraft. It is argued here that simulation training is not an option anymore but should be a mandatory component of CHS training.

Mixed Reality in der Gefäßchirurgie – ein Scoping Review

Hintergrund „Mixed Reality“ (MR) erlaubt die Projektion von virtuellen Objekten in das Sichtfeld des Anwenders durch ein Head-mounted Display (HMD). Im gefäßchirurgischen Behandlungsspektrum könnten MR-Anwendungen in Zukunft einen Nutzen darstellen. Im folgenden Scoping Review soll eine Orientierung über die aktuelle Anwendung der genannten Technologien im Bereich der Gefäßchirurgie gegeben und Forschungsziele für die Zukunft definiert werden. Material und Methoden Es erfolgte eine systematische Literaturrecherche in PubMed (MEDLINE) mit den Suchbegriffen „aorta“, „intervention“, „endovsacular intervention“, „vascular surgery“, „aneurysm“, „endovascular“, „vascular access“ jeweils in Kombination mit „mixed reality“ oder „augmented reality“. Die Suche erfolgte nach PRISMA-Leitlinie (Preferred Reporting Items for Systematic reviews and Meta-Analyses) für Scoping Reviews. Ergebnisse Aus 547 Literaturstellen konnten 8 relevante Studien identifiziert werden. Die Suchergebnisse konnten in 2 Anwendungskategorien eingeteilt werden: (1) MR mit dem Ziel des Informationsmanagements und zur Verbesserung der periprozeduralen Ergonomie gefäßchirurgischer Eingriffe (n = 3) sowie (2) MR mit dem Ziel der intraoperativen Navigation bei gefäßchirurgischen Eingriffen (n = 5). Die Registrierung des physischen Patienten mit dem virtuellen Objekt und das Tracking von Instrumenten in der MR-Umgebung zur intraoperativen Navigation ist dabei im Fokus des wissenschaftlichen Interesses und konnte technisch erfolgreich am Phantom- und Tiermodell gezeigt werden. Die bisher vorgestellten Methoden sind jedoch mit hohem infrastrukturellem Aufwand und relevanten Limitationen verbunden. Schlussfolgerung Der Einsatz von MR im Bereich der Gefäßchirurgie ist grundsätzlich vielversprechend. Für die Zukunft sollten alternative, pragmatische Registrierungsmethoden mit entsprechender Quantifizierung des Positionierungsfehlers angestrebt werden. Die entwickelten Soft- und Hardwarelösungen sollten auf das Anforderungsprofil der Gefäßchirurgie angepasst werden. Das elektromagnetische Instrumenten-Tracking erscheint als sinnvolle, komplementäre Technologie zur Umsetzung der MR-assistierten Navigation.

Immersive Three-dimensional Computed Tomography to Plan Chest Wall Resection for Lung Cancer

PURPOSE

Chest wall resections for lung cancer treatment remain difficult to plan using standard two-dimensional computed tomography. Although virtual reality headsets have been used in many medical contexts, they have not been employed in chest wall resection planning.

DESCRIPTION

We compared preoperative planning of a chest wall surgical resection for lung cancer treatment between senior and resident surgeons who used an immersive virtual reality device and a two-dimensional computed tomography.

EVALUATION

Chest wall resection planning was more accurate when surgeons used virtual reality versus computed tomography analysis (28.6% vs. 18.3%, p = 0.018), and this was particularly true in the resident surgeon group (27.4% vs. 8.3%, p = 0.0025). Predictions regarding the need for chest wall substitutes were also more accurate when they were made using virtual reality versus computed tomography analysis in all groups (96% vs. 68.5%, p < 0.0001). Other studied parameters were not affected by use of the virtual reality tool.

CONCLUSION

Virtual reality may offer enhanced accuracy for chest wall resection and reconstruction planning for lung cancer treatment.

What is Significant in Modern Augmented Reality: A Systematic Analysis of Existing Reviews

Augmented reality (AR) is a field of technology that has evolved drastically during the last decades, due to its vast range of applications in everyday life. The aim of this paper is to provide researchers with an overview of what has been surveyed since 2010 in terms of AR application areas as well as in terms of its technical aspects, and to discuss the extent to which both application areas and technical aspects have been covered, as well as to examine whether one can extract useful evidence of what aspects have not been covered adequately and whether it is possible to define common taxonomy criteria for performing AR reviews in the future. To this end, a search with inclusion and exclusion criteria has been performed in the Scopus database, producing a representative set of 47 reviews, covering the years from 2010 onwards. A proper taxonomy of the results is introduced, and the findings reveal, among others, the lack of AR application reviews covering all suggested criteria.