Intraoperative holography navigation using a mixed-reality wearable computer during laparoscopic cholecystectomy

The efficacy of 3D hologram support with mixed-reality technique in pancreatobiliary endoscopy

BACKGROUND AND AIMS

Mixed-reality (MR) technology is an advanced holographic imaging system that uses wearable devices to display three-dimensional (3D) images in clinical environments. In pancreatobiliary endoscopy, MR technology can facilitate 3D visualization of the bile and pancreatic ducts, enhancing spatial awareness and providing a more intuitive understanding of the anatomy. We evaluated the safety and efficacy of 3D hologram support (3D-HS) using MR in pancreatobiliary endoscopy.

METHODS

This study included 30 patients who underwent pancreatobiliary endoscopy using 3D-HS between June 2023 and July 2024. The procedures included ERCP (n = 13) and interventional EUS (iEUS) (n = 17). The primary outcome was technical success, with secondary outcomes including total procedure-related time and adverse events. The conventional treatment group, matched for age and disease type, was compared to the 3D-HS group. a questionnaire was used to evaluate the MR device, 3D hologram images, and the overall evaluation with 3D-HS.

RESULTS

The overall technical success rate was 96.7% (ERCP, 100%; iEUS, 93.8%). Adverse events occurred in 16.7% of cases; all were managed conservatively. comparison with conventional treatment showed no significant differences in outcomes, although conventional methods had shorter procedural times (p < 0.05). The questionnaire results indicated positive feedback on operability and image quality of 3D-HS, with enhanced anatomical understanding relative to 2D imaging, although some ERCP cases received lower ratings, particularly for intrahepatic bile duct stones.

CONCLUSIONS

Pancreatobiliary endoscopy using 3D-HS can be safe and effective in enhancing anatomical understanding. Further studies are needed to optimize its use and reduce the total procedure time.

Transforming neonatal care: a position paper on the potential of augmented and mixed reality

Mixed reality (MR) and augmented reality (AR) technologies bridge elements of the real and virtual worlds, emerging as tools that allow users to engage with digital cues to aid with tasks encountered in the physical environment. Thus, these holographic-based innovations are potential tools to support real-time patient care. The applications of MR and AR in neonatal care remain significantly underexplored. In the present article, we highlight the applications of MR and AR across medical procedures, physical examinations, medical diagnoses, and telemedicine, further underscoring their transformative potential within neonatal care. The use of MR and AR can be relevant across diverse economic and clinical landscapes, and in-depth research is required to evaluate the advantages of these tools in caring for neonates.

Utilization of apple vision pro in ophthalmic surgery: A pilot study

PURPOSE

This paper explores the application of Apple Vision Pro in ophthalmic surgery, assessing its potential benefits in providing real-time imaging overlay, surgical guidance, and collaborative opportunities.

MATERIALS AND METHOD

The device was worn by 10 ophthalmic surgeons during eyelid malposition surgery. All surgeons performed the entire surgery while wearing the visor. At the end of procedure, all operators had to rate Apple Vision Pro visor according to 10 specific item and system usability scale (SUS) questionnaire.

RESULTS

The surgeons used the Apple Vision Pro during the entire procedure, and the results were positive, with high ratings for practicality, freedom of movement, integration into workflow, and learning. All surgeons rated the Apple Vision Pro above 85/100 in the SUS.

CONCLUSION

The incorporation of Apple Vision Pro in oculoplastic surgery offers several advantages, including improved visualization, enhanced precision, and streamlined communication among surgical teams. According to our preliminary results Apple Vision Pro could represents a valuable tool in ophthalmic surgery, with implications for enhancing surgical techniques and advancing XR research in the surgical field.

Enhancing Sentinel Lymph Node Biopsy in Endometrial Cancer Using Augmented and Mixed Reality

Sentinel lymph node biopsy offers a less invasive alternative to full lymphadenectomy in endometrial cancer, reducing complications while maintaining diagnostic accuracy. This case report highlights the integration of holography and augmented reality (AR) in a 34-year-old woman undergoing robotic-assisted surgery for endometrial cancer. Preoperative imaging combined with single-photon emission computed tomography and computed tomography identified sentinel lymph nodes, which were visualized using mixed reality (MR) technology. This approach enabled the surgical team to accurately understand the three-dimensional spatial relationships between lymph nodes and surrounding critical structures. Holographic projections provided precise guidance during surgery, improving lymph node identification and minimizing invasiveness. No lymph node metastasis was detected, but a diagnosis of the International Federation of Gynecology and Obstetrics (FIGO) stage IIIA was confirmed due to tumor seeding in the peritoneum. The patient underwent successful adjuvant chemotherapy with no recurrence observed. This report demonstrates the significant potential of holography and AR to enhance spatial awareness and surgical precision. These technologies represent a promising advancement in sentinel lymph node biopsy for patients with gynecologic cancers, contributing to reduced surgical invasiveness and alleviating stress for surgeons.

The Depth Estimation and Visualization of Dermatological Lesions: Development and Usability Study

BACKGROUND

Thus far, considerable research has been focused on classifying a lesion as benign or malignant. However, there is a requirement for quick depth estimation of a lesion for the accurate clinical staging of the lesion. The lesion could be malignant and quickly grow beneath the skin. While biopsy slides provide clear information on lesion depth, it is an emerging domain to find quick and noninvasive methods to estimate depth, particularly based on 2D images.

OBJECTIVE

This study proposes a novel methodology for the depth estimation and visualization of skin lesions. Current diagnostic methods are approximate in determining how much a lesion may have proliferated within the skin. Using color gradients and depth maps, this method will give us a definite estimate and visualization procedure for lesions and other skin issues. We aim to generate 3D holograms of the lesion depth such that dermatologists can better diagnose melanoma.

METHODS

We started by performing classification using a convolutional neural network (CNN), followed by using explainable artificial intelligence to localize the image features responsible for the CNN output. We used the gradient class activation map approach to perform localization of the lesion from the rest of the image. We applied computer graphics for depth estimation and developing the 3D structure of the lesion. We used the depth from defocus method for depth estimation from single images and Gabor filters for volumetric representation of the depth map. Our novel method, called red spot analysis, measures the degree of infection based on how a conical hologram is constructed. We collaborated with a dermatologist to analyze the 3D hologram output and received feedback on how this method can be introduced to clinical implementation.

RESULTS

The neural model plus the explainable artificial intelligence algorithm achieved an accuracy of 86% in classifying the lesions correctly as benign or malignant. For the entire pipeline, we mapped the benign and malignant cases to their conical representations. We received exceedingly positive feedback while pitching this idea at the King Edward Memorial Institute in India. Dermatologists considered this a potentially useful tool in the depth estimation of lesions. We received a number of ideas for evaluating the technique before it can be introduced to the clinical scene.

CONCLUSIONS

When we map the CNN outputs (benign or malignant) to the corresponding hologram, we observe that a malignant lesion has a higher concentration of red spots (infection) in the upper and deeper portions of the skin, and that the malignant cases have deeper conical sections when compared with the benign cases. This proves that the qualitative results map with the initial classification performed by the neural model. The positive feedback provided by the dermatologist suggests that the qualitative conclusion of the method is sufficient.

Integration of Augmented Reality in Temporal Bone and Skull Base Surgeries

Augmented reality technologies provide transformative solutions in various surgical fields. Our research focuses on the use of an advanced augmented reality system that projects 3D holographic images directly into surgical footage, potentially improving the surgeon's orientation to the surgical field and lowering the cognitive load. We created a novel system that combines exoscopic surgical footage from the "ORBEYE" and displays both the surgical field and 3D holograms on a single screen. This setup enables surgeons to use the system without using head-mounted displays, instead viewing the integrated images on a 3D monitor. Thirteen surgeons and surgical assistants completed tasks with 2D and 3D graphical surgical guides. The NASA Task Load Index was used to assess mental, physical, and temporal demands. The use of 3D graphical surgical guides significantly improved performance metrics in cochlear implant surgeries by lowering mental, physical, temporal, and frustration levels. However, for Bonebridge implantation, the 2D graphical surgical guide performed better overall (p = 0.045). Participants found the augmented reality system's video latency to be imperceptible, measuring 0.13 ± 0.01 s. This advanced augmented reality system significantly improves the efficiency and precision of cochlear implant surgeries by lowering cognitive load and improving spatial orientation.

Leveraging Space-Flown Technologies to Deliver Healthcare with Holographic Physical Examinations

INTRODUCTION: Musculoskeletal injuries are one of the more common injuries in spaceflight. Physical assessment of an injury is essential for diagnosis and treatment. Unfortunately, when musculoskeletal injuries occur in space, the flight surgeon is limited to two-dimensional videoconferencing and, potentially, observations made by the crew medical officer. To address these limitations, we investigated the feasibility of performing physical examinations on a three-dimensional augmented reality projection using a mixed-reality headset, specifically evaluating a standard shoulder examination.METHODS: A simulated patient interaction was set up between Western University in London, Ontario, Canada, and Huntsville, AL, United States. The exam was performed by a medical student, and a healthy adult man volunteered to enable the physical exam.RESULTS: All parts of the standard shoulder physical examination according to the Bates Guide to the Physical Exam were performed with holoportation. Adaptation was required for the palpation and some special tests.DISCUSSION: All parts of the physical exam were able to be completed. The true to anatomical size of the holograms permitted improved inspection of the anatomy compared to traditional videoconferencing. Palpation was completed by instructing the patient to palpate themselves and comment on relevant findings asked by the examiner. Range of motion and special tests for specific pathologies were also able to be completed with some modifications due to the examiner not being present to provide resistance. Future work should aim to improve the graphics, physician communication, and haptic feedback during holoportation.Levschuk A, Whittal J, Trejos AL, Sirek A. Leveraging space-flown technologies to deliver healthcare with holographic physical examinations. Aerosp Med Hum Perform. 2024; 95(4):214-218.

Comparing Different Registration and Visualization Methods for Navigated Common Femoral Arterial Access-A Phantom Model Study Using Mixed Reality

Mixed reality (MxR) enables the projection of virtual three-dimensional objects into the user's field of view via a head-mounted display (HMD). This phantom model study investigated three different workflows for navigated common femoral arterial (CFA) access and compared it to a conventional sonography-guided technique as a control. A total of 160 punctures were performed by 10 operators (5 experts and 5 non-experts). A successful CFA puncture was defined as puncture at the mid-level of the femoral head with the needle tip at the central lumen line in a 0° coronary insertion angle and a 45° sagittal insertion angle. Positional errors were quantified using cone-beam computed tomography following each attempt. Mixed effect modeling revealed that the distance from the needle entry site to the mid-level of the femoral head is significantly shorter for navigated techniques than for the control group. This highlights that three-dimensional visualization could increase the safety of CFA access. However, the navigated workflows are infrastructurally complex with limited usability and are associated with relevant cost. While navigated techniques appear as a potentially beneficial adjunct for safe CFA access, future developments should aim to reduce workflow complexity, avoid optical tracking systems, and offer more pragmatic methods of registration and instrument tracking.

Application of Extended Reality (Virtual Reality and Mixed Reality) Technology in Laparoscopic Liver Resections

Background and purpose Laparoscopic liver resection (LLR) has recently gained popularity owing to advances in surgical techniques. Difficulties in LLR may be influenced by anatomical factors. This study presents a comprehensive overview of LLR performed using extended reality (XR) technology. Methods Six patients underwent LLR performed wearing HoloLens2® XR (Microsoft Corporation, Redmond, Washington, United States) technology. We performed dynamic contrast-enhanced CT scans before surgery and used the data to construct three-dimensional images. Results Of the six patients, two were diagnosed with colorectal liver metastases, two with hepatocellular carcinoma, and one with intrahepatic cholangiocarcinoma. The median maximum tumor diameter was 31 mm (range, 23-80 mm). One patient had liver cirrhosis, with Child-Pugh classification grade B. Anatomical resection was performed in three patients (60%), with a median difficulty score of 7 (intermediate). No conversions to open surgery were necessary. The median operative time and estimated blood loss were 444 minutes (range, 337-597 minutes) and 200 mL (range, 100-1000 mL), respectively. Postoperative complications (Clavien-Dindo classification grade II) were observed in one patient. All six cases achieved negative surgical margins. Conclusions LLR using XR technology enhances surgical visualization and anatomical recognition. The incorporation of XR technology into LLR offers advantages over traditional two-dimensional imaging.

Mixed Reality in Modern Surgical and Interventional Practice: Narrative Review of the Literature

BACKGROUND

Mixed reality (MR) and its potential applications have gained increasing interest within the medical community over the recent years. The ability to integrate virtual objects into a real-world environment within a single video-see-through display is a topic that sparks imagination. Given these characteristics, MR could facilitate preoperative and preinterventional planning, provide intraoperative and intrainterventional guidance, and aid in education and training, thereby improving the skills and merits of surgeons and residents alike.

OBJECTIVE

In this narrative review, we provide a broad overview of the different applications of MR within the entire spectrum of surgical and interventional practice and elucidate on potential future directions.

METHODS

A targeted literature search within the PubMed, Embase, and Cochrane databases was performed regarding the application of MR within surgical and interventional practice. Studies were included if they met the criteria for technological readiness level 5, and as such, had to be validated in a relevant environment.

RESULTS

A total of 57 studies were included and divided into studies regarding preoperative and interventional planning, intraoperative and interventional guidance, as well as training and education.

CONCLUSIONS

The overall experience with MR is positive. The main benefits of MR seem to be related to improved efficiency. Limitations primarily seem to be related to constraints associated with head-mounted display. Future directions should be aimed at improving head-mounted display technology as well as incorporation of MR within surgical microscopes, robots, and design of trials to prove superiority.

Microsoft HoloLens 2 in Medical and Healthcare Context: State of the Art and Future Prospects

In the world reference context, although virtual reality, augmented reality and mixed reality have been emerging methodologies for several years, only today technological and scientific advances have made them suitable to revolutionize clinical care and medical contexts through the provision of enhanced functionalities and improved health services. This systematic review provides the state-of-the-art applications of the Microsoft^® HoloLens 2 in a medical and healthcare context. Focusing on the potential that this technology has in providing digitally supported clinical care, also but not only in relation to the COVID-19 pandemic, studies that proved the applicability and feasibility of HoloLens 2 in a medical and healthcare scenario were considered. The review presents a thorough examination of the different studies conducted since 2019, focusing on HoloLens 2 medical sub-field applications, device functionalities provided to users, software/platform/framework used, as well as the study validation. The results provided in this paper could highlight the potential and limitations of the HoloLens 2-based innovative solutions and bring focus to emerging research topics, such as telemedicine, remote control and motor rehabilitation.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

We conducted a systematic review of recent literature to understand the current challenges in the use of optical see-through head-mounted displays (OST-HMDs) for augmented reality (AR) assisted surgery. Using Google Scholar, 57 relevant articles from 1 January 2021 through 18 March 2022 were identified. Selected articles were then categorized based on a taxonomy that described the required components of an effective AR-based navigation system: data, processing, overlay, view, and validation. Our findings indicated a focus on orthopedic (n=20) and maxillofacial surgeries (n=8). For preoperative input data, computed tomography (CT) (n=34), and surface rendered models (n=39) were most commonly used to represent image information. Virtual content was commonly directly superimposed with the target site (n=47); this was achieved by surface tracking of fiducials (n=30), external tracking (n=16), or manual placement (n=11). Microsoft HoloLens devices (n=24 in 2021, n=7 in 2022) were the most frequently used OST-HMDs; gestures and/or voice (n=32) served as the preferred interaction paradigm. Though promising system accuracy in the order of 2–5 mm has been demonstrated in phantom models, several human factors and technical challenges—perception, ease of use, context, interaction, and occlusion—remain to be addressed prior to widespread adoption of OST-HMD led surgical navigation.

Digital Transformation Will Change Medical Education and Rehabilitation in Spine Surgery

The concept of minimally invasive spine therapy (MIST) has been proposed as a treatment strategy to reduce the need for overall patient care, including not only minimally invasive spine surgery (MISS) but also conservative treatment and rehabilitation. To maximize the effectiveness of patient care in spine surgery, the educational needs of medical students, residents, and patient rehabilitation can be enhanced by digital transformation (DX), including virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR), three-dimensional (3D) medical images and holograms; wearable sensors, high-performance video cameras, fifth-generation wireless system (5G) and wireless fidelity (Wi-Fi), artificial intelligence, and head-mounted displays (HMDs). Furthermore, to comply with the guidelines for social distancing due to the unexpected COVID-19 pandemic, the use of DX to maintain healthcare and education is becoming more innovative than ever before. In medical education, with the evolution of science and technology, it has become mandatory to provide a highly interactive educational environment and experience using DX technology for residents and medical students, known as digital natives. This study describes an approach to pre- and intraoperative medical education and postoperative rehabilitation using DX in the field of spine surgery that was implemented during the COVID-19 pandemic and will be utilized thereafter.

Mixed-Reality-Assisted Puncture of the Common Femoral Artery in a Phantom Model

Percutaneous femoral arterial access is daily practice in a variety of medical specialties and enables physicians worldwide to perform endovascular interventions. The reported incidence of percutaneous femoral arterial access complications is 3–18% and often results from suboptimal puncture location due to insufficient visualization of the target vessel. The purpose of this proof-of-concept study was to evaluate the feasibility and the positional error of a mixed-reality (MR)-assisted puncture of the common femoral artery in a phantom model using a commercially available navigation system. In total, 15 MR-assisted punctures were performed. Cone-beam computed tomography angiography (CTA) was used following each puncture to allow quantification of positional error of needle placements in the axial and sagittal planes. Technical success was achieved in 14/15 cases (93.3%) with a median axial positional error of 1.0 mm (IQR 1.3) and a median sagittal positional error of 1.1 mm (IQR 1.6). The median duration of the registration process and needle insertion was 2 min (IQR 1.0). MR-assisted puncture of the common femoral artery is feasible with acceptable positional errors in a phantom model. Future studies should aim to measure and reduce the positional error resulting from MR registration.