Development of a Mixed Reality Platform for Lateral Skull Base Anatomy

Augmented Reality-Guided Mastoidectomy Simulation: A Randomized Controlled Trial Assessing Surgical Proficiency

OBJECTIVE

Mastoidectomy surgical training is challenging due to the complex nature of the anatomical structures involved. Traditional training methods based on direct patient care and cadaveric temporal bone training have practical shortcomings. 3D-printed temporal bone models and augmented reality (AR) have emerged as promising solutions, particularly for mastoidectomy surgery, which demands an understanding of intricate anatomical structures. Evidence is needed to explore the potential of AR technology in addressing these training challenges.

METHODS

21 medical students in their clinical clerkship were recruited for this prospective, randomized controlled trial assessing mastoidectomy skills. The participants were randomly assigned to the AR group, which received real-time guidance during drilling on 3D-printed temporal bone models, or to the control group, which received traditional training methods. Skills were assessed on a modified Welling scale and evaluated independently by two senior otologists.

RESULTS

The AR group outperformed the control group, with a mean overall drilling score of 19.5 out of 25, compared with the control group's score of 12 (p < 0.01). The AR group was significantly better at defining mastoidectomy margins (p < 0.01), exposing the antrum, preserving the lateral semicircular canal (p < 0.05), sharpening the sinodural angle (p < 0.01), exposing the tegmen and attic, preserving the ossicles (p < 0.01), and thinning and preserving the external auditory canal (p < 0.05).

CONCLUSION

AR simulation in mastoidectomy, even in a single session, improved the proficiency of novice surgeons compared with traditional methods.

LEVEL OF EVIDENCE

NA Laryngoscope, 135:894-900, 2025.

Targeting accuracy of neuronavigation: a comparative evaluation of an innovative wearable AR platform vs. traditional EM navigation

Wearable augmented reality in neurosurgery offers significant advantages by enabling the visualization of navigation information directly on the patient, seamlessly integrating virtual data with the real surgical field. This ergonomic approach can facilitate a more intuitive understanding of spatial relationships and guidance cues, potentially reducing cognitive load and enhancing the accuracy of surgical gestures by aligning critical information with the actual anatomy in real-time. This study evaluates the benefits of a novel AR platform, VOSTARS, by comparing its targeting accuracy to that of the gold-standard electromagnetic (EM) navigation system, Medtronic StealthStation® S7®. Both systems were evaluated in phantom and human studies. In the phantom study, participants targeted 13 predefined landmarks using identical pointers to isolate system performance. In the human study, three facial landmarks were targeted in nine volunteers post-brain tumor surgery. The performance of the VOSTARS system was superior to that of the standard neuronavigator in both the phantom and human studies. In the phantom study, users achieved a median accuracy of 1.4 mm (IQR: 1.2 mm) with VOSTARS compared to 2.9 mm (IQR: 1.4 mm) with the standard neuronavigator. In the human study, the median targeting accuracy with VOSTARS was significantly better for selected landmarks in the outer eyebrow (3.7 mm vs. 6.6 mm, p = 0.05) and forehead (4.5 mm vs. 6.3 mm, p = 0.021). Although the difference for the pronasal point was not statistically significant (2.7 mm vs. 3.5 mm, p = 0.123), the trend towards improved accuracy with VOSTARS is clear. These findings suggest that the proposed AR technology has the potential to significantly improve surgical outcomes in neurosurgery.

Mixed reality-assisted versus landmark-guided spinal puncture in elderly patients: protocol for a stratified randomized controlled trial

BACKGROUND

Performing spinal anesthesia in elderly patients with spine degeneration is challenging for novice practitioners. This stratified randomized controlled trial aims to compare the effectiveness of mixed reality-assisted spinal puncture (MRasp) with that of landmark-guided spinal puncture (LGsp) performed by novice practitioners in elderly patients.

METHODS

This prospective, single-center, stratified, blocked, parallel randomized controlled trial will include 168 patients (aged ≥ 65 years) scheduled for elective surgery involving spinal anesthesia. All spinal punctures will be performed by anesthesiology interns and residents trained at Huadong Hospital. Patients will be randomly assigned to the MRasp group (n = 84) or the LGsp group (n = 84). Based on each intern/resident's experience in spinal puncture, participants will be stratified into three clusters: the primary group, intermediate group, and advanced group. The primary outcome will be the comparison of the rate of successful first-attempt needle insertion between the MRasp group and the LGsp group. Secondary outcomes will include the number of needle insertion attempts, the number of redirection attempts, the number of passes, the rate of successful first needle pass, the spinal puncture time, the total procedure time, and the incidence of perioperative complications. A stratified subgroup analysis will also be conducted for interns/residents at different experience levels.

DISCUSSION

The findings from this trial establish the effectiveness of MRasp by novice practitioners in elderly patients. This trial may provide experimental evidence for exploring an effective visualization technology to assist in spinal puncture.

TRIAL REGISTRATION

Chinese Clinical Trials Registry ChiCTR2300075291. Registered on August 31, 2023. https://www.chictr.org.cn/bin/project/edit?pid=189622 .

Narrative review of patient-specific 3D visualization and reality technologies in skull base neurosurgery: enhancements in surgical training, planning, and navigation

Recent advances in medical imaging, computer vision, 3-dimensional (3D) modeling, and artificial intelligence (AI) integrated technologies paved the way for generating patient-specific, realistic 3D visualization of pathological anatomy in neurosurgical conditions. Immersive surgical simulations through augmented reality (AR), virtual reality (VR), mixed reality (MxR), extended reality (XR), and 3D printing applications further increased their utilization in current surgical practice and training. This narrative review investigates state-of-the-art studies, the limitations of these technologies, and future directions for them in the field of skull base surgery. We begin with a methodology summary to create accurate 3D models customized for each patient by combining several imaging modalities. Then, we explore how these models are employed in surgical planning simulations and real-time navigation systems in surgical procedures involving the anterior, middle, and posterior cranial skull bases, including endoscopic and open microsurgical operations. We also evaluate their influence on surgical decision-making, performance, and education. Accumulating evidence demonstrates that these technologies can enhance the visibility of the neuroanatomical structures situated at the cranial base and assist surgeons in preoperative planning and intraoperative navigation, thus showing great potential to improve surgical results and reduce complications. Maximum effectiveness can be achieved in approach selection, patient positioning, craniotomy placement, anti-target avoidance, and comprehension of spatial interrelationships of neurovascular structures. Finally, we present the obstacles and possible future paths for the broader implementation of these groundbreaking methods in neurosurgery, highlighting the importance of ongoing technological advancements and interdisciplinary collaboration to improve the accuracy and usefulness of 3D visualization and reality technologies in skull base surgeries.

Enhancing Surgical Vision: Augmented Reality in Otolaryngology-Head and Neck Surgery

Augmented reality (AR) technology has become widely established in otolaryngology-head and neck surgery. Over the past 20 years, numerous AR systems have been investigated and validated across the subspecialties, both in cadaveric and in live surgical studies. AR displays projected through head-mounted devices, microscopes, and endoscopes, most commonly, have demonstrated utility in preoperative planning, intraoperative guidance, and improvement of surgical decision-making. Specifically, they have demonstrated feasibility in guiding tumor margin resections, identifying critical structures intraoperatively, and displaying patient-specific virtual models derived from preoperative imaging, with millimetric accuracy. This review summarizes both established and emerging AR technologies, detailing how their systems work, what features they offer, and their clinical impact across otolaryngology subspecialties. As AR technology continues to advance, its integration holds promise for enhancing surgical precision, simulation training, and ultimately, improving patient outcomes.

Head model dataset for mixed reality navigation in neurosurgical interventions for intracranial lesions

Mixed reality navigation (MRN) technology is emerging as an increasingly significant and interesting topic in neurosurgery. MRN enables neurosurgeons to "see through" the head with an interactive, hybrid visualization environment that merges virtual- and physical-world elements. Offering immersive, intuitive, and reliable guidance for preoperative and intraoperative intervention of intracranial lesions, MRN showcases its potential as an economically efficient and user-friendly alternative to standard neuronavigation systems. However, the clinical research and development of MRN systems present challenges: recruiting a sufficient number of patients within a limited timeframe is difficult, and acquiring low-cost, commercially available, medically significant head phantoms is equally challenging. To accelerate the development of novel MRN systems and surmount these obstacles, the study presents a dataset designed for MRN system development and testing in neurosurgery. It includes CT and MRI data from 19 patients with intracranial lesions and derived 3D models of anatomical structures and validation references. The models are available in Wavefront object (OBJ) and Stereolithography (STL) formats, supporting the creation and assessment of neurosurgical MRN applications.

Inner Ear Breaches from Vestibular Schwannoma Surgery: Revisiting the Incidence of Otologic Injury from Retrosigmoid and Middle Cranial Fossa Approaches

OBJECTIVE

To assess the rate of iatrogenic injury to the inner ear in vestibular schwannoma resections.

STUDY DESIGN

Retrospective case review.

SETTING

Multiple academic tertiary care hospitals.

PATIENTS

Patients who underwent retrosigmoid or middle cranial fossa approaches for vestibular schwannoma resection between 1993 and 2015.

INTERVENTION

Diagnostic with therapeutic implications.

MAIN OUTCOME MEASURE

Drilling breach of the inner ear as confirmed by operative note or postoperative computed tomography (CT).

RESULTS

21.5% of patients undergoing either retrosigmoid or middle fossa approaches to the internal auditory canal were identified with a breach of the vestibulocochlear system. Because of the lack of postoperative CT imaging in this cohort, this is likely an underestimation of the true incidence of inner ear breaches. Of all postoperative CT scans reviewed, 51.8% had an inner ear breach. As there may be bias in patients undergoing postoperative CT, a middle figure based on sensitivity analyses estimates the incidence of inner ear breaches from lateral skull base surgery to be 34.7%.

CONCLUSIONS

A high percentage of vestibular schwannoma surgeries via retrosigmoid and middle cranial fossa approaches result in drilling breaches of the inner ear. This study reinforces the value of preoperative image analysis for determining risk of inner ear breaches during vestibular schwannoma surgery and the importance of acquiring CT studies postoperatively to evaluate the integrity of the inner ear.

Augmented Reality Integration in Skull Base Neurosurgery: A Systematic Review

Background and Objectives: To investigate the role of augmented reality (AR) in skull base (SB) neurosurgery. Materials and Methods: Utilizing PRISMA methodology, PubMed and Scopus databases were explored to extract data related to AR integration in SB surgery. Results: The majority of 19 included studies (42.1%) were conducted in the United States, with a focus on the last five years (77.8%). Categorization included phantom skull models (31.2%, n = 6), human cadavers (15.8%, n = 3), or human patients (52.6%, n = 10). Microscopic surgery was the predominant modality in 10 studies (52.6%). Of the 19 studies, surgical modality was specified in 18, with microscopic surgery being predominant (52.6%). Most studies used only CT as the data source (n = 9; 47.4%), and optical tracking was the prevalent tracking modality (n = 9; 47.3%). The Target Registration Error (TRE) spanned from 0.55 to 10.62 mm. Conclusion: Despite variations in Target Registration Error (TRE) values, the studies highlighted successful outcomes and minimal complications. Challenges, such as device practicality and data security, were acknowledged, but the application of low-cost AR devices suggests broader feasibility.

Mixed Reality Surgical Navigation System; Positional Accuracy Based on Food and Drug Administration Standard

BACKGROUND

Computer assisted surgical navigation systems are designed to improve outcomes by providing clinicians with procedural guidance information. The use of new technologies, such as mixed reality, offers the potential for more intuitive, efficient, and accurate procedural guidance. The goal of this study is to assess the positional accuracy and consistency of a clinical mixed reality system that utilizes commercially available wireless head-mounted displays (HMDs), custom software, and localization instruments.

METHODS

Independent teams using the second-generation Microsoft HoloLens© hardware, Medivis SurgicalAR© software, and localization instruments, tested the accuracy of the combined system at different institutions, times, and locations. The ASTM F2554-18 consensus standard for computer-assisted surgical systems, as recognized by the U.S. FDA, was utilized to measure the performance. 288 tests were performed.

RESULTS

The system demonstrated consistent results, with an average accuracy performance that was better than one millimeter (.75 ± SD .37 mm).

CONCLUSION

Independently acquired positional tracking accuracies exceed conventional in-market surgical navigation tracking systems and FDA standards. Importantly, the performance was achieved at two different institutions, using an international testing standard, and with a system that included a commercially available off-the-shelf wireless head mounted display and software.

The Feasibility and Accuracy of Holographic Navigation with Laser Crosshair Simulator Registration on a Mixed-Reality Display

Addressing conventional neurosurgical navigation systems' high costs and complexity, this study explores the feasibility and accuracy of a simplified, cost-effective mixed reality navigation (MRN) system based on a laser crosshair simulator (LCS). A new automatic registration method was developed, featuring coplanar laser emitters and a recognizable target pattern. The workflow was integrated into Microsoft's HoloLens-2 for practical application. The study assessed the system's precision by utilizing life-sized 3D-printed head phantoms based on computed tomography (CT) or magnetic resonance imaging (MRI) data from 19 patients (female/male: 7/12, average age: 54.4 ± 18.5 years) with intracranial lesions. Six to seven CT/MRI-visible scalp markers were used as reference points per case. The LCS-MRN's accuracy was evaluated through landmark-based and lesion-based analyses, using metrics such as target registration error (TRE) and Dice similarity coefficient (DSC). The system demonstrated immersive capabilities for observing intracranial structures across all cases. Analysis of 124 landmarks showed a TRE of 3.0 ± 0.5 mm, consistent across various surgical positions. The DSC of 0.83 ± 0.12 correlated significantly with lesion volume (Spearman rho = 0.813, p < 0.001). Therefore, the LCS-MRN system is a viable tool for neurosurgical planning, highlighting its low user dependency, cost-efficiency, and accuracy, with prospects for future clinical application enhancements.

Modern Image-Guided Surgery: A Narrative Review of Medical Image Processing and Visualization

Medical image analysis forms the basis of image-guided surgery (IGS) and many of its fundamental tasks. Driven by the growing number of medical imaging modalities, the research community of medical imaging has developed methods and achieved functionality breakthroughs. However, with the overwhelming pool of information in the literature, it has become increasingly challenging for researchers to extract context-relevant information for specific applications, especially when many widely used methods exist in a variety of versions optimized for their respective application domains. By being further equipped with sophisticated three-dimensional (3D) medical image visualization and digital reality technology, medical experts could enhance their performance capabilities in IGS by multiple folds. The goal of this narrative review is to organize the key components of IGS in the aspects of medical image processing and visualization with a new perspective and insights. The literature search was conducted using mainstream academic search engines with a combination of keywords relevant to the field up until mid-2022. This survey systemically summarizes the basic, mainstream, and state-of-the-art medical image processing methods as well as how visualization technology like augmented/mixed/virtual reality (AR/MR/VR) are enhancing performance in IGS. Further, we hope that this survey will shed some light on the future of IGS in the face of challenges and opportunities for the research directions of medical image processing and visualization.

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Immersive technologies have thrived on a strong foundation of software and hardware, injecting vitality into medical training. This surge has witnessed numerous endeavors incorporating immersive technologies into surgery simulation for surgical skills training, with a growing number of researchers delving into this domain. Relevant experiences and patterns need to be summarized urgently to enable researchers to establish a comprehensive understanding of this field, thus promoting its continuous growth. This study provides a forward-looking perspective by reviewing the latest development of immersive interactive technologies for surgery simulation. The investigation commences from a technological standpoint, delving into the core aspects of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies, namely, haptic rendering and tracking. Subsequently, we summarize recent work based on the categorization of minimally invasive surgery (MIS) and open surgery simulations. Finally, the study showcases the impressive performance and expansive potential of immersive technologies in surgical simulation while also discussing the current limitations. We find that the design of interaction and the choice of immersive technology in virtual surgery development should be closely related to the corresponding interactive operations in the real surgical speciality. This alignment facilitates targeted technological adaptations in the direction of greater applicability and fidelity of simulation.

A Novel Registration Method for a Mixed Reality Navigation System Based on a Laser Crosshair Simulator: A Technical Note

Mixed Reality Navigation (MRN) is pivotal in augmented reality-assisted intelligent neurosurgical interventions. However, existing MRN registration methods face challenges in concurrently achieving low user dependency, high accuracy, and clinical applicability. This study proposes and evaluates a novel registration method based on a laser crosshair simulator, evaluating its feasibility and accuracy. A novel registration method employing a laser crosshair simulator was introduced, designed to replicate the scanner frame's position on the patient. The system autonomously calculates the transformation, mapping coordinates from the tracking space to the reference image space. A mathematical model and workflow for registration were designed, and a Universal Windows Platform (UWP) application was developed on HoloLens-2. Finally, a head phantom was used to measure the system's target registration error (TRE). The proposed method was successfully implemented, obviating the need for user interactions with virtual objects during the registration process. Regarding accuracy, the average deviation was 3.7 ± 1.7 mm. This method shows encouraging results in efficiency and intuitiveness and marks a valuable advancement in low-cost, easy-to-use MRN systems. The potential for enhancing accuracy and adaptability in intervention procedures positions this approach as promising for improving surgical outcomes.

Virtual reality-based training and pre-operative planning for head and neck sentinel lymph node biopsy

OBJECTIVE

Sentinel lymph node biopsy (SLNB) is crucial for managing head and neck skin cancer. However, variable lymphatic drainage can complicate SLN detection when using Single-Photon Emission Computed Tomography (SPECT) or lymphoscintigraphy. Virtual Reality (VR) can contribute to pre-operative planning by simulating a realistic 3D model, which improves orientation. VR can also facilitate real-patient training outside the operating room. This study explored using a VR platform for pre-operative planning in head and neck skin cancer patients undergoing SLNBs and assessed its value for residential training.

MATERIALS AND METHODS

In this prospective technology pilot study, attending surgeons and residents who performed 21 SLNB operations on patients with head and neck skin cancers (81% males, mean age 69.2 ± 11.3) used a VR simulation model based on each patient's pre-operative SPECT scan to examine patient-specific anatomy. After surgery, they completed a questionnaire on the efficiency of the VR simulation as a pre-operative planning tool and training device for residents.

RESULTS

The attending surgeons rated the VR model's accuracy at 8.3 ± 1.6 out of 10. Three-quarters (76%) of residents reported increased confidence after using VR. The physicians rated the platform's contribution to residents' training at 7.4 ± 2.1 to 8.9 ± 1.3 out of 10.

CONCLUSION

A VR SLNB simulation can accurately portray marked sentinel lymph nodes. It was rated high as a surgical planning and teaching tool among attending surgeons and residents alike and may play a role in pre-operative planning and resident training. Further studies are needed to explore its applications in practice.

Iatrogenic inner ear dehiscence associated with lateral skull base surgery: a systematic analysis of drilling injuries and their causal factors

PURPOSE

Drilling injuries of the inner ear are an underreported complication of lateral skull base (LSB) surgery. Inner ear breaches can cause hearing loss, vestibular dysfunction, and third window phenomenon. This study aims to elucidate primary factors causing iatrogenic inner ear dehiscences (IED) in 9 patients who presented to a tertiary care center with postoperative symptoms of IED following LSB surgery for vestibular schwannoma, endolymphatic sac tumor, Meniere's disease, paraganglioma jugulare, and vagal schwannoma.

METHODS

Utilizing 3D Slicer image processing software, geometric and volumetric analysis was applied to both preoperative and postoperative imaging to identify causal factors iatrogenic inner ear breaches. Segmentation analyses, craniotomy analyses, and drilling trajectory analyses were performed. Cases of retrosigmoid approaches for vestibular schwannoma resection were compared to matched controls.

RESULTS

Excessive lateral drilling and breach of a single inner ear structure occurred in 3 cases undergoing transjugular (n=2) and transmastoid (n=1) approaches. Inadequate drilling trajectory breaching ≥1 inner ear structure occurred in 6 cases undergoing retrosigmoid (n=4), transmastoid (n=1), and middle cranial fossa approaches (n=1). In retrosigmoid approaches the 2-cm visualization window and craniotomy limits did not provide drilling angles to the entire tumor without causing IED in comparison to matched controls.

CONCLUSIONS

Inappropriate drill depth, errant lateral drilling, inadequate drill trajectory, or a combination of these led to iatrogenic IED. Image-based segmentation, individualized 3D anatomical model generation, and geometric and volumetric analyses can optimize operative plans and possibly reduce inner ear breaches from lateral skull base surgery.

Artificial intelligence directed development of a digital twin to measure soft tissue shift during head and neck surgery

Digital twins derived from 3D scanning data were developed to measure soft tissue deformation in head and neck surgery by an artificial intelligence approach. This framework was applied suggesting feasibility of soft tissue shift detection as a hitherto unsolved problem. In a pig head cadaver model 104 soft tissue resection had been performed. The surface of the removed soft tissue (RTP) and the corresponding resection cavity (RC) was scanned (N = 416) to train an artificial intelligence (AI) with two different 3D object detectors (HoloLens 2; ArtecEva). An artificial tissue shift (TS) was created by changing the tissue temperature from 7,91±4,1°C to 36,37±1,28°C. Digital twins of RTP and RC in cold and warm conditions had been generated and volumes were calculated based on 3D surface meshes. Significant differences in number of vertices created by the different 3D scanners (HoloLens2 51313 vs. ArtecEva 21694, p<0.0001) hence result in differences in volume measurement of the RTC (p = 0.0015). A significant TS could be induced by changing the temperature of the tissue of RC (p = 0.0027) and RTP (p = <0.0001). RC showed more correlation in TS by heating than RTP with a volume increase of 3.1 μl or 9.09% (p = 0.449). Cadaver models are suitable for training a machine learning model for deformable registration through creation of a digital twin. Despite different point cloud densities, HoloLens and ArtecEva provide only slightly different estimates of volume. This means that both devices can be used for the task.TS can be simulated and measured by temperature change, in which RC and RTP react differently. This corresponds to the clinical behaviour of tumour and resection cavity during surgeries, which could be used for frozen section management and a range of other clinical applications.

Augmented Reality in Otology/Neurotology: A Scoping Review with Implications for Practice and Education

OBJECTIVE

To determine how augmented reality (AR) has been applied to the field of otology/neurotology, examine trends and gaps in research, and provide an assessment of the future potential of this technology within surgical practice and education.

DATA SOURCES

PubMed, EMBASE, and Cochrane Library were assessed from their inceptions through October 2022. A manual bibliography search was also conducted.

REVIEW METHODS

A scoping review was conducted and reported according to PRISMA-ScR guidelines. Data from studies describing the application of AR to the field of otology/neurotology were evaluated, according to a priori inclusion/exclusion criteria. Exclusion criteria included non-English language articles, abstracts, letters/commentaries, conference papers, and review articles.

RESULTS

Eighteen articles covering a diverse range of AR platforms were included. Publication dates spanned from 2007 to 2022 and the rate of publication increased over this time. Six of 18 studies were case series in human patients although the remaining were proof of concepts in cadaveric/artificial/animal models. The most common application of AR was for surgical navigation (14 of 18 studies). Computed tomography was the most common source of input data. Few studies noted potential applications to surgical training.

CONCLUSION

Interest in the application of AR to otology/neurotology is growing based on the number of recent publications that use a broad range of hardware, software, and AR platforms. Large gaps in research such as the need for submillimeter registration error must be addressed prior to adoption in the operating room and for educational purposes.

LEVEL OF EVIDENCE

N/A Laryngoscope, 133:1786-1795, 2023.

Accuracy Evaluation Trial of Mixed Reality-Guided Spinal Puncture Technology

Purpose

To evaluate the accuracy of mixed reality (MR)-guided visualization technology for spinal puncture (MRsp).

Methods

MRsp involved the following three steps: 1. Lumbar spine computed tomography (CT) data were obtained to reconstruct virtual 3D images, which were imported into a HoloLens (2nd gen). 2. The patented MR system quickly recognized the spatial orientation and superimposed the virtual image over the real spine in the HoloLens. 3. The operator performed the spinal puncture with structural information provided by the virtual image. A posture fixation cushion was used to keep the subjects' lateral decubitus position consistent. 12 subjects were recruited to verify the setup error and the registration error. The setup error was calculated using the first two CT scans and measuring the displacement of two location markers. The projection points of the upper edge of the L3 spinous process (L3↑), the lower edge of the L3 spinous process (L3↓), and the lower edge of the L4 spinous process (L4↓) in the virtual image were positioned and marked on the skin as the registration markers. A third CT scan was performed to determine the registration error by measuring the displacement between the three registration markers and the corresponding real spinous process edges.

Results

The setup errors in the position of the cranial location marker between CT scans along the left-right (LR), anterior-posterior (AP), and superior-inferior (SI) axes of the CT bed measured 0.09 ± 0.06 cm, 0.30 ± 0.28 cm, and 0.22 ± 0.12 cm, respectively, while those of the position of the caudal location marker measured 0.08 ± 0.06 cm, 0.29 ± 0.18 cm, and 0.18 ± 0.10 cm, respectively. The registration errors between the three registration markers and the subject's real L3↑, L3↓, and L4↓ were 0.11 ± 0.09 cm, 0.15 ± 0.13 cm, and 0.13 ± 0.10 cm, respectively, in the SI direction.

Conclusion

This MR-guided visualization technology for spinal puncture can accurately and quickly superimpose the reconstructed 3D CT images over a real human spine.

Assessment of Accuracy of Mixed Reality Device for Neuronavigation: Proposed Methodology and Results

Intraoperative neuronavigation is currently an essential component of neurosurgical operations in several contexts. Recent progress in mixed reality (MR) technology has attempted to overcome the disadvantages of standard neuronavigation systems allowing the surgeon to superimpose a 3D rendered image onto the patient's anatomy. We present the first study in the literature to assess the surface matching accuracy of MR rendered image. For the purposes of this study, we used HoloLens 2 with virtual surgery intelligence providing the software capability for image rendering. To assess the accuracy of using mixed reality device for neuronavigation intraoperatively. This study seeks to assess the accuracy of rendered holographic images from a mixed reality device as a means for neuronavigation intraoperatively. We used the Realistic Operative Workstation for Educating Neurosurgical Apprentices to represent a patient's skull with intracranial components which underwent standardized computed tomography (CT) and MRI imaging. Eleven predefined points were used for purposes of assessing the accuracy of the rendered image, compared with the intraoperative gold standard neuronavigation. The mean HoloLens values against the ground truth were significantly higher when compared with Stealth using CT scan as the imaging modality. Using extracranial anatomic landmarks, the HoloLens error values continued to be significantly higher in magnitude when compared with Stealth across CT and MRI. This study provides a relatively easy and feasible method to assess accuracy of MR-based navigation without requiring any additions to the established imaging protocols. We failed to show the equivalence of MR-based navigation over the current neuronavigation systems.

Mandibular resection and defect reconstruction guided by a contour registration-based augmented reality system: A preclinical trial

The aim of this study was to verify the feasibility and accuracy of a contour registration-based augmented reality (AR) system in jaw surgery. An AR system was developed to display the interaction between virtual planning and images of the surgical site in real time. Several trials were performed with the guidance of the AR system and the surgical guide. The postoperative cone beam CT (CBCT) data were matched with the preoperatively planned data to evaluate the accuracy of the system by comparing the deviations in distance and angle. All procedures were performed successfully. In nine model trials, distance and angular deviations for the mandible, reconstructed fibula, and fixation screws were 1.62 ± 0.38 mm, 1.86 ± 0.43 mm, 1.67 ± 0.70 mm, and 3.68 ± 0.71°, 5.48 ± 2.06°, 7.50 ± 1.39°, respectively. In twelve animal trials, results of the AR system were compared with the surgical guide. Distance deviations for the bilateral condylar outer poles were 0.93 ± 0.63 mm and 0.81 ± 0.30 mm, respectively (p = 0.68). Distance deviations for the bilateral mandibular posterior angles were 2.01 ± 2.49 mm and 2.89 ± 1.83 mm, respectively (p = 0.50). Distance and angular deviations for the mandible were 1.41 ± 0.61 mm, 1.21 ± 0.18 mm (p = 0.45), and 6.81 ± 2.21°, 6.11 ± 2.93° (p = 0.65), respectively. Distance and angular deviations for the reconstructed tibiofibular bones were 0.88 ± 0.22 mm, 0.84 ± 0.18 mm (p = 0.70), and 6.47 ± 3.03°, 6.90 ± 4.01° (p = 0.84), respectively. This study proposed a contour registration-based AR system to assist surgeons in intuitively observing the surgical plan intraoperatively. The trial results indicated that this system had similar accuracy to the surgical guide.

Skull-Base Surgery-A Narrative Review on Current Approaches and Future Developments in Surgical Navigation

Surgical navigation technology combines patient imaging studies with intraoperative real-time data to improve surgical precision and patient outcomes. The navigation workflow can also include preoperative planning, which can reliably simulate the intended resection and reconstruction. The advantage of this approach in skull-base surgery is that it guides access into a complex three-dimensional area and orients tumors intraoperatively with regard to critical structures, such as the orbit, carotid artery and brain. This enhances a surgeon's capabilities to preserve normal anatomy while resecting tumors with adequate margins. The aim of this narrative review is to outline the state of the art and the future directions of surgical navigation in the skull base, focusing on the advantages and pitfalls of this technique. We will also present our group experience in this field, within the frame of the current research trends.

Improved Accuracy and Lowered Learning Curve of Ventricular Targeting Using Augmented Reality-Phantom and Cadaveric Model Testing

BACKGROUND

Augmented reality (AR) has demonstrated significant potential in neurosurgical cranial, spine, and teaching applications. External ventricular drain (EVD) placement remains a common procedure, but with error rates in targeting between 10% and 40%.

OBJECTIVE

To evaluate Novarad VisAR guidance system for the placement of EVDs in phantom and cadaveric models.

METHODS

Two synthetic ventricular phantom models and a third cadaver model underwent computerized tomography imaging and registration with the VisAR system (Novarad). Root mean square (RMS), angular error (γ), and Euclidian distance were measured by multiple methods for various standard EVD placements.

RESULTS

Computerized tomography measurements on a phantom model (0.5-mm targets showed a mean Euclidean distance error of 1.20 ± 0.98 mm and γ of 1.25° ± 1.02°. Eight participants placed EVDs in lateral and occipital burr holes using VisAR in a second phantom anatomic ventricular model (mean RMS: 3.9 ± 1.8 mm, γ: 3.95° ± 1.78°). There were no statistically significant differences in accuracy for postgraduate year level, prior AR experience, prior EVD experience, or experience with video games ( P > .05). In comparing EVDs placed with anatomic landmarks vs VisAR navigation in a cadaver, VisAR demonstrated significantly better RMS and γ, 7.47 ± 0.94 mm and 7.12° ± 0.97°, respectively ( P ≤ .05).

CONCLUSION

The novel VisAR AR system resulted in accurate placement of EVDs with a rapid learning curve, which may improve clinical treatment and patient safety. Future applications of VisAR can be expanded to other cranial procedures.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Application of mixed reality-based surgical navigation system in craniomaxillofacial trauma bone reconstruction

OBJECTIVES

This study aimed to build a surgical navigation system based on mixed reality (MR) and optical positioning technique and evaluate its clinical applicability in craniomaxillofacial trauma bone reconstruction. Me-thods We first integrated the software and hardware platforms of the MR-based surgical navigation system and explored the system workflow. The systematic error, target registration error, and osteotomy application error of the system were then analyzed via 3D printed skull model experiment. The feasibility of the MR-based surgical navigation system in craniomaxillofacial trauma bone reconstruction was verified via zygomatico-maxillary complex (ZMC) reduction experiment of the skull model and preliminary clinical study.

RESULTS

The system error of this MR-based surgical navigation system was 1.23 mm±0.52 mm, the target registration error was 2.83 mm±1.18 mm, and the osteotomy application error was 3.13 mm±1.66 mm. Virtual surgical planning and the reduction of the ZMC model were successfully conducted. In addition, with the guidance of the MR-based navigation system, the frontal bone defect was successfully reconstructed, and the clinical outcome was satisfactory.

CONCLUSIONS

The MR-based surgical navigation system has its advantages in virtual reality fusion effect and dynamic navigation stability. It provides a new method for doctor-patient communications, education, preoperative planning, and intraoperative navigation in craniomaxillofacial surgery.

The role of spatial ability in mixed reality learning with the HoloLens

The use of mixed reality in science education has been increasing and as such it has become more important to understand how information is learned in these virtual environments. Spatial ability is important in many learning contexts, but especially in neuroanatomy education where learning the locations and spatial relationships between brain regions is paramount. It is currently unclear what role spatial ability plays in mixed reality learning environments, and whether it is different compared to traditional physical environments. To test this, a learning experiment was conducted where students learned neuroanatomy using both mixed reality and a physical plastic model of a brain (N = 27). Spatial ability was assessed and analyzed to determine its effect on performance across the two learning modalities. The results showed that spatial ability facilitated learning in mixed reality (β = 0.21, P = 0.003), but not when using a plastic model (β = 0.08, P = 0.318). A non-significant difference was observed between the modalities in terms of knowledge test performance (d = 0.39, P = 0.052); however, mixed reality was more engaging (d = 0.59, P = 0.005) and learners were more confident in the information they learned compared to using a physical model (d = 0.56, P = 0.007). Overall, these findings suggest that spatial ability is more relevant in virtual learning environments, where the ability to manipulate and interact with an object is diminished or abstracted through a virtual user interface.

Attitudes Toward Neurosurgery Education for the Nonneurosurgeon: A Survey Study and Critical Analysis of U.S. Military Training Techniques and Future Prospects

BACKGROUND

The U.S. military requires medical readiness to support forward-deployed combat operations. Because time and distance to neurosurgical capabilities vary within the deployed trauma system, nonneurosurgeons are required to perform emergent cranial procedures in select cases. It is unclear whether these surgeons have sufficient training in these procedures.

METHODS

This quality-improvement study involved a voluntary, anonymized specialty-specific survey of active-duty surgeons about their experience and attitudes toward U.S. military emergency neurosurgical training.

RESULTS

Survey responses were received from 104 general surgeons and 26 neurosurgeons. Among general surgeons, 81% have deployed and 53% received training in emergency neurosurgical procedures before deployment. Only 16% of general surgeons reported participating in craniotomy/craniectomy procedures in the last year. Nine general surgeons reported performing an emergency neurosurgical procedure while on deployment/humanitarian mission, and 87% of respondents expressed interest in further predeployment emergency neurosurgery training. Among neurosurgeons, 81% had participated in training nonneurosurgeons and 73% believe that more comprehensive training for nonneurosurgeons before deployment is needed. General surgeons proposed lower procedure minimums for competency for external ventricular drain placement and craniotomy/craniectomy than did neurosurgeons. Only 37% of general surgeons had used mixed/augmented reality in any capacity previously; for combat procedures, most (90%) would prefer using synchronous supervision via high-fidelity video teleconferencing over mixed reality.

CONCLUSIONS

These survey results show a gap in readiness for neurosurgical procedures for forward-deployed general surgeons. Capitalizing on capabilities such as mixed/augmented reality would be a force multiplier and a potential means of improving neurosurgical capabilities in the forward-deployed environments.

Augmented Reality-Based Surgery on the Human Cadaver Using a New Generation of Optical Head-Mounted Displays: Development and Feasibility Study

Background

Although nearly one-third of the world’s disease burden requires surgical care, only a small proportion of digital health applications are directly used in the surgical field. In the coming decades, the application of augmented reality (AR) with a new generation of optical-see-through head-mounted displays (OST-HMDs) like the HoloLens (Microsoft Corp) has the potential to bring digital health into the surgical field. However, for the application to be performed on a living person, proof of performance must first be provided due to regulatory requirements. In this regard, cadaver studies could provide initial evidence.

Objective

The goal of the research was to develop an open-source system for AR-based surgery on human cadavers using freely available technologies.

Methods

We tested our system using an easy-to-understand scenario in which fractured zygomatic arches of the face had to be repositioned with visual and auditory feedback to the investigators using a HoloLens. Results were verified with postoperative imaging and assessed in a blinded fashion by 2 investigators. The developed system and scenario were qualitatively evaluated by consensus interview and individual questionnaires.

Results

The development and implementation of our system was feasible and could be realized in the course of a cadaver study. The AR system was found helpful by the investigators for spatial perception in addition to the combination of visual as well as auditory feedback. The surgical end point could be determined metrically as well as by assessment.

Conclusions

The development and application of an AR-based surgical system using freely available technologies to perform OST-HMD–guided surgical procedures in cadavers is feasible. Cadaver studies are suitable for OST-HMD–guided interventions to measure a surgical end point and provide an initial data foundation for future clinical trials. The availability of free systems for researchers could be helpful for a possible translation process from digital health to AR-based surgery using OST-HMDs in the operating theater via cadaver studies.

Mixed Reality Technology in Total Knee Arthroplasty: An Updated Review With a Preliminary Case Report

Background

Augmented reality and mixed reality have been used to help surgeons perform complex surgeries. With the development of technology, mixed reality (MR) technology has been used to improve the success rate of complex hip arthroplasty due to its unique advantages. At present, there are few reports on the application of MR technology in total knee arthroplasty. We presented a case of total knee arthroplasty with the help of mixed reality technology.

Case Presentation

We presented a case of a 71-year-old woman who was diagnosed with bilateral knee osteoarthritis with varus deformity, especially on the right side. After admission, the right total knee arthroplasty was performed with the assistance of MR technology. Before the operation, the three-dimensional virtual model of the knee joint of the patient was reconstructed for condition analysis, operation plan formulation, and operation simulation. During the operation, the three-dimensional virtual images of the femur and tibia coincided with the real body of the patient, showing the osteotomy plane designed before the operation, which can accurately guide the completion of osteotomy and prosthesis implantation.

Conclusions

As far as we know, this is the first report on total knee arthroplasty under the guidance of mixed reality technology.

Application of Fused Reality Holographic Image and Navigation Technology in the Puncture Treatment of Hypertensive Intracerebral Hemorrhage

Objective

Minimally invasive puncture and drainage (MIPD) of hematomas was the preferred option for appropriate patients with hypertensive intracerebral hemorrhage (HICH). The goal of our research was to introduce the MIPD surgery using mixed reality holographic navigation technology (MRHNT).

Method

We provided the complete workflow for hematoma puncture using MRHNT included three-dimensional model reconstruction by preoperative CT examination, puncture trajectory design, immersive presentation of model, and real environment and hematoma puncture using dual-plane navigation by wearing special equipment. We collected clinical data on eight patients with HICH who underwent MIPD using MRHNT from March 2021 to August 2021, including the hematoma evacuation rate, operation time, deviation in drainage tube target, postoperative complications, and 2-week postoperative GCS.

Result

The workflow for hematoma puncture using MRHNT were performed in all eight cases, in which the average hematoma evacuation rate was 47.36±9.16%, the average operation time was 82.14±15.74 min, and the average deviation of the drainage tube target was 5.76±0.80 mm. There was no delayed bleeding, acute ischemic stroke, intracranial infection, or epilepsy 2 weeks after surgery. The 2-week postoperative GCS was improved compared with the preoperative GCS.

Conclusion

The research concluded it was feasible to perform the MIPD by MRHNT on patients with HICH. The risk of general anesthesia and highly professional holographic information processing restricted the promotion of the technology, it was necessary for technical innovation and the accumulation of more case experience and verification of its superiority.

Mixed-Reality-Assisted Puncture of the Common Femoral Artery in a Phantom Model

Percutaneous femoral arterial access is daily practice in a variety of medical specialties and enables physicians worldwide to perform endovascular interventions. The reported incidence of percutaneous femoral arterial access complications is 3–18% and often results from suboptimal puncture location due to insufficient visualization of the target vessel. The purpose of this proof-of-concept study was to evaluate the feasibility and the positional error of a mixed-reality (MR)-assisted puncture of the common femoral artery in a phantom model using a commercially available navigation system. In total, 15 MR-assisted punctures were performed. Cone-beam computed tomography angiography (CTA) was used following each puncture to allow quantification of positional error of needle placements in the axial and sagittal planes. Technical success was achieved in 14/15 cases (93.3%) with a median axial positional error of 1.0 mm (IQR 1.3) and a median sagittal positional error of 1.1 mm (IQR 1.6). The median duration of the registration process and needle insertion was 2 min (IQR 1.0). MR-assisted puncture of the common femoral artery is feasible with acceptable positional errors in a phantom model. Future studies should aim to measure and reduce the positional error resulting from MR registration.

Validation and accuracy evaluation of automatic segmentation for knee joint pre-planning

BACKGROUND

Proper use of three-dimensional (3D) models generated from medical imaging data in clinical preoperative planning, training and consultation is based on the preliminary proved accuracy of the replication of the patient anatomy. Therefore, this study investigated the dimensional accuracy of 3D reconstructions of the knee joint generated from computed tomography scans via automatic segmentation by comparing them with 3D models generated through manual segmentation.

METHODS

Three unpaired, fresh-frozen right legs were investigated. Three-dimensional models of the femur and the tibia of each leg were manually segmented using a commercial software and compared in terms of geometrical accuracy with the 3D models automatically segmented using proprietary software. Bony landmarks were identified and used to calculate clinically relevant distances: femoral epicondylar distance; posterior femoral epicondylar distance; femoral trochlear groove length; tibial knee center tubercle distance (TKCTD). Pearson's correlation coefficient and Bland and Altman plots were used to evaluate the level of agreement between measured distances.

RESULTS

Differences between parameters measured on 3D models manually and automatically segmented were below 1 mm (range: -0.06 to 0.72 mm), except for TKCTD (between 1.00 and 1.40 mm in two specimens). In addition, there was a significant strong correlation between measurements.

CONCLUSIONS

The results obtained are comparable to those reported in previous studies where accuracy of bone 3D reconstruction was investigated. Automatic segmentation techniques can be used to quickly reconstruct reliable 3D models of bone anatomy and these results may contribute to enhance the spread of this technology in preoperative and operative settings, where it has shown considerable potential.

Augmented Reality for Retrosigmoid Craniotomy Planning

While medical imaging data have traditionally been viewed on two-dimensional (2D) displays, augmented reality (AR) allows physicians to project the medical imaging data on patient's bodies to locate important anatomy. We present a surgical AR application to plan the retrosigmoid craniotomy, a standard approach to access the posterior fossa and the internal auditory canal. As a simple and accurate alternative to surface landmarks and conventional surgical navigation systems, our AR application augments the surgeon's vision to guide the optimal location of cortical bone removal. In this work, two surgeons performed a retrosigmoid approach 14 times on eight cadaver heads. In each case, the surgeon manually aligned a computed tomography (CT)-derived virtual rendering of the sigmoid sinus on the real cadaveric heads using a see-through AR display, allowing the surgeon to plan and perform the craniotomy accordingly. Postprocedure CT scans were acquired to assess the accuracy of the retrosigmoid craniotomies with respect to their intended location relative to the dural sinuses. The two surgeons had a mean margin of davg = 0.6 ± 4.7 mm and davg = 3.7 ± 2.3 mm between the osteotomy border and the dural sinuses over all their cases, respectively, and only positive margins for 12 of the 14 cases. The intended surgical approach to the internal auditory canal was successfully achieved in all cases using the proposed method, and the relatively small and consistent margins suggest that our system has the potential to be a valuable tool to facilitate planning a variety of similar skull-base procedures.

Fully automatic brain tumor segmentation for 3D evaluation in augmented reality

OBJECTIVE

For currently available augmented reality workflows, 3D models need to be created with manual or semiautomatic segmentation, which is a time-consuming process. The authors created an automatic segmentation algorithm that generates 3D models of skin, brain, ventricles, and contrast-enhancing tumor from a single T1-weighted MR sequence and embedded this model into an automatic workflow for 3D evaluation of anatomical structures with augmented reality in a cloud environment. In this study, the authors validate the accuracy and efficiency of this automatic segmentation algorithm for brain tumors and compared it with a manually segmented ground truth set.

METHODS

Fifty contrast-enhanced T1-weighted sequences of patients with contrast-enhancing lesions measuring at least 5 cm3 were included. All slices of the ground truth set were manually segmented. The same scans were subsequently run in the cloud environment for automatic segmentation. Segmentation times were recorded. The accuracy of the algorithm was compared with that of manual segmentation and evaluated in terms of Sørensen-Dice similarity coefficient (DSC), average symmetric surface distance (ASSD), and 95th percentile of Hausdorff distance (HD95).

RESULTS

The mean ± SD computation time of the automatic segmentation algorithm was 753 ± 128 seconds. The mean ± SD DSC was 0.868 ± 0.07, ASSD was 1.31 ± 0.63 mm, and HD95 was 4.80 ± 3.18 mm. Meningioma (mean 0.89 and median 0.92) showed greater DSC than metastasis (mean 0.84 and median 0.85). Automatic segmentation had greater accuracy for measuring DSC (mean 0.86 and median 0.87) and HD95 (mean 3.62 mm and median 3.11 mm) of supratentorial metastasis than those of infratentorial metastasis (mean 0.82 and median 0.81 for DSC; mean 5.26 mm and median 4.72 mm for HD95).

CONCLUSIONS

The automatic cloud-based segmentation algorithm is reliable, accurate, and fast enough to aid neurosurgeons in everyday clinical practice by providing 3D augmented reality visualization of contrast-enhancing intracranial lesions measuring at least 5 cm3. The next steps involve incorporation of other sequences and improving accuracy with 3D fine-tuning in order to expand the scope of augmented reality workflow.

Holographic mixed-reality neuronavigation with a head-mounted device: technical feasibility and clinical application

OBJECTIVE

The authors aimed to evaluate the technical feasibility of a mixed-reality neuronavigation (MRN) system with a wearable head-mounted device (HMD) and to determine its clinical application and accuracy.

METHODS

A semiautomatic registration MRN system on HoloLens smart glasses was developed and tested for accuracy and feasibility. Thirty-seven patients with intracranial lesions were prospectively identified. For each patient, multimodal imaging-based holograms of lesions, markers, and surrounding eloquent structures were created and then imported to the MRN HMD. After a point-based registration, the holograms were projected onto the patient's head and observed through the HMD. The contour of the holograms was compared with standard neuronavigation (SN). The projection of the lesion boundaries perceived by the neurosurgeon on the patient's scalp was then marked with MRN and SN. The distance between the two contours generated by MRN and SN was measured so that the accuracy of MRN could be assessed.

RESULTS

MRN localization was achieved in all patients. The mean additional time required for MRN was 36.3 ± 6.3 minutes, in which the mean registration time was 2.6 ± 0.9 minutes. A trend toward a shorter time required for preparation was observed with the increase of neurosurgeon experience with the MRN system. The overall median deviation was 4.1 mm (IQR 3.0 mm-4.7 mm), and 81.1% of the lesions localized by MRN were found to be highly consistent with SN (deviation < 5.0 mm). There was a significant difference between the supine position and the prone position (3.7 ± 1.1 mm vs 5.4 ± 0.9 mm, p = 0.001). The magnitudes of deviation vectors did not correlate with lesion volume (p = 0.126) or depth (p = 0.128). There was no significant difference in additional operating time between different operators (37.4 ± 4.8 minutes vs 34.6 ± 4.8 minutes, p = 0.237) or in localization deviation (3.7 ± 1.0 mm vs 4.6 ± 1.5 mm, p = 0.070).

CONCLUSIONS

This study provided a complete set of a clinically applicable workflow on an easy-to-use MRN system using a wearable HMD, and has shown its technical feasibility and accuracy. Further development is required to improve the accuracy and clinical efficacy of this system.

Evaluation of a Wearable AR Platform for Guiding Complex Craniotomies in Neurosurgery

Today, neuronavigation is widely used in daily clinical routine to perform safe and efficient surgery. Augmented reality (AR) interfaces can provide anatomical models and preoperative planning contextually blended with the real surgical scenario, overcoming the limitations of traditional neuronavigators. This study aims to demonstrate the reliability of a new-concept AR headset in navigating complex craniotomies. Moreover, we aim to prove the efficacy of a patient-specific template-based methodology for fast, non-invasive, and fully automatic planning-to-patient registration. The AR platform navigation performance was assessed with an in-vitro study whose goal was twofold: to measure the real-to-virtual 3D target visualization error (TVE), and assess the navigation accuracy through a user study involving 10 subjects in tracing a complex craniotomy. The feasibility of the template-based registration was preliminarily tested on a volunteer. The TVE mean and standard deviation were 1.3 and 0.6 mm. The results of the user study, over 30 traced craniotomies, showed that 97% of the trajectory length was traced within an error margin of 1.5 mm, and 92% within a margin of 1 mm. The in-vivo test confirmed the feasibility and reliability of the patient-specific template for registration. The proposed AR headset allows ergonomic and intuitive fruition of preoperative planning, and it can represent a valid option to support neurosurgical tasks.

The utility of augmented reality in lateral skull base surgery: A preliminary report

OBJECTIVE

To discuss the utility of augmented reality in lateral skull base surgery.

PATIENTS

Those undergoing lateral skull base surgery at our institution.

INTERVENTION(S)

Cerebellopontine angle tumor resection using an augmented reality interface.

MAIN OUTCOME MEASURE(S)

Ease of use, utility of, and future directions of augmented reality in lateral skull base surgery.

RESULTS

Anecdotally we have found an augmented reality interface helpful in simulating cerebellopontine angle tumor resection as well as assisting in planning the incision and craniotomy.

CONCLUSIONS

Augmented reality has the potential to be a useful adjunct in lateral skull base surgery, but more study is needed with large series.

Mixed reality models based on low-dose computed tomography technology in nephron-sparing surgery are better than models based on normal-dose computed tomography

Background

Nephron-sparing surgery has been widely applied in the treatment of renal tumors. Previous studies have confirmed the advantages of mixed reality technology in surgery. The study aimed to explore the optimization of mixed reality technology and its application value in nephron-sparing surgery.

Methods

In this prospective study of 150 patients with complex renal tumors (RENAL nephrometry score ≥7) who underwent nephron-sparing surgery, patients were randomly divided into Group A (the normal-dose mixed reality group, n=50), Group B (the low-dose mixed reality group, n=50), and Group C (the traditional computed tomography image group, n=50). Group A and Group C received the normal-dose computed tomography scan protocol: 120 kVp, 400 mA, and 350 mgI/mL, while Group B received the low-dose computed tomography scan protocol: 80 kVp, automatic tube current modulation, and 320 mgI/mL. All computed tomography data were transmitted to a three-dimensional visualization workstation and underwent modeling and mixed reality imaging. Two senior surgeons evaluated mixed reality quality. Objective indexes and perioperative indexes were calculated and compared.

Results

Compared with Group A, the radiation effective dose in Group B was decreased by 39.6%. The subjective scores of mixed reality quality in Group B were significantly higher than those of Group A (Z=-4.186, P<0.001). The inter-observer agreement between the two senior surgeons in mixed reality quality was excellent (K=0.840, P<0.001). The perioperative indexes showed that the mixed reality groups were significantly different from the computed tomography image group (all P<0.017). More cases underwent nephron-sparing surgery in the mixed reality groups than in the computed tomography image group (P<0.0017).

Conclusions

Low-dose computed tomography technology can be effectively applied to mixed reality optimization, reducing the effective dose and improving mixed reality quality. Optimized mixed reality can significantly increase the cases of successful nephron-sparing surgery and improve perioperative indexes.

Patient-specific virtual and mixed reality for immersive, experiential anatomy education and for surgical planning in temporal bone surgery

OBJECTIVE

The recent development of extended reality technology has attracted interest in medicine. We explored the use of patient-specific virtual reality (VR) and mixed reality (MR) temporal bone models in anatomical teaching, pre-operative surgical planning and intra-operative surgical referencing.

METHODS

VR and MR temporal bone models were created and visualized on head-mounted display (HMD) and MR headset respectively, by a novel webservice that allows users to convert computed tomography images to VR and MR images without specific knowledge of programming. Eleven otorhinolaryngology trainees and specialists were asked to manipulate the healthy VR temporal bone model and to assess its validity by filling out a questionnaire. Additionally, VR and MR pathological models of petrous apex cholesteatoma were utilized for surgical planning pre-operatively and for referring to the anatomy during the surgery.

RESULTS

Most participants were favorable about the VR model and considered HMD as superior to a flat computer screen. 91% of the participants agreed or somewhat agreed that VR through HMD is cost effective. In addition, the VR pathological model was used for planning and sharing the surgical approach during a pre-operative surgical conference. The MR headset was worn intra-operatively to clarify the relationship between the pathological lesion and vital anatomical structures.

CONCLUSION

Regardless of the participants' training level in otorhinolaryngology or VR experience, all participants agreed that the VR temporal bone model is useful for anatomical education. Furthermore, the creation of patient-specific VR and MR models using the webservice and their pre- and intra-operative usages indicated the potential of innovative adjunctive surgical instrument.

Holographic patient tracking after bed movement for augmented reality neuronavigation using a head-mounted display

BACKGROUND

Holographic neuronavigation has several potential advantages compared to conventional neuronavigation systems. We present the first report of a holographic neuronavigation system with patient-to-image registration and patient tracking with a reference array using an augmented reality head-mounted display (AR-HMD).

METHODS

Three patients undergoing an intracranial neurosurgical procedure were included in this pilot study. The relevant anatomy was first segmented in 3D and then uploaded as holographic scene in our custom neuronavigation software. Registration was performed using point-based matching using anatomical landmarks. We measured the fiducial registration error (FRE) as the outcome measure for registration accuracy. A custom-made reference array with QR codes was integrated in the neurosurgical setup and used for patient tracking after bed movement.

RESULTS

Six registrations were performed with a mean FRE of 8.5 mm. Patient tracking was achieved with no visual difference between the registration before and after movement.

CONCLUSIONS

This first report shows a proof of principle of intraoperative patient tracking using a standalone holographic neuronavigation system. The navigation accuracy should be further optimized to be clinically applicable. However, it is likely that this technology will be incorporated in future neurosurgical workflows because the system improves spatial anatomical understanding for the surgeon.

Augmented Reality with HoloLens® in Parotid Tumor Surgery: A Prospective Feasibility Study

INTRODUCTION

Augmented reality can improve planning and execution of surgical procedures. Head-mounted devices such as the HoloLens® (Microsoft, Redmond, WA, USA) are particularly suitable to achieve these aims because they are controlled by hand gestures and enable contactless handling in a sterile environment.

OBJECTIVES

So far, these systems have not yet found their way into the operating room for surgery of the parotid gland. This study explored the feasibility and accuracy of augmented reality-assisted parotid surgery.

METHODS

2D MRI holographic images were created, and 3D holograms were reconstructed from MRI DICOM files and made visible via the HoloLens. 2D MRI slices were scrolled through, 3D images were rotated, and 3D structures were shown and hidden only using hand gestures. The 3D model and the patient were aligned manually.

RESULTS

The use of augmented reality with the HoloLens in parotic surgery was feasible. Gestures were recognized correctly. Mean accuracy of superimposition of the holographic model and patient's anatomy was 1.3 cm. Highly significant differences were seen in position error of registration between central and peripheral structures (p = 0.0059), with a least deviation of 10.9 mm (centrally) and highest deviation for the peripheral parts (19.6-mm deviation).

CONCLUSION

This pilot study offers a first proof of concept of the clinical feasibility of the HoloLens for parotid tumor surgery. Workflow is not affected, but additional information is provided. The surgical performance could become safer through the navigation-like application of reality-fused 3D holograms, and it improves ergonomics without compromising sterility. Superimposition of the 3D holograms with the surgical field was possible, but further invention is necessary to improve the accuracy.

[Application of mixed reality technique for the surgery of oral and maxillofacial tumors]

OBJECTIVE

To explore the application of mixed reality technique for the surgery of oral and maxillofacial tumors.

METHODS

In this study, patients with a diagnosis of an oral and maxillofacial tumor who were referred to Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology from December 2018 to January 2020 were selected. The preoperative contrast-enhanced computed tomography data of the patients were imported into StarAtlas Holographic Medical Imaging System (Visual 3D Corp., Beijing, China). Three-dimensional (3D) model of tumor and key structures, such as skeleton and vessels were reconstructed to three-dimensionally present the spatial relationship between them, followed with the key structures delineation and preoperative virtual surgical planning. By using mixed reality technique, the real-time 3D model was displayed stereotactically in the surgical site. While keeping sterile during operation, the surgeon could use simple gestures to adjust the 3D model, and observed the location, range, and size of tumor and the key structures adjacent to the tumor. Mixed reality technique was used to assist the operation: 3D model registration was performed for guidance before tumor excision; intraoperative real-time verification was performed during tumor exposure and after excision of the tumor. The Likert scale was used to evaluate the application of mixed reality technique after the operation.

RESULTS

Eight patients underwent mixed reality assisted tumor resection, and all of them successfully completed the operation. The average time of the 3D model registration was 12.0 minutes. In all the cases, the surgeon could intuitively and three-dimensionally observe the 3D model of the tumor and the surrounding anatomical structures, and could adjust the model during the operation. The results of the Likert scale showed that mixed reality technique got high scores in terms of perceptual accuracy, helping to locate the anatomical parts, the role of model guidance during surgery, and the potential for improving surgical safety (4.22, 4.19, 4.16, and 4.28 points respectively). Eight patients healed well without perioperative complications.

CONCLUSION

By providing real-time stereotactic visualization of anatomy of surgical site and guiding the operation process through 3D model, mixed reality technique could improve the accuracy and safety of the excision of oral and maxillofacial tumors.

[Application of mixed reality technique for the surgery of oral and maxillofacial tumors]

OBJECTIVE

To explore the application of mixed reality technique for the surgery of oral and maxillofacial tumors.

METHODS

In this study, patients with a diagnosis of an oral and maxillofacial tumor who were referred to Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology from December 2018 to January 2020 were selected. The preoperative contrast-enhanced computed tomography data of the patients were imported into StarAtlas Holographic Medical Imaging System (Visual 3D Corp., Beijing, China). Three-dimensional (3D) model of tumor and key structures, such as skeleton and vessels were reconstructed to three-dimensionally present the spatial relationship between them, followed with the key structures delineation and preoperative virtual surgical planning. By using mixed reality technique, the real-time 3D model was displayed stereotactically in the surgical site. While keeping sterile during operation, the surgeon could use simple gestures to adjust the 3D model, and observed the location, range, and size of tumor and the key structures adjacent to the tumor. Mixed reality technique was used to assist the operation: 3D model registration was performed for guidance before tumor excision; intraoperative real-time verification was performed during tumor exposure and after excision of the tumor. The Likert scale was used to evaluate the application of mixed reality technique after the operation.

RESULTS

Eight patients underwent mixed reality assisted tumor resection, and all of them successfully completed the operation. The average time of the 3D model registration was 12.0 minutes. In all the cases, the surgeon could intuitively and three-dimensionally observe the 3D model of the tumor and the surrounding anatomical structures, and could adjust the model during the operation. The results of the Likert scale showed that mixed reality technique got high scores in terms of perceptual accuracy, helping to locate the anatomical parts, the role of model guidance during surgery, and the potential for improving surgical safety (4.22, 4.19, 4.16, and 4.28 points respectively). Eight patients healed well without perioperative complications.

CONCLUSION

By providing real-time stereotactic visualization of anatomy of surgical site and guiding the operation process through 3D model, mixed reality technique could improve the accuracy and safety of the excision of oral and maxillofacial tumors.

Registration Techniques for Clinical Applications of Three-Dimensional Augmented Reality Devices

Many clinical procedures would benefit from direct and intuitive real-time visualization of anatomy, surgical plans, or other information crucial to the procedure. Three-dimensional augmented reality (3D-AR) is an emerging technology that has the potential to assist physicians with spatial reasoning during clinical interventions. The most intriguing applications of 3D-AR involve visualizations of anatomy or surgical plans that appear directly on the patient. However, commercially available 3D-AR devices have spatial localization errors that are too large for many clinical procedures. For this reason, a variety of approaches for improving 3D-AR registration accuracy have been explored. The focus of this review is on the methods, accuracy, and clinical applications of registering 3D-AR devices with the clinical environment. The works cited represent a variety of approaches for registering holograms to patients, including manual registration, computer vision-based registration, and registrations that incorporate external tracking systems. Evaluations of user accuracy when performing clinically relevant tasks suggest that accuracies of approximately 2 mm are feasible. 3D-AR device limitations due to the vergence-accommodation conflict or other factors attributable to the headset hardware add on the order of 1.5 mm of error compared to conventional guidance. Continued improvements to 3D-AR hardware will decrease these sources of error.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

Current Accuracy of Augmented Reality Neuronavigation Systems: Systematic Review and Meta-Analysis

BACKGROUND

Augmented reality neuronavigation (ARN) systems can overlay three-dimensional anatomy and disease without the need for a two-dimensional external monitor. Accuracy is crucial for their clinical applicability. We performed a systematic review regarding the reported accuracy of ARN systems and compared them with the accuracy of conventional infrared neuronavigation (CIN).

METHODS

PubMed and Embase were searched for ARN and CIN systems. For ARN, type of system, method of patient-to-image registration, accuracy method, and accuracy of the system were noted. For CIN, navigation accuracy, expressed as target registration error (TRE), was noted. A meta-analysis was performed comparing the TRE of ARN and CIN systems.

RESULTS

Thirty-five studies were included, 12 for ARN and 23 for CIN. ARN systems could be divided into head-mounted display and heads-up display. In ARN, 4 methods were encountered for patient-to-image registration, of which point-pair matching was the one most frequently used. Five methods for assessing accuracy were described. Ninety-four TRE measurements of ARN systems were compared with 9058 TRE measurements of CIN systems. Mean TRE was 2.5 mm (95% confidence interval, 0.7-4.4) for ARN systems and 2.6 mm (95% confidence interval, 2.1-3.1) for CIN systems.

CONCLUSIONS

In ARN, there seems to be lack of agreement regarding the best method to assess accuracy. Nevertheless, ARN systems seem able to achieve an accuracy comparable to CIN systems. Future studies should be prospective and compare TREs, which should be measured in a standardized fashion.

Clinical Accuracy of Holographic Navigation Using Point-Based Registration on Augmented-Reality Glasses

BACKGROUND

As current augmented-reality (AR) smart glasses are self-contained, powerful computers that project 3-dimensional holograms that can maintain their position in physical space, they could theoretically be used as a low-cost, stand-alone neuronavigation system.

OBJECTIVE

To determine feasibility and accuracy of holographic neuronavigation (HN) using AR smart glasses.

METHODS

We programmed a fully functioning neuronavigation system on commercially available smart glasses (HoloLens®, Microsoft, Redmond, Washington) and tested its accuracy and feasibility in the operating room. The fiducial registration error (FRE) was measured for both HN and conventional neuronavigation (CN) (Brainlab, Munich, Germany) by using point-based registration on a plastic head model. Subsequently, we measured HN and CN FRE on 3 patients.

RESULTS

A stereoscopic view of the holograms was successfully achieved in all experiments. In plastic head measurements, the mean HN FRE was 7.2 ± 1.8 mm compared to the mean CN FRE of 1.9 ± 0.45 (mean difference: –5.3 mm; 95% confidence interval [CI]: –6.7 to –3.9). In the 3 patients, the mean HN FRE was 4.4 ± 2.5 mm compared to the mean CN FRE of 3.6 ± 0.5 (mean difference: –0.8 mm; 95% CI: –3.0 to 4.6).

CONCLUSION

Owing to the potential benefits and promising results, we believe that HN could eventually find application in operating rooms. However, several improvements will have to be made before the device can be used in clinical practice.

The Effect of Stereoscopic Augmented Reality Visualization on Learning Anatomy and the Modifying Effect of Visual-Spatial Abilities: A Double-Center Randomized Controlled Trial

Monoscopically projected three-dimensional (3D) visualization technology may have significant disadvantages for students with lower visual-spatial abilities despite its overall effectiveness in teaching anatomy. Previous research suggests that stereopsis may facilitate a better comprehension of anatomical knowledge. This study evaluated the educational effectiveness of stereoscopic augmented reality (AR) visualization and the modifying effect of visual-spatial abilities on learning. In a double-center randomized controlled trial, first- and second-year (bio)medical undergraduates studied lower limb anatomy with stereoscopic 3D AR model (n = 20), monoscopic 3D desktop model (n = 20), or two-dimensional (2D) anatomical atlas (n = 18). Visual-spatial abilities were tested with Mental Rotation Test (MRT), Paper Folding Test (PFT), and Mechanical Reasoning (MR) Test. Anatomical knowledge was assessed by the validated 30-item paper posttest. The overall posttest scores in the stereoscopic 3D AR group (47.8%) were similar to those in the monoscopic 3D desktop group (38.5%; P = 0.240) and the 2D anatomical atlas group (50.9%; P = 1.00). When stratified by visual-spatial abilities test scores, students with lower MRT scores achieved higher posttest scores in the stereoscopic 3D AR group (49.2%) as compared to the monoscopic 3D desktop group (33.4%; P = 0.015) and similar to the scores in the 2D group (46.4%; P = 0.99). Participants with higher MRT scores performed equally well in all conditions. It is instrumental to consider an aptitude-treatment interaction caused by visual-spatial abilities when designing research into 3D learning. Further research is needed to identify contributing features and the most effective way of introducing this technology into current educational programs.

Development and Human Factors Considerations for Extended Reality Applications in Medicine: The Enhanced ELectrophysiology Visualization and Interaction System (ĒLVIS)

With the rapid expansion of hardware options in the extended realities (XRs), there has been widespread development of applications throughout many fields, including engineering, entertainment and medicine. Development of medical applications for the XRs have a unique set of considerations during development and human factors testing. Additionally, understanding the constraints of the user and the use case allow for iterative improvement. In this manuscript, the authors discuss the considerations when developing and performing human factors testing for XR applications, using the Enhanced ELectrophysiology Visualization and Interaction System (ĒLVIS) as an example. Additionally, usability and critical interpersonal interaction data from first-in-human testing of ĒLVIS are presented.

Performance Evaluation of Mixed Reality Display for Guidance During Transcatheter Cardiac Mapping and Ablation

Cardiac electrophysiology procedures present the physician with a wealth of 3D information, typically presented on fixed 2D monitors. New developments in wearable mixed reality displays offer the potential to simplify and enhance 3D visualization while providing hands-free, dynamic control of devices within the procedure room.

OBJECTIVE

This work aims to evaluate the performance and quality of a mixed reality system designed for intraprocedural use in cardiac electrophysiology.

METHOD

The Enhanced Electrophysiology Visualization and Interaction System (ĒLVIS) mixed reality system performance criteria, including image quality, hardware performance, and usability were evaluated using existing display validation procedures adapted to the electrophysiology specific use case. Additional performance and user validation were performed through a 10 patient, in-human observational study, the Engineering ĒLVIS (E2) Study.

RESULTS

The ĒLVIS system achieved acceptable frame rate, latency, and battery runtime with acceptable dynamic range and depth distortion as well as minimal geometric distortion. Bench testing results corresponded with physician feedback in the observational study, and potential improvements in geometric understanding were noted.

CONCLUSION

The ĒLVIS system, based on current commercially available mixed reality hardware, is capable of meeting the hardware performance, image quality, and usability requirements of the electroanatomic mapping display for intraprocedural, real-time use in electrophysiology procedures. Verifying off the shelf mixed reality hardware for specific clinical use can accelerate the adoption of this transformative technology and provide novel visualization, understanding, and control of clinically relevant data in real-time.