Augmented reality in neurosurgical navigation: a survey

Head pose-assisted localization of facial landmarks for enhanced fast registration in skull base surgery

In skull base surgery, the method of using a probe to draw or 3D scanners to acquire intraoperative facial point clouds for spatial registration presents several issues. Manual manipulation results in inefficiency and poor consistency. Traditional registration algorithms based on point clouds are highly dependent on the initial pose. The complexity of registration algorithms can also extend the required time. To address these issues, we used an RGB-D camera to capture real-time facial point clouds during surgery. The initial registration of the 3D model reconstructed from preoperative CT/MR images and the point cloud collected during surgery is accomplished through corresponding facial landmarks. The facial point clouds collected intraoperatively often contain rotations caused by the free-angle camera. Benefit from the close spatial geometric relationship between head pose and facial landmarks coordinates, we propose a facial landmarks localization network assisted by estimating head pose. The shared representation head pose estimation module boosts network performance by enhancing its perception of global facial features. The proposed network facilitates the localization of landmark points in both preoperative and intraoperative point clouds, enabling rapid automatic registration. A free-view human facial landmarks dataset called 3D-FVL was synthesized from clinical CT images for training. The proposed network achieves leading localization accuracy and robustness on two public datasets and the 3D-FVL. In clinical experiments, using the Artec Eva scanner, the trained network achieved a concurrent reduction in average registration time to 0.28 s, with an average registration error of 2.33 mm. The proposed method significantly reduced registration time, while meeting clinical accuracy requirements for surgical navigation. Our research will help to improving the efficiency and quality of skull base surgery.

Concurrent validity of objective assessment by finger tracking for open surgical suturing training

BACKGROUND

The aim of this study is to evaluate whether the parameters 'time' and 'distance', measured by SurgTrac, correlate with the assessment of the same skills by blinded experts.

METHODS

Basic open suturing tasks were executed by medical students. SurgTrac software measured objective parameters by tracking fingers. The executed tasks were recorded by a tablet and additionally assessed by a blinded expert with a Competency Assessment Tool-form (CAT-form). A Pearson's correlation was used to investigate the correlation between the parameters and the outcomes of the expert assessment.

RESULTS

A strong correlation between the measured parameters of SurgTrac and the expert-assessment was found for knot tying by hand (r ​= ​-0.703) and vertical mattress suture (r ​= ​-0.644) and a moderate correlation for transcutaneous suture (r ​= ​-0.555) and intracutaneous suture (r ​= ​-0.451).

CONCLUSION

The use of finger tracking by SurgTrac showed a good concurrent validity for the basic open suturing tasks knot tying by hand, transcutaneous suture and vertical mattress suture.

Exploring the Potential Use of Virtual Reality with a Supraorbital Keyhole Craniotomy for Anterior Skull Base Meningiomas: Two Case Reports

INTRODUCTION

A supraorbital keyhole craniotomy (SOKC) is a novel alternative to frontal craniotomies for accessing the anterior fossa for resecting tumors and clipping aneurysms; however, its implementation is limited in patients at a high risk of complications. We present two cases involving the use of augmented reality (AR) and virtual reality (VR) for patient selection and preoperative planning for a supraorbital tumor resection of anterior fossa meningiomas.

METHODS

This is a prospective, single-center case series at a research institute. We identified patients with an anterior or middle fossa meningioma regardless of age, gender, and tumor characteristics who could undergo an SOKC and MRI. The preoperative planning was performed with the BrainLab Magic Leap AR/VR platform. The meningiomas were resected through the SOKC under neuronavigation.

RESULTS

We identified two cases: a 37-year-old male with a meningioma in the sellar region and an 84-year-old male with a right anterior fossa meningioma, both confirmed by MRI. Both patients had a complete tumor resection by a minimally invasive SOKC after preoperative planning with the AR/VR platform. Postoperatively, hyponatremia complicated the first case, while the second case developed an intracranial hemorrhage. They both recovered after the appropriate interventions.

CONCLUSIONS

The use of an SOKC for anterior skull base meningiomas should be individualized after considering the lesion characteristics, vascular control needs, and the surgeon's expertise. VR/AR-assisted preoperative evaluation and planning will optimize the patient selection and surgical outcomes. We can utilize VR/AR technologies to identify patients that will benefit from an SOKC and expand the implementation of the approach beyond its current limitations.

The bibliometric analysis of extended reality in surgical training: Global and Chinese perspective

Objectives

The prospect of extended reality (XR) being integrated with surgical training curriculum has attracted scholars. However, there is a lack of bibliometric analysis to help them better understand this field. Our aim is to analyze relevant literature focusing on development trajectory and research directions since the 21st century to provide valuable insights.

Methods

Papers were retrieved from the Web of Science Core Collection. Microsoft Excel, VOSviewer, and CiteSpace were used for bibliometric analysis.

Results

Of the 3337 papers published worldwide, China contributed 204, ranking fifth. The world's enthusiasm for this field has been growing since 2000, whereas China has been gradually entering since 2001. Although China had a late start, its growth has accelerated since around 2016 due to the reform of the medical postgraduate education system and the rapid development of Chinese information technology, despite no research explosive period has been yet noted. International institutions, notably the University of Toronto, worked closely with others, while Chinese institutions lacked of international and domestic cooperation. Sixteen stable cooperation clusters of international scholars were formed, while the collaboration between Chinese scholars was not yet stable. XR has been primarily applied in orthopedic surgery, cataract surgery, laparoscopic training and intraoperative use in neurosurgery worldwide.

Conclusions

There is strong enthusiasm and cooperation in the international research on the XR-based surgical training. Chinese scholars are making steady progress and have great potential in this area. There has not been noted an explosive research phase yet in the Chinese pace. The research on several surgical specialties has been summarized at the very first time. AR will gradually to be more involved and take important role of the research.

The Development And Applications Of Augmented And Virtual Reality Technology In Spine Surgery Training: A Systematic Review

BACKGROUND

The COVID-19 pandemic has accelerated the growing global interest in the role of augmented and virtual reality in surgical training. While this technology grows at a rapid rate, its efficacy remains unclear. To that end, we offer a systematic review of the literature summarizing the role of virtual and augmented reality on spine surgery training.

METHODS

A systematic review of the literature was conducted on May 13th, 2022. PubMed, Web of Science, Medline, and Embase were reviewed for relevant studies. Studies from both orthopedic and neurosurgical spine programs were considered. There were no restrictions placed on the type of study, virtual/augmented reality modality, nor type of procedure. Qualitative data analysis was performed, and all studies were assigned a Medical Education Research Study Quality Instrument (MERSQI) score.

RESULTS

The initial review identified 6752 studies, of which 16 were deemed relevant and included in the final review, examining a total of nine unique augmented/virtual reality systems. These studies had a moderate methodological quality with a MERSQI score of 12.1 + 1.8; most studies were conducted at single-center institutions, and unclear response rates. Statistical pooling of the data was limited by the heterogeneity of the study designs.

CONCLUSION

This review examined the applications of augmented and virtual reality systems for training residents in various spine procedures. As this technology continues to advance, higher-quality, multi-center, and long-term studies are required to further the adaptation of VR/AR technologies in spine surgery training programs.

Mixed Reality for Cranial Neurosurgical Planning: A Single-Center Applicability Study With the First 107 Subsequent Holograms

BACKGROUND AND OBJECTIVES

Mixed reality (MxR) benefits neurosurgery by improving anatomic visualization, surgical planning and training. We aim to validate the usability of a dedicated certified system for this purpose.

METHODS

All cases prepared with MxR in our center in 2022 were prospectively collected. Holographic rendering was achieved using an incorporated fully automatic algorithm in the MxR application, combined with contrast-based semiautomatic rendering and/or manual segmentation where necessary. Hologram segmentation times were documented. Visualization during surgical preparation (defined as the interval between finalized anesthesiological induction and sterile draping) was performed using MxR glasses and direct streaming to a side screen. Surgical preparation times were compared with a matched historical cohort of 2021. Modifications of the surgical approach after 3-dimensional (3D) visualization were noted. Usability was assessed by evaluating 7 neurosurgeons with more than 3 months of experience with the system using a Usefulness, Satisfaction and Ease of use (USE) questionnaire.

RESULTS

One hundred-seven neurosurgical cases prepared with a 3D hologram were collected. Surgical indications were oncologic (63/107, 59%), cerebrovascular (27/107, 25%), and carotid endarterectomy (17/107, 16%). Mean hologram segmentation time was 39.4 ± 20.4 minutes. Average surgical preparation time was 48.0 ± 17.3 minutes for MxR cases vs 52 ± 17 minutes in the matched 2021 cohort without MxR (mean difference 4, 95% CI 1.7527-9.7527). Based on the 3D hologram, the surgical approach was modified in 3 cases. Good usability was found by 57% of the users.

CONCLUSION

The perioperative use of 3D holograms improved direct anatomic visualization while not significantly increasing intraoperative surgical preparation time. Usability of the system was adequate. Further technological development is necessary to improve the automatic algorithms and reduce the preparation time by circumventing manual and semiautomatic segmentation. Future studies should focus on quantifying the potential benefits in teaching, training, and the impact on surgical and functional outcomes.

Augmented and virtual reality usage in awake craniotomy: a systematic review

Augmented and virtual reality (AR, VR) are becoming promising tools in neurosurgery. AR and VR can reduce challenges associated with conventional approaches via the simulation and mimicry of specific environments of choice for surgeons. Awake craniotomy (AC) enables the resection of lesions from eloquent brain areas while monitoring higher cortical and subcortical functions. Evidence suggests that both surgeons and patients benefit from the various applications of AR and VR in AC. This paper investigates the application of AR and VR in AC and assesses its prospective utility in neurosurgery. A systematic review of the literature was performed using PubMed, Scopus, and Web of Science databases in accordance with the PRISMA guidelines. Our search results yielded 220 articles. A total of six articles consisting of 118 patients have been included in this review. VR was used in four papers, and the other two used AR. Tumour was the most common pathology in 108 patients, followed by vascular lesions in eight patients. VR was used for intraoperative mapping of language, vision, and social cognition, while AR was incorporated in preoperative training of white matter dissection and intraoperative visualisation and navigation. Overall, patients and surgeons were satisfied with the applications of AR and VR in their cases. AR and VR can be safely incorporated during AC to supplement, augment, or even replace conventional approaches in neurosurgery. Future investigations are required to assess the feasibility of AR and VR in various phases of AC.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

Augmented reality visualization in brain lesions: a prospective randomized controlled evaluation of its potential and current limitations in navigated microneurosurgery

Background

Augmented reality (AR) has the potential to support complex neurosurgical interventions by including visual information seamlessly. This study examines intraoperative visualization parameters and clinical impact of AR in brain tumor surgery.

Methods

Fifty-five intracranial lesions, operated either with AR-navigated microscope (n = 39) or conventional neuronavigation (n = 16) after randomization, have been included prospectively. Surgical resection time, duration/type/mode of AR, displayed objects (n, type), pointer-based navigation checks (n), usability of control, quality indicators, and overall surgical usefulness of AR have been assessed.

Results

AR display has been used in 44.4% of resection time. Predominant AR type was navigation view (75.7%), followed by target volumes (20.1%). Predominant AR mode was picture-in-picture (PiP) (72.5%), followed by 23.3% overlay display. In 43.6% of cases, vision of important anatomical structures has been partially or entirely blocked by AR information. A total of 7.7% of cases used MRI navigation only, 30.8% used one, 23.1% used two, and 38.5% used three or more object segmentations in AR navigation. A total of 66.7% of surgeons found AR visualization helpful in the individual surgical case. AR depth information and accuracy have been rated acceptable (median 3.0 vs. median 5.0 in conventional neuronavigation). The mean utilization of the navigation pointer was 2.6 × /resection hour (AR) vs. 9.7 × /resection hour (neuronavigation); navigation effort was significantly reduced in AR (P < 0.001).

Conclusions

The main benefit of HUD-based AR visualization in brain tumor surgery is the integrated continuous display allowing for pointer-less navigation. Navigation view (PiP) provides the highest usability while blocking the operative field less frequently. Visualization quality will benefit from improvements in registration accuracy and depth impression.

German clinical trials registration number.

DRKS00016955.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00701-021-05045-1.

Development of an inside-out augmented reality technique for neurosurgical navigation

OBJECTIVE

With the advancement of 3D modeling techniques and visualization devices, augmented reality (AR)-based navigation (AR navigation) is being developed actively. The authors developed a pilot model of their newly developed inside-out tracking AR navigation system.

METHODS

The inside-out AR navigation technique was developed based on the visual inertial odometry (VIO) algorithm. The Quick Response (QR) marker was created and used for the image feature-detection algorithm. Inside-out AR navigation works through the steps of visualization device recognition, marker recognition, AR implementation, and registration within the running environment. A virtual 3D patient model for AR rendering and a 3D-printed patient model for validating registration accuracy were created. Inside-out tracking was used for the registration. The registration accuracy was validated by using intuitive, visualization, and quantitative methods for identifying coordinates by matching errors. Fine-tuning and opacity-adjustment functions were developed.

RESULTS

ARKit-based inside-out AR navigation was developed. The fiducial marker of the AR model and those of the 3D-printed patient model were correctly overlapped at all locations without errors. The tumor and anatomical structures of AR navigation and the tumors and structures placed in the intracranial space of the 3D-printed patient model precisely overlapped. The registration accuracy was quantified using coordinates, and the average moving errors of the x-axis and y-axis were 0.52 ± 0.35 and 0.05 ± 0.16 mm, respectively. The gradients from the x-axis and y-axis were 0.35° and 1.02°, respectively. Application of the fine-tuning and opacity-adjustment functions was proven by the videos.

CONCLUSIONS

The authors developed a novel inside-out tracking-based AR navigation system and validated its registration accuracy. This technical system could be applied in the novel navigation system for patient-specific neurosurgery.