A Holographic Augmented Reality Interface for Visualizing of MRI Data and Planning of Neurosurgical Procedures

HyperMRI: hyperspectral and magnetic resonance fusion methodology for neurosurgery applications

PURPOSE

Magnetic resonance imaging (MRI) is a common technique in image-guided neurosurgery (IGN). Recent research explores the integration of methods like ultrasound and tomography, among others, with hyperspectral (HS) imaging gaining attention due to its non-invasive real-time tissue classification capabilities. The main challenge is the registration process, often requiring manual intervention. This work introduces an automatic, markerless method for aligning HS images with MRI.

METHODS

This work presents a multimodal system that combines RGB-Depth (RGBD) and HS cameras. The RGBD camera captures the patient's facial geometry, which is used for registration with the preoperative MR through ICP. Once MR-depth registration is complete, the integration of HS data is achieved using a calibrated homography transformation. The incorporation of external tracking with a novel calibration method allows camera mobility from the registration position to the craniotomy area. This methodology streamlines the fusion of RGBD, HS and MR images within the craniotomy area.

RESULTS

Using the described system and an anthropomorphic phantom head, the system has been characterised by registering the patient's face in 25 positions and 5 positions resulted in a fiducial registration error of 1.88 ± 0.19 mm and a target registration error of 4.07 ± 1.28 mm, respectively.

CONCLUSIONS

This work proposes a new methodology to automatically register MR and HS information with a sufficient accuracy. It can support the neurosurgeons to guide the diagnosis using multimodal data over an augmented reality representation. However, in its preliminary prototype stage, this system exhibits significant promise, driven by its cost-effectiveness and user-friendly design.

Evaluation Metrics for Augmented Reality in Neurosurgical Preoperative Planning, Surgical Navigation, and Surgical Treatment Guidance: A Systematic Review

BACKGROUND AND OBJECTIVE

Recent years have shown an advancement in the development of augmented reality (AR) technologies for preoperative visualization, surgical navigation, and intraoperative guidance for neurosurgery. However, proving added value for AR in clinical practice is challenging, partly because of a lack of standardized evaluation metrics. We performed a systematic review to provide an overview of the reported evaluation metrics for AR technologies in neurosurgical practice and to establish a foundation for assessment and comparison of such technologies.

METHODS

PubMed, Embase, and Cochrane were searched systematically for publications on assessment of AR for cranial neurosurgery on September 22, 2022. The findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

RESULTS

The systematic search yielded 830 publications; 114 were screened full text, and 80 were included for analysis. Among the included studies, 5% dealt with preoperative visualization using AR, with user perception as the most frequently reported metric. The majority (75%) researched AR technology for surgical navigation, with registration accuracy, clinical outcome, and time measurements as the most frequently reported metrics. In addition, 20% studied the use of AR for intraoperative guidance, with registration accuracy, task outcome, and user perception as the most frequently reported metrics.

CONCLUSION

For quality benchmarking of AR technologies in neurosurgery, evaluation metrics should be specific to the risk profile and clinical objectives of the technology. A key focus should be on using validated questionnaires to assess user perception; ensuring clear and unambiguous reporting of registration accuracy, precision, robustness, and system stability; and accurately measuring task performance in clinical studies. We provided an overview suggesting which evaluation metrics to use per AR application and innovation phase, aiming to improve the assessment of added value of AR for neurosurgical practice and to facilitate the integration in the clinical workflow.

Augmented reality during parotid surgery: real-life evaluation of voice control of a head mounted display

PURPOSE

Augmented Reality can improve surgical planning and performance in parotid surgery. For easier application we implemented a voice control manual for our augmented reality system. The aim of the study was to evaluate the feasibility of the voice control in real-life situations.

METHODS

We used the HoloLens 1^® (Microsoft Corporation) with a special speech recognition software for parotid surgery. The evaluation took place in a audiometry cubicle and during real surgical procedures. Voice commands were used to display various 3D structures of the patient with the HoloLens 1^®. Commands had different variations (male/female, 65 dB SPL)/louder, various structures).

RESULTS

In silence, 100% of commands were recognized. If the volume of the operation room (OR) background noise exceeds 42 dB, the recognition rate decreases significantly, and it drops below 40% at > 60 dB SPL. With constant speech volume at 65 dB SPL male speakers had a significant better recognition rate than female speakers (p = 0.046). Higher speech volumes can compensate this effect. The recognition rate depends on the type of background noise. Mixed OR noise (52 dB(A)) reduced the detection rate significantly compared to single suction noise at 52 dB(A) (p ≤ 0.00001). The recognition rate was significantly better in the OR than in the audio cubicle (p = 0.00013 both genders, 0.0086 female, and 0.0036 male).

CONCLUSIONS

The recognition rate of voice commands can be enhanced by increasing the speech volume and by singularizing ambient noises. The detection rate depends on the loudness of the OR noise. Male voices are understood significantly better than female voices.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Holographic visualization for stereotactic neurosurgery research

Background:

Stereotactic neurosurgical planning for the placement of depth electrodes requires the integration of wide-ranging 3D datasets on the anatomy of the patient.

Objective:

Our goal was to create an interactive group-based holographic visualization tool (HoloSNS) that facilitates evaluation of depth electrode positioning relative to the available medical imaging data, as well as models of the anatomical nuclei and structural connectivity of the brain.

Methods:

HoloSNS is currently designed to run on the HoloLens 2 platform, and was developed using the Unity Game Engine and the Mixed Reality Toolkit from Microsoft.

Results:

HoloSNS currently supports research analyses with deep brain stimulation (DBS) and/or stereo-electroencephalography (SEEG) electrodes. Two example software applications (HoloDBS and HoloSEEG) are available for free download on the Microsoft App Store.

Conclusions:

HoloSNS is the latest culmination of our efforts to integrate advances in brain imaging data, intracranial electrode modeling, and advanced visualization techniques to enhance stereotactic neurosurgery research.

Use of Mixed Reality in Neuro-Oncology: A Single Centre Experience

(1) Background: Intra-operative neuronavigation is currently an essential component to most neurosurgical operations. Recent progress in mixed reality (MR) technology has attempted to overcome the disadvantages of the neuronavigation systems. We present our experience using the HoloLens 2 in neuro-oncology for both intra- and extra-axial tumours. (2) Results: We describe our experience with three patients who underwent tumour resection. We evaluated surgeon experience, accuracy of superimposed 3D image in tumour localisation with standard neuronavigation both pre- and intra-operatively. Surgeon training and usage for HoloLens 2 was short and easy. The process of image overlay was relatively straightforward for the three cases. Registration in prone position with a conventional neuronavigation system is often difficult, which was easily overcome during use of HoloLens 2. (3) Conclusion: Although certain limitations were identified, the authors feel that this system is a feasible alternative device for intra-operative visualization of neurosurgical pathology. Further studies are being planned to assess its accuracy and suitability across various surgical disciplines.

A systematic comparison of the accuracy of monocular RGB tracking and LiDAR for neuronavigation

With the advent of augmented reality (AR), the use of AR-guided systems in the field of medicine has gained traction. However, the wide-scale adaptation of these systems requires highly accurate and reliable tracking. In this work, the tracking accuracy of two technology platforms, LiDAR and Vuforia, are developed and rigorously tested for a catheter placement neurological procedure. Several experiments (900) are performed for each technology across various combinations of catheter lengths and insertion trajectories. This analysis shows that the LiDAR platform outperformed Vuforia; which is the state-of-the-art in monocular RGB tracking solutions. LiDAR had 75% less radial distance error and 26% less angle deviation error. Results provide key insights into the value and utility of LiDAR-based tracking in AR guidance systems.

Virtual Reality as Tool for Bioprinting Quality Inspection: A Proof of Principle

As virtual reality (VR) has drastically evolved over the past few years, the field of applications of VR flourished way beyond the gaming industry. While commercial VR solutions might be available, there is a need to develop a workflow for specific applications. Bioprinting represents such an example. Here, complex 3D data is generated and needs to be visualized in the context of quality control. We demonstrate that the transfer to a commercially available VR software is possible by introducing an optimized workflow. In the present work, we developed a workflow for the visualization of the critical quality attribute (cQA) cell distribution in bioprinted (extrusion-based) samples in VR. The cQA cell distribution is directly influenced by the pre-processing step mixing of cell material in the bioink. Magnetic Resonance Imaging (MRI) was used as an analytical tool to generate spatially resolved 2.5 and 3D data of the bioprinted objects. A sample with poor quality in respect of the cQA cell distribution was identified as its inhomogeneous cell distribution could be displayed spatially resolved in VR. The described workflow facilitates the usage of VR as a tool for quality inspection in the field of bioprinting and represents a powerful tool for visualization of complex 3D MRI data.

A Dedicated Tool for Presurgical Mapping of Brain Tumors and Mixed-Reality Navigation During Neurosurgery

Brain tumor surgery requires a delicate tradeoff between complete removal of neoplastic tissue while minimizing loss of brain function. Functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) have emerged as valuable tools for non-invasive assessment of human brain function and are now used to determine brain regions that should be spared to prevent functional impairment after surgery. However, image analysis requires different software packages, mainly developed for research purposes and often difficult to use in a clinical setting, preventing large-scale diffusion of presurgical mapping. We developed a specialized software able to implement an automatic analysis of multimodal MRI presurgical mapping in a single application and to transfer the results to the neuronavigator. Moreover, the imaging results are integrated in a commercially available wearable device using an optimized mixed-reality approach, automatically anchoring 3-dimensional holograms obtained from MRI with the physical head of the patient. This will allow the surgeon to virtually explore deeper tissue layers highlighting critical brain structures that need to be preserved, while retaining the natural oculo-manual coordination. The enhanced ergonomics of this procedure will significantly improve accuracy and safety of the surgery, with large expected benefits for health care systems and related industrial investors.