SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality

Volume Rendering of CT Images to Inform Closure of Complex Ventricular Septal Defects

The anatomy of structurally complex ventricular septal defects (SC-VSD) can be difficult to assess using 2-dimensional (2D) images and traditional 2D multiplanar views of 3D images. Direct identification and visualization are not always possible via the standard tricuspid approach in surgical repair. Volume rendering is a near instant method for 3D visualization of computed tomography angiography images, but application of this method to the planning of SC-VSD closure has not been described. We describe the integration of virtual patch and device placement within volume-rendered computed tomography angiography images within SlicerHeart to inform surgical and transcatheter closure of SC-VSDs in 3 patients. Virtual heart models were created and examined by a multidisciplinary team. Virtual device placement and surgical patch design was applied to better inform patient candidacy and procedural planning. Volume rendering-based visualization of SC-VSDs is feasible and may inform understanding of anatomy and conceptualization of the optimal repair. Further study is needed to demonstrate improvement in outcomes.

PreVISE: an efficient virtual reality system for SEEG surgical planning

Epilepsy is a neurological disorder characterized by recurring seizures that can cause a wide range of symptoms. Stereo-electroencephalography (SEEG) is a diagnostic procedure where multiple electrodes are stereotactically implanted within predefined brain regions to identify the seizure onset zone, which needs to be surgically removed or disconnected to achieve remission of focal epilepsy. This procedure is complex and challenging due to two main reasons. First, as electrode placement requires good accuracy in desired brain regions, excellent knowledge and understanding of the 3D brain anatomy is required. Second, as typically multiple SEEG electrodes need to be implanted, the positioning of intracerebral electrodes must avoid critical structures (e.g., blood vessels) to ensure patient safety. Traditional SEEG surgical planning relies on 2D display of multi-contrast volumetric medical imaging data, and places a high cognitive demand for surgeons' spatial understanding, resulting in potentially sub-optimal surgical plans and extensive planning time (~ 15 min per electrode). In contrast, virtual reality (VR) presents an intuitive and immersive approach that can offer more intuitive visualization of 3D data as well as potentially enhanced efficiency for neurosurgical planning. Unfortunately, existing VR systems for SEEG surgery only focus on the visualization of post-surgical scans to confirm electrode placement. To address the need, we introduce the first VR system for SEEG planning that integrates user-friendly and efficient visualization and interaction strategies while providing real-time feedback metrics, including distances to nearest blood vessels, angles of insertion, and the overall surgical quality scores. The system reduces the surgical planning time by 91%.

Role of eXtended Reality use in medical imaging interpretation for pre-surgical planning and intraoperative augmentation

Purpose

eXtended Reality (XR) technology, including virtual reality (VR), augmented reality (AR), and mixed reality (MR), is a growing field in healthcare. Each modality offers unique benefits and drawbacks for medical education, simulation, and clinical care. We review current studies to understand how XR technology uses medical imaging to enhance surgical diagnostics, planning, and performance. We also highlight current limitations and future directions.

Approach

We reviewed the literature on immersive XR technologies for surgical planning and intraoperative augmentation, excluding studies on telemedicine and 2D video-based training. We cited publications highlighting XR's advantages and limitations in these categories.

Results

A review of 556 papers on XR for medical imaging in surgery yielded 155 relevant papers reviewed utilizing the aid of chatGPT. XR technology may improve procedural times, reduce errors, and enhance surgical workflows. It aids in preoperative planning, surgical navigation, and real-time data integration, improving surgeon ergonomics and enabling remote collaboration. However, adoption faces challenges such as high costs, infrastructure needs, and regulatory hurdles. Despite these, XR shows significant potential in advancing surgical care.

Conclusions

Immersive technologies in healthcare enhance visualization and understanding of medical conditions, promising better patient outcomes and innovative treatments but face adoption challenges such as cost, technological constraints, and regulatory hurdles. Addressing these requires strategic collaborations and improvements in image quality, hardware, integration, and training.

ColocZStats: a z-stack signal colocalization extension tool for 3D slicer

Confocal microscopy has evolved to be a widely adopted imaging technique in molecular biology and is frequently utilized to achieve accurate subcellular localization of proteins. Applying colocalization analysis on image z-stacks obtained from confocal fluorescence microscopes is a dependable method of revealing the relationship between different molecules. In addition, despite the established advantages and growing adoption of 3D visualization software in various microscopy research domains, there have been few systems that can support colocalization analysis within a user-specified region of interest (ROI). In this context, several broadly employed biological image visualization platforms are meticulously explored in this study to understand the current landscape. It has been observed that while these applications can generate three-dimensional (3D) reconstructions for z-stacks, and in some cases transfer them into an immersive virtual reality (VR) scene, there is still little support for performing quantitative colocalization analysis on such images based on a user-defined ROI and thresholding levels. To address these issues, an extension called ColocZStats (pronounced Coloc-Zee-Stats) has been developed for 3D Slicer, a widely used free and open-source software package for image analysis and scientific visualization. With a custom-designed user-friendly interface, ColocZStats allows investigators to conduct intensity thresholding and ROI selection on imported 3D image stacks. It can deliver several essential colocalization metrics for structures of interest and produce reports in the form of diagrams and spreadsheets.

Design and development of a personalized virtual reality-based training system for vascular intervention surgery

BACKGROUND AND OBJECTIVES

Virtual training has emerged as an exceptionally effective approach for training healthcare practitioners in the field of vascular intervention surgery. By providing a simulated environment and blood vessel model that enables repeated practice, virtual training facilitates the acquisition of surgical skills in a safe and efficient manner for trainees. However, the current state of research in this area is characterized by limitations in the fidelity of blood vessel and guidewire models, which restricts the effectiveness of training. Additionally, existing approaches lack the necessary real-time responsiveness and precision, while the blood vessel models suffer from incompleteness and a lack of scientific rigor.

METHODS

To address these challenges, this paper integrates position-based dynamics (PBD) and its extensions, shape matching, and Cosserat elastic rods. By combining these approaches within a unified particle framework, accurate and realistic deformation simulation of personalized blood vessel and guidewire models is achieved, thereby enhancing the training experience. Furthermore, a multi-level progressive continuous collision detection method, leveraging spatial hashing, is proposed to improve the accuracy and efficiency of collision detection.

RESULTS

Our proposed blood vessel model demonstrated acceptable performance with the reduced deformation simulation response times of 7 ms, improving the real-time capability at least of 43.75 %. Experimental validation confirmed that the guidewire model proposed in this paper can dynamically adjust the density of its elastic rods to alter the degree of bending and torsion. It also exhibited a deformation process comparable to that of real guidewires, with an average response time of 6 ms. In the interaction of blood vessel and guidewire models, the simulator blood vessel model used for coronary vascular intervention training exhibited an average response time of 15.42 ms, with a frame rate of approximately 64 FPS.

CONCLUSIONS

The method presented in this paper achieves deformation simulation of both vascular and guidewire models, demonstrating sufficient real-time performance and accuracy. The interaction efficiency between vascular and guidewire models is enhanced through the unified simulation framework and collision detection. Furthermore, it can be integrated with virtual training scenarios within the system, making it suitable for developing more advanced vascular interventional surgery training systems.

CyPVICS: A framework to prevent or minimise cybersickness in immersive virtual clinical simulation

Cybersickness is a global issue affecting users of immersive virtual reality. However, there is no agreement on the exact cause of cybersickness. Taking into consideration how it can differ greatly from one person to another, it makes it even more difficult to determine the exact cause or find a solution. Because cybersickness excludes so many prospective users, including healthcare professionals, from using immersive virtual reality as a learning tool, this research sought to find solutions in existing literature and construct a framework that can be used to prevent or minimise cybersickness during immersive virtual clinical simulation (CyPVICS). The Bestfit Framework by Carrol and authors were used to construct the CyPVICS framework. The process started by conducting two separate literature searchers using the BeHEMoTh (for models, theories, and frameworks) and SPIDER (for primary research articles) search techniques. Once the literature searches were completed the models, theories and framework were used to construct a priori framework. The models' theories and frameworks were analysed to determine aspects relevant to causes, reducing, eliminating, and detecting cybersickness. The priori framework was expanded by, first coding the findings of the primary research study into the existing aspects of the priori framework. Once coded the aspects that could not be coded were added in the relevant category, for example causes. After reviewing 1567 abstracts and titles as part of the BeHEMoTh search string,19 full text articles, a total of 15 papers containing models, theories, and frameworks, were used to construct the initial CyPVICS framework. Once the initial CyPVICS was created, a total 904 primary research studies (SPIDER) were evaluated, based on their titles and abstracts, of which 100 were reviewed in full text. In total, 67 articles were accepted and coded to expand the initial CyPVICS framework. This paper presents the CyPVICS framework for use, not only in health professions' education, but also in other disciplines, since the incorporated models, theories, frameworks, and primary research studies were not specific to virtual clinical simulation.

Virtual reality-based surgical planning simulator for tumorous resection in FreeForm Modeling: an illustrative case of clinical teaching

The importance of virtual reality (VR) has been emphasized by many medical studies, yet it has been relatively under-applied to surgical operation. This study characterized how VR has been applied in clinical education and evaluated its tutorial utility by designing a surgical model of tumorous resection as a simulator for preoperative planning and medical tutorial. A 36-year-old male patient with a femoral tumor who was admitted to the Affiliated Jiangmen Traditional Chinese Medicine Hospital was randomly selected and scanned by computed tomography (CT). The data in digital imaging and communications in medicine (*.DICOM) format were imported into Mimics to reconstruct a femoral model, and were generated to the format of *.stl executing in the computer-aided design (CAD) software SenSable FreeForm Modeling (SFM). A bony tumor was simulated by adding clay to the femur, the procedure of tumorous resection was virtually performed with a toolkit called Phantom, and its bony defect was filled with virtual cement. A 3D workspace was created to enable the individual multimodality manipulation, and a virtual operation of tumorous excision was successfully carried out with indefinitely repeated running. The precise delineation of surgical margins was shown to be achieved with expert proficiency and inexperienced hands among 43 of 50 participants. This simulative educator presented an imitation of high definition, those trained by VR models achieved a higher success rate of 86% than the rate of 74% achieved by those trained by conventional methods. This tumorous resection was repeatably handled by SFM, including the establishment of surgical strategy, whereby participants felt that respondent force feedback was beneficial to surgical teaching programs, enabling engagement of learning experiences by immersive events which mimic real-world circumstances to reinforce didactic and clinical concepts.

Mixed Reality Platforms in Telehealth Delivery: Scoping Review

BACKGROUND

The distinctive features of the digital reality platforms, namely augmented reality (AR), virtual reality (VR), and mixed reality (MR) have extended to medical education, training, simulation, and patient care. Furthermore, this digital reality technology seamlessly merges with information and communication technology creating an enriched telehealth ecosystem. This review provides a composite overview of the prospects of telehealth delivered using the MR platform in clinical settings.

OBJECTIVE

This review identifies various clinical applications of high-fidelity digital display technology, namely AR, VR, and MR, delivered using telehealth capabilities. Next, the review focuses on the technical characteristics, hardware, and software technologies used in the composition of AR, VR, and MR in telehealth.

METHODS

We conducted a scoping review using the methodological framework and reporting design using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. Full-length articles in English were obtained from the Embase, PubMed, and Web of Science databases. The search protocol was based on the following keywords and Medical Subject Headings to obtain relevant results: "augmented reality," "virtual reality," "mixed-reality," "telemedicine," "telehealth," and "digital health." A predefined inclusion-exclusion criterion was developed in filtering the obtained results and the final selection of the articles, followed by data extraction and construction of the review.

RESULTS

We identified 4407 articles, of which 320 were eligible for full-text screening. A total of 134 full-text articles were included in the review. Telerehabilitation, telementoring, teleconsultation, telemonitoring, telepsychiatry, telesurgery, and telediagnosis were the segments of the telehealth division that explored the use of AR, VR, and MR platforms. Telerehabilitation using VR was the most commonly recurring segment in the included studies. AR and MR has been mainly used for telementoring and teleconsultation. The most important technical features of digital reality technology to emerge with telehealth were virtual environment, exergaming, 3D avatars, telepresence, anchoring annotations, and first-person viewpoint. Different arrangements of technology-3D modeling and viewing tools, communication and streaming platforms, file transfer and sharing platforms, sensors, high-fidelity displays, and controllers-formed the basis of most systems.

CONCLUSIONS

This review constitutes a recent overview of the evolving digital AR and VR in various clinical applications using the telehealth setup. This combination of telehealth with AR, VR, and MR allows for remote facilitation of clinical expertise and further development of home-based treatment. This review explores the rapidly growing suite of technologies available to users within the digital health sector and examines the opportunities and challenges they present.

Leading Transformation in Medical Education Through Extended Reality

Extended reality (XR) has exponentially developed over the past decades to incorporate technology whereby users can visualise, explore, and interact with 3-dimensional-generated computer environments, and superimpose virtual reality (VR) onto real-world environments, thus displaying information and data on various levels of the reality-virtuality continuum. In the context of medicine, VR tools allow for anatomical assessment and diagnosis, surgical training through lifelike procedural simulations, planning of surgeries and biopsies, intraprocedural guidance, and medical education. The following chapter aims to provide an overview of the currently available evidence and perspectives on the application of XR within medical education. It will focus on undergraduate and postgraduate teaching, medical education within Low-Middle Income Countries, key practical steps in implementing a successful XR programme, and the limitations and future of extended reality within medical education.

MiCellAnnGELo: annotate microscopy time series of complex cell surfaces with 3D virtual reality

SUMMARY

Advances in 3D live cell microscopy are enabling high-resolution capture of previously unobserved processes. Unleashing the power of modern machine learning methods to fully benefit from these technologies is, however, frustrated by the difficulty of manually annotating 3D training data. MiCellAnnGELo virtual reality software offers an immersive environment for viewing and interacting with 4D microscopy data, including efficient tools for annotation. We present tools for labelling cell surfaces with a wide range of applications, including cell motility, endocytosis and transmembrane signalling.

AVAILABILITY AND IMPLEMENTATION

MiCellAnnGELo employs the cross-platform (Mac/Unix/Windows) Unity game engine and is available under the MIT licence at https://github.com/CellDynamics/MiCellAnnGELo.git, together with sample data. MiCellAnnGELo can be run in desktop mode on a 2D screen or in 3D using a standard VR headset with a compatible GPU.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

SlicerHeart: An open-source computing platform for cardiac image analysis and modeling

Cardiovascular disease is a significant cause of morbidity and mortality in the developed world. 3D imaging of the heart's structure is critical to the understanding and treatment of cardiovascular disease. However, open-source tools for image analysis of cardiac images, particularly 3D echocardiographic (3DE) data, are limited. We describe the rationale, development, implementation, and application of SlicerHeart, a cardiac-focused toolkit for image analysis built upon 3D Slicer, an open-source image computing platform. We designed and implemented multiple Python scripted modules within 3D Slicer to import, register, and view 3DE data, including new code to volume render and crop 3DE. In addition, we developed dedicated workflows for the modeling and quantitative analysis of multi-modality image-derived heart models, including heart valves. Finally, we created and integrated new functionality to facilitate the planning of cardiac interventions and surgery. We demonstrate application of SlicerHeart to a diverse range of cardiovascular modeling and simulation including volume rendering of 3DE images, mitral valve modeling, transcatheter device modeling, and planning of complex surgical intervention such as cardiac baffle creation. SlicerHeart is an evolving open-source image processing platform based on 3D Slicer initiated to support the investigation and treatment of congenital heart disease. The technology in SlicerHeart provides a robust foundation for 3D image-based investigation in cardiovascular medicine.

Clinical 3D modeling to guide pediatric cardiothoracic surgery and intervention using 3D printed anatomic models, computer aided design and virtual reality

BACKGROUND

Surgical and catheter-based interventions for congenital heart disease require precise understanding of complex anatomy. The use of three-dimensional (3D) printing and virtual reality to enhance visuospatial understanding has been well documented, but integration of these methods into routine clinical practice has not been well described. We review the growth and development of a clinical 3D modeling service to inform procedural planning within a high-volume pediatric heart center.

METHODS

Clinical 3D modeling was performed using cardiac magnetic resonance (CMR) or computed tomography (CT) derived data. Image segmentation and post-processing was performed using FDA-approved software. Patient-specific anatomy was visualized using 3D printed models, digital flat screen models and virtual reality. Surgical repair options were digitally designed using proprietary and open-source computer aided design (CAD) based modeling tools.

RESULTS

From 2018 to 2020 there were 112 individual 3D modeling cases performed, 16 for educational purposes and 96 clinically utilized for procedural planning. Over the 3-year period, demand for clinical modeling tripled and in 2020, 3D modeling was requested in more than one-quarter of STAT category 3, 4 and 5 cases. The most common indications for modeling were complex biventricular repair (n = 30, 31%) and repair of multiple ventricular septal defects (VSD) (n = 11, 12%).

CONCLUSIONS

Using a multidisciplinary approach, clinical application of 3D modeling can be seamlessly integrated into pre-procedural care for patients with congenital heart disease. Rapid expansion and increased demand for utilization of these tools within a high-volume center demonstrate the high value conferred on these techniques by surgeons and interventionalists alike.

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

A Predictable Approach of a Rare and Frequently Misdiagnosed Entity: Laryngeal Nerve Schwannoma

(1) Background: Schwannoma, a mesenchymal neoplasm derived from Schwann cells that line peripheral nerve sheaths, has a challenging diagnosis, due to the non-specific medical history and clinical examination. Nowadays, virtual reality (VR) is increasingly more used for enhancing diagnosis and for preoperative planning of surgical procedures. With VR, the surgeon can interact, before any surgery, with a virtual environment that is completely generated by a computer, offering them a real experience inside a virtual 3D model. (2) Methods and Results: The aim of the present paper was to present a case of surgically removal of a schwannoma, which originated from the fibers of the superior laryngeal nerve, in a predictable and minimally invasive fashion, upon using VR for diagnosis and surgical procedure planning. (3) Conclusions: The current clinical report attracted the attention of including schwannoma in the possible differential diagnosis of a swelling in the anterior cervical region, mainly when a nonspecific radiological appearance is noticed, even with the use of multiple imaging modalities. Virtual reality can increase the predictability and success rate of the surgical procedure, being in the meantime a good tool for communication with the patient.

Application of a virtual and mixed reality-navigation system using commercially available devices to the lateral temporal bone resection

Background

Lateral temporal bone resection (LTBR) is performed for stage T1-2 external ear malignant tumors and requires spatial anatomical knowledge of the rare surgical field.

Objective

This paper presents a novel virtual reality (VR) based surgical simulation and navigation system using only commercially available display device and an online software, to assist in the understanding of the anatomy pre and intraoperatively.

Result and conclusion

VR model created by 3D Slicer modules and visualized on head mounted display enabled users to simulate and learn surgical techniques of a rare surgical case. 3D hologram through HoloLens assisted the surgeon in comprehending the spatial relationship between crucial vital structures and the pathological lesion during the operation. This platform does not require the users to possess specific programming skill or knowledge, and is therefore applicable in daily clinical usage.

●

Lateral temporal bone resection (LTBR) is standard operative procedure for early-staged malignant tumors of external ear canals.

●

However, many surgeons lack the opportunity to learn the surgical techniques because of its rarity.

●

We report a usage of novel virtual reality based surgical simulation and navigation system for studying the anatomy and the operative steps in LTBR.

●

3D holograms with head-mounted display will provide revolutionary tool in assisting surgical planning, intraoperative referencing and navigation of otologic and skull base surgery.

Use of Virtual Reality for Hybrid Closure of Multiple Ventricular Septal Defects

A 28-month-old girl with multiple ventricular septal defects previously underwent surgical and transcatheter attempts at repair. Three-dimensional models were created from cardiac magnetic resonance–derived images. Viewing the models in virtual reality allowed the team to precisely locate the defects and decide on a hybrid transcatheter and surgical approach to ensure successful repair. (Level of Difficulty: Advanced.)

A Virtual Reality System for Improved Image-Based Planning of Complex Cardiac Procedures

The intricate nature of congenital heart disease requires understanding of the complex, patient-specific three-dimensional dynamic anatomy of the heart, from imaging data such as three-dimensional echocardiography for successful outcomes from surgical and interventional procedures. Conventional clinical systems use flat screens, and therefore, display remains two-dimensional, which undermines the full understanding of the three-dimensional dynamic data. Additionally, the control of three-dimensional visualisation with two-dimensional tools is often difficult, so used only by imaging specialists. In this paper, we describe a virtual reality system for immersive surgery planning using dynamic three-dimensional echocardiography, which enables fast prototyping for visualisation such as volume rendering, multiplanar reformatting, flow visualisation and advanced interaction such as three-dimensional cropping, windowing, measurement, haptic feedback, automatic image orientation and multiuser interactions. The available features were evaluated by imaging and nonimaging clinicians, showing that the virtual reality system can help improve the understanding and communication of three-dimensional echocardiography imaging and potentially benefit congenital heart disease treatment.

Modeling Tool for Rapid Virtual Planning of the Intracardiac Baffle in Double-Outlet Right Ventricle

PURPOSE

Biventricular repair of double-outlet right ventricle (DORV) necessitates the creation of a complex intracardiac baffle. Creation of the optimal baffle design and placement thereof can be challenging to conceptualize, even with 2-dimensional and 3-dimensional images. This report describes a recently developed methodology for creating virtual baffles to inform intraoperative repair.

DESCRIPTION

A total of 3 heart models of DORV were created from cardiac magnetic resonance images. Baffles were created and visualized using custom software.

EVALUATION

This report demonstrates application of the tool to virtual planning of the baffle for repair of DORV in 3 cases. Models were examined by a multidisciplinary team, on screen and in virtual reality. Baffles could be rapidly created and revised to facilitate planning of the surgical procedure.

CONCLUSIONS

Virtual modeling of the baffle pathway by using cardiac magnetic resonance, creation of physical templates for the baffle, and visualization in virtual reality are feasible and may be beneficial for preoperative planning of complex biventricular repairs in DORV. Further work is needed to demonstrate clinical benefit or improvement in outcomes.