Cinematic Rendering in Mixed-Reality Holograms: A New 3D Preoperative Planning Tool in Pediatric Heart Surgery

Holography-guided procedural planning for modifying Venus P-valve implantation technique in patients with left pulmonary artery stents: a case-series

Background

Venus P-valve™ (Venus Medtech, Hangzhou, China) is a self-expandable bioprosthetic valve that can be transcatheter-implanted in native right ventricular outflow tract (RVOT) patients. Valve implantation is technically challenging. Due to the implantation technique, left pulmonary artery (LPA) stents represent a relative contraindication to Venus P-valve. In this case series, we describe our experience in implanting Venus P-valve in patients with previous LPA stents and the use of holographic models to facilitate procedural planning.

Methods and results

From January to October 2023, 17 patients were scheduled for Venus P-Valve implantation. 16/17 (94%) patients were successfully implanted. 3/16 (18.7%) patients underwent Venus P-valve implantation with LPA stents. All patients underwent pre-operative CT scan. CT data set were employed to create three-dimensional (3D) holographic models (Artiness, Milan, Italy) of the entire heart, which were useful to plan valve implantation with a modified technique. Procedural success rate was 100%. No procedural complications occurred. All three patients presented good haemodynamic and angiographic results at discharge and follow-up visits.

Conclusion

This case-series underscores the feasibility of Venus P-valve implantation in patients with previous LPA stents. The use of holographic models facilitated procedural planning in these challenging anatomical scenarios.

Intraoperative gaze guidance with mixed reality

Efficient communication and collaboration are essential in the operating room for successful and safe surgery. While many technologies are improving various aspects of surgery, communication between attending surgeons, residents, and surgical teams is still limited to verbal interactions that are prone to misunderstandings. Novel modes of communication can increase speed and accuracy, and transform operating rooms. A mixed reality (MR) based gaze sharing application on Microsoft HoloLens 2 headset that can help expert surgeons indicate specific regions, communicate with decreased verbal effort, and guide novices throughout an operation is presented. The utility of the application is tested with a user study of endoscopic kidney stone localization completed by urology experts and novice surgeons. Improvement is observed in the NASA task load index surveys (up to 25.23%), in the success rate of the task (6.98% increase in localized stone percentage), and in gaze analyses (up to 31.99%). The proposed application shows promise in both operating room applications and surgical training tasks.

Expanded experience with cardiovascular black blood cinematic rendering

Black blood cinematic rendering (BBCR) is a newly described preset for cinematic rendering, which creates photorealistic displays from volumetric data sets with the contrast-enhanced blood pool displayed as dark and transparent. That set of features potentially provides for enhanced visualization of endomyocardial and intraluminal pathology, as well as cardiac devices. The similarity of the images to black-blood magnetic resonance imaging (MRI) may allow for expansion of the evaluation of certain types of pathology into patient populations unable to undergo MRI. In the emergency setting, the rapid acquisition time and reasonable post-processing time make this technique clinically feasible. In this expanded experience, we demonstrate an expanded clinical experience with the BBCR technique, highlighting the applications for intraluminal cardiovascular evaluation, especially focused on current and potential emergency radiology applications.

Comparison of two 3D reconstruction models for understanding of complicated female pelvic tumors

OBJECTIVE

Three-dimensional (3D) reconstructed models have been shown to improve visualization in complex female pelvic tumors. Cinematic rendering (CR) is a 3D imaging technique for computed tomography (CT) images, which creates more realistic images with the ability to enhance imaging of anatomical features for diagnosis. This study was set up to compare two types of 3D models and to validate the use of 3D anatomical techniques for the diagnosis of complex female pelvic tumors.

METHODS

The preclinical, randomized, two-sequence crossover investigation was performed from December 2022 to January 2023 at First Affiliated Hospital of Chongqing Medical University. Sixteen residents and 10 attending surgeons assessed the cases of 23 patients with two types of 3D model images. The surgeons were randomly assigned to two assessment sequences (CR-3D model group and CT-3D model group). For each case, participants selected one question that probed fundamental questions about the tumor's genesis throughout each assessment period. Following a 4-week washout period, case assessments were transferred to the other image modality.

RESULTS

The main result assessment was the accuracy of the answers. The time to answer the questions and the case assessment questionnaire was added as a secondary outcome. The mean scores in the CR-3D models (19.35 ± 1.87) varied significantly from those in the CT-CR group (16.77 ± 1.8) (P < 0.001), and solving the questions in the CT-3D model sequence (41.96 ± 6.31 s) varied significantly from that in the CR-3D model sequence (52.88 ± 5.95 s) (P < 0.001). Subgroup analysis revealed that there were statistically significant variations in the scores of female reproductive tumors, pelvic tumors other than the reproductive system, and retroperitoneal tumors (P = 0.005). Analysis of the assessment questionnaire showed that more surgeons choose CR 3D reconstruction (8.31 ± 0.76 vs 7.15 ± 1.19, P < 0.001).

CONCLUSIONS

The results suggest that each 3D reconstruction method has its own advantages. Surgeons feel that CR reconstruction models are a useful technique that can improve their comprehension of complex pelvic tumors, while traditional 3D models have an advantage in terms of speed to diagnosis.

Imaging of Double Inlet Left Ventricle

Double inlet left ventricle (DILV) is a form of single ventricle heart disease where both atrioventricular valves enter a single left ventricle. Surgical intervention may be needed in the neonatal period secondary to systemic outflow tract obstruction or less commonly pulmonary obstruction. Two-dimensional echocardiography can adequately assess newborn anatomy and define the need for surgery. Beyond the newborn period, there is a renewed interest in septation of DILV using intracardiac baffles in a staged approach. Cross sectional imaging can aid in surgical planning. This article will review common anatomic features of DILV and imaging considerations for both single ventricle palliation and DILV septation.

Cardiovascular Computed Tomography in the Diagnosis of Cardiovascular Disease: Beyond Lumen Assessment

Cardiovascular CT is being widely used in the diagnosis of cardiovascular disease due to the rapid technological advancements in CT scanning techniques. These advancements include the development of multi-slice CT, from early generation to the latest models, which has the capability of acquiring images with high spatial and temporal resolution. The recent emergence of photon-counting CT has further enhanced CT performance in clinical applications, providing improved spatial and contrast resolution. CT-derived fractional flow reserve is superior to standard CT-based anatomical assessment for the detection of lesion-specific myocardial ischemia. CT-derived 3D-printed patient-specific models are also superior to standard CT, offering advantages in terms of educational value, surgical planning, and the simulation of cardiovascular disease treatment, as well as enhancing doctor-patient communication. Three-dimensional visualization tools including virtual reality, augmented reality, and mixed reality are further advancing the clinical value of cardiovascular CT in cardiovascular disease. With the widespread use of artificial intelligence, machine learning, and deep learning in cardiovascular disease, the diagnostic performance of cardiovascular CT has significantly improved, with promising results being presented in terms of both disease diagnosis and prediction. This review article provides an overview of the applications of cardiovascular CT, covering its performance from the perspective of its diagnostic value based on traditional lumen assessment to the identification of vulnerable lesions for the prediction of disease outcomes with the use of these advanced technologies. The limitations and future prospects of these technologies are also discussed.

Modern tools in congenital heart disease imaging and procedure planning: a European survey

AIMS

Congenital heart diseases (CHDs) often show a complex 3D anatomy that must be well understood to assess the pathophysiological consequences and to guide therapy. Three-dimensional imaging technologies have the potential to enhance the physician's comprehension of such spatially complex anatomies. Unfortunately, due to the new introduction in clinical practice, there is no evidence on the current applications. We conducted a survey to examine how 3D technologies are currently used among CHD European centres.

METHODS

Data were collected using an online self-administered survey via SurveyMonkey. The questionnaire was sent via e-mail and the responses were collected between January and June 2022.

RESULTS

Ninety-eight centres correctly completed the survey. Of these, 22 regularly perform 3D rotational angiography, 43 have the availability to print in-silico models, and 22 have the possibility to visualize holographic imaging/virtual reality. The costs were mostly covered by the hospital or the department of financial resources.

CONCLUSION

From our survey, it emerges that these technologies are quite spread across Europe, despite not being part of a routine practice. In addition, there are still not enough data supporting the improvement of clinical management for CHD patients. For this reason, further studies are needed to develop clinical recommendations for the use of 3D imaging technologies in medical practice.

Optimized preoperative planning of double outlet right ventricle patients by 3D printing and virtual reality: a pilot study

OBJECTIVES

In complex double outlet right ventricle (DORV) patients, the optimal surgical approach may be difficult to assess based on conventional 2-dimensional (2D) ultrasound (US) and computed tomography (CT) imaging. The aim of this study is to assess the added value of 3-dimensional (3D) printed and 3D virtual reality (3D-VR) models of the heart used for surgical planning in DORV patients, supplementary to the gold standard 2D imaging modalities.

METHODS

Five patients with different DORV subtypes and high-quality CT scans were selected retrospectively. 3D prints and 3D-VR models were created. Twelve congenital cardiac surgeons and paediatric cardiologists, from 3 different hospitals, were shown 2D-CT first, after which they assessed the 3D print and 3D-VR models in random order. After each imaging method, a questionnaire was filled in on the visibility of essential structures and the surgical plan.

RESULTS

Spatial relationships were generally better visualized using 3D methods (3D printing/3D-VR) than in 2D. The feasibility of ventricular septum defect patch closure could be determined best using 3D-VR reconstructions (3D-VR 92%, 3D print 66% and US/CT 46%, P < 0.01). The percentage of proposed surgical plans corresponding to the performed surgical approach was 66% for plans based on US/CT, 78% for plans based on 3D printing and 80% for plans based on 3D-VR visualization.

CONCLUSIONS

This study shows that both 3D printing and 3D-VR have additional value for cardiac surgeons and cardiologists over 2D imaging, because of better visualization of spatial relationships. As a result, the proposed surgical plans based on the 3D visualizations matched the actual performed surgery to a greater extent.

Photon-counting versus Dual-Source CT of Congenital Heart Defects in Neonates and Infants: Initial Experience

Background Photon-counting CT (PCCT) has been shown to improve cardiovascular CT imaging in adults. Data in neonates, infants, and young children under the age of 3 years are missing. Purpose To compare image quality and radiation dose of ultrahigh-pitch PCCT with that of ultrahigh-pitch dual-source CT (DSCT) in children suspected of having congenital heart defects. Materials and Methods This is a prospective analysis of existing clinical CT studies in children suspected of having congenital heart defects who underwent contrast-enhanced PCCT or DSCT in the heart and thoracic aorta between January 2019 and October 2022. CT dose index and dose-length product were used to calculate effective radiation dose. Signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) were calculated by standardized region-of-interest analysis. SNR and CNR dose ratios were calculated. Visual image quality was assessed by four independent readers on a five-point scale: 5, excellent or absent; 4, good or minimal; 3, moderate; 2, limited or substantial; and 1, poor or massive. Results Contrast-enhanced PCCT (n = 30) or DSCT (n = 84) was performed in 113 children (55 female and 58 male participants; median age, 66 days [IQR, 15-270]; median height, 56 cm [IQR, 52-67]; and median weight, 4.5 kg [IQR, 3.4-7.1]). A diagnostic image quality score of at least 3 was obtained in 29 of 30 (97%) with PCCT versus 65 of 84 (77%) with DSCT. Mean overall image quality ratings were higher for PCCT versus DSCT (4.17 vs 3.16, respectively; P < .001). SNR and CNR were higher for PCCT versus DSCT with SNR (46.3 ± 16.3 vs 29.9 ± 15.3, respectively; P = .007) and CNR (62.0 ± 50.3 vs 37.2 ± 20.8, respectively; P = .001). Mean effective radiation doses were similar for PCCT and DSCT (0.50 mSv vs 0.52 mSv; P = .47). Conclusion At a similar radiation dose, PCCT offers a higher SNR and CNR and thus better cardiovascular imaging quality than DSCT in children suspected of having cardiac heart defects. © RSNA, 2023.

Exploring the Potential of Three-Dimensional Imaging, Printing, and Modeling in Pediatric Surgical Oncology: A New Era of Precision Surgery

Pediatric surgical oncology is a technically challenging field that relies on CT and MRI as the primary imaging tools for surgical planning. However, recent advances in 3D reconstructions, including Cinematic Rendering, Volume Rendering, 3D modeling, Virtual Reality, Augmented Reality, and 3D printing, are increasingly being used to plan complex cases bringing new insights into pediatric tumors to guide therapeutic decisions and prognosis in different pediatric surgical oncology areas and locations including thoracic, brain, urology, and abdominal surgery. Despite this, challenges to their adoption remain, especially in soft tissue-based specialties such as pediatric surgical oncology. This work explores the main innovative imaging reconstruction techniques, 3D modeling technologies (CAD, VR, AR), and 3D printing applications through the analysis of three real cases of the most common and surgically challenging pediatric tumors: abdominal neuroblastoma, thoracic inlet neuroblastoma, and a bilateral Wilms tumor candidate for nephron-sparing surgery. The results demonstrate that these new imaging and modeling techniques offer a promising alternative for planning complex pediatric oncological cases. A comprehensive analysis of the advantages and limitations of each technique has been carried out to assist in choosing the optimal approach.

Feedback attention network for cardiac magnetic resonance imaging super-resolution

BACKGROUND AND OBJECTIVE

Atrial fibrillation (AF) is a common clinical arrhythmia with a high disability and mortality rate. Improving the resolution of atrial structure and its changes in patients with AF is very important for understanding and treating AF.

METHODS

Aiming at the problems of previous deep learning-based image super-resolution (SR) reconstruction methods simply deepening the network, loss of upsampling information, and difficulty in the reconstruction of high-frequency information, we propose the Feedback Attention Network (FBAN) for cardiac magnetic resonance imaging (CMRI) super-resolution. The network comprises a preprocessing module, a multi-scale residual group module, an upsampling module, and a reconstruction module. The preprocessing module uses a convolutional layer to extract shallow features and dilate the number of channels of the feature map. The multi-scale residual group module adds a multi-channel network, a mixed attention mechanism, and a long and short skip connection to expand the receptive field of the feature map, improve the multiplexing of multi-scale features and strengthen the reconstruction of high-frequency information. The upsampling module adopts the sub-pixel method to upsample the feature map to the target image size. The reconstruction module consists of a convolutional layer, which is used to restore the number of channels of the feature map to the original number to obtain the reconstructed high-resolution (HR) image.

RESULTS

Furthermore, the test results on the public dataset of CMRI show that the HR images reconstructed by the FBAN method not only have a good improvement in reconstructed edge and texture information but also have a good improvement in the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) objective evaluation indicators.

CONCLUSION

Compared with the local magnified image, the edge information of the FBAN method reconstructed image has been enhanced, more high-frequency information of the CMRI is restored, the texture details are less lost, and the reconstructed image is less blurry. Overall, the reconstructed image has a lighter feeling of smearing, and the visual experience is more apparent and sharper.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Mixed Reality in Modern Surgical and Interventional Practice: Narrative Review of the Literature

BACKGROUND

Mixed reality (MR) and its potential applications have gained increasing interest within the medical community over the recent years. The ability to integrate virtual objects into a real-world environment within a single video-see-through display is a topic that sparks imagination. Given these characteristics, MR could facilitate preoperative and preinterventional planning, provide intraoperative and intrainterventional guidance, and aid in education and training, thereby improving the skills and merits of surgeons and residents alike.

OBJECTIVE

In this narrative review, we provide a broad overview of the different applications of MR within the entire spectrum of surgical and interventional practice and elucidate on potential future directions.

METHODS

A targeted literature search within the PubMed, Embase, and Cochrane databases was performed regarding the application of MR within surgical and interventional practice. Studies were included if they met the criteria for technological readiness level 5, and as such, had to be validated in a relevant environment.

RESULTS

A total of 57 studies were included and divided into studies regarding preoperative and interventional planning, intraoperative and interventional guidance, as well as training and education.

CONCLUSIONS

The overall experience with MR is positive. The main benefits of MR seem to be related to improved efficiency. Limitations primarily seem to be related to constraints associated with head-mounted display. Future directions should be aimed at improving head-mounted display technology as well as incorporation of MR within surgical microscopes, robots, and design of trials to prove superiority.

Augmented Reality With Cinematic Rendered 3-Dimensional Images From Volumetric Computed Tomography Data

ABSTRACT

Recent advances in 3-dimensional visualization of volumetric computed tomography data have led to the novel technique of cinematic rendering (CR), which provides photorealistic images with enhanced surface detail and realistic shadowing effects that are generally not possible with older methods such as volume rendering. The emergence of CR coincides with the increasingly widespread availability of virtual reality (VR)/augmented reality (AR) interfaces including wearable headsets. The intersection of these technologies suggests many potential advances, including the ability of interpreting radiologists to look at photorealistic images of patient pathology in real time with surgeons and other referring providers, so long as VR/AR headsets are deployed and readily available. In this article, we will present our initial experience with viewing and manipulating CR images in the context of a VR/AR headset. We include a description of key aspects of the software and user interface, and provide relevant pictorial examples that may help potential adopters understand the initial steps of using this exciting convergence of technologies. Ultimately, trials evaluating the added value of the combination of CR with VR/AR will be necessary to understand the potential impact of these methods on medical practice.

Photorealistic three-dimensional visualization of fusion datasets: cinematic rendering of PET/CT

PURPOSE

Cinematic rendering (CR) is a method of photorealistic 3D visualization of volumetric imaging data. We applied this technique to fusion PET/CT data.

METHODS

Two recent PET/CT cases were selected, one each of prostate-specific membrane antigen (PSMA)-targeted ¹⁸F-DCFPyL, and somatostatin-receptor-targeted ⁶⁸ Ga-DOTATATE. Targeted radiotracers were selected in order to provide high-contrast images for this proof-of-principle study. Cinematic rendering was performed with an enhanced algorithm that incorporated internal lighting within the PET-avid organs and lesions to allow for a distinct visual signature.

RESULTS

The use of internal lighting for PET data provided CR of fused PET/CT scans. The interpreting radiologist must make judicious use of presets and cut planes in order to ensure important findings are not missed.

CONCLUSIONS

CR of PET/CT data provides a photorealistic means of visualizing complex fusion imaging datasets. Such visualizations may aid anatomic understanding for surgical or procedural applications, may improve teaching of trainees, and may allow improved communication with patients.

Clinical Applications of Mixed Reality and 3D Printing in Congenital Heart Disease

Understanding the anatomical features and generation of realistic three-dimensional (3D) visualization of congenital heart disease (CHD) is always challenging due to the complexity and wide spectrum of CHD. Emerging technologies, including 3D printing and mixed reality (MR), have the potential to overcome these limitations based on 2D and 3D reconstructions of the standard DICOM (Digital Imaging and Communications in Medicine) images. However, very little research has been conducted with regard to the clinical value of these two novel technologies in CHD. This study aims to investigate the usefulness and clinical value of MR and 3D printing in assisting diagnosis, medical education, pre-operative planning, and intraoperative guidance of CHD surgeries through evaluations from a group of cardiac specialists and physicians. Two cardiac computed tomography angiography scans that demonstrate CHD of different complexities (atrial septal defect and double outlet right ventricle) were selected and converted into 3D-printed heart models (3DPHM) and MR models. Thirty-four cardiac specialists and physicians were recruited. The results showed that the MR models were ranked as the best modality amongst the three, and were significantly better than DICOM images in demonstrating complex CHD lesions (mean difference (MD) = 0.76, p = 0.01), in enhancing depth perception (MD = 1.09, p = 0.00), in portraying spatial relationship between cardiac structures (MD = 1.15, p = 0.00), as a learning tool of the pathology (MD = 0.91, p = 0.00), and in facilitating pre-operative planning (MD = 0.87, p = 0.02). The 3DPHM were ranked as the best modality and significantly better than DICOM images in facilitating communication with patients (MD = 0.99, p = 0.00). In conclusion, both MR models and 3DPHM have their own strengths in different aspects, and they are superior to standard DICOM images in the visualization and management of CHD.

Photorealistic Depiction of Intracranial Tumors Using Cinematic Rendering of Volumetric 3T MRI Data

RATIONALE AND OBJECTIVES

Cinematic Rendering (CR) incorporates a complex lightning model that creates photorealistic models from isotropic 3D imaging data. The utility of CR in depicting volumetric MRI data for pre-therapeutic planning is discussed, with intracranial tumors as a demonstrative example.

MATERIALS AND METHODS

We present a series of Cinematically Rendered intracranial tumors and discuss their utility in multidisciplinary pre-therapeutic evaluation. Isotropic, high-resolution, volumetric MRI data was collected, and CR was performed utilizing a proprietary application, "Anatomy Education" Siemens, Munich, Germany.

RESULTS

Discrimination of cortex to white matter, brain surface to vessels, subarachnoid space to cortex and skull to intracranial structures was achieved and optimized by using various display settings on the Anatomy education application. Progressive removal of tissue layers allowed for a comprehensive assessment of the entire region of interest. Complex, small structures were demonstrated in very high detail. The depth and architecture of the sulci was appreciated in a format that more closely mimicked gross pathology than traditional imaging modalities. With appropriate display settings, the relationship of the cortical surface to the adjacent vasculature was also delineated.

CONCLUSION

CR depicts the anatomic location of brain tumors in a format that depicts the relative proximity of adjacent structures in all dimensions and degrees of freedom. This allows for better conceptualization of the pathology and greater ease of communication between radiologists and other clinical teams, especially in the context of pretherapeutic planning.

Advances and Innovations in Ablative Head and Neck Oncologic Surgery Using Mixed Reality Technologies in Personalized Medicine

The benefit of computer-assisted planning in head and neck ablative and reconstructive surgery has been extensively documented over the last decade. This approach has been proven to offer a more secure surgical procedure. In the treatment of cancer of the head and neck, computer-assisted surgery can be used to visualize and estimate the location and extent of the tumor mass. Nowadays, some software tools even allow the visualization of the structures of interest in a mixed reality environment. However, the precise integration of mixed reality systems into a daily clinical routine is still a challenge. To date, this technology is not yet fully integrated into clinical settings such as the tumor board, surgical planning for head and neck tumors, or medical and surgical education. As a consequence, the handling of these systems is still of an experimental nature, and decision-making based on the presented data is not yet widely used. The aim of this paper is to present a novel, user-friendly 3D planning and mixed reality software and its potential application for ablative and reconstructive head and neck surgery.