Application and evaluation of interactive 3D PDF for presenting and sharing planning results for liver surgery in clinical routine

Comparing a virtual reality head-mounted display to on-screen three-dimensional visualization and two-dimensional computed tomography data for training in decision making in hepatic surgery: a randomized controlled study

OBJECTIVE

Evaluation of the benefits of a virtual reality (VR) environment with a head-mounted display (HMD) for decision-making in liver surgery.

BACKGROUND

Training in liver surgery involves appraising radiologic images and considering the patient's clinical information. Accurate assessment of 2D-tomography images is complex and requires considerable experience, and often the images are divorced from the clinical information. We present a comprehensive and interactive tool for visualizing operation planning data in a VR environment using a head-mounted-display and compare it to 3D visualization and 2D-tomography.

METHODS

Ninety medical students were randomized into three groups (1:1:1 ratio). All participants analyzed three liver surgery patient cases with increasing difficulty. The cases were analyzed using 2D-tomography data (group "2D"), a 3D visualization on a 2D display (group "3D") or within a VR environment (group "VR"). The VR environment was displayed using the "Oculus Rift ™" HMD technology. Participants answered 11 questions on anatomy, tumor involvement and surgical decision-making and 18 evaluative questions (Likert scale).

RESULTS

Sum of correct answers were significantly higher in the 3D (7.1 ± 1.4, p < 0.001) and VR (7.1 ± 1.4, p < 0.001) groups than the 2D group (5.4 ± 1.4) while there was no difference between 3D and VR (p = 0.987). Times to answer in the 3D (6:44 ± 02:22 min, p < 0.001) and VR (6:24 ± 02:43 min, p < 0.001) groups were significantly faster than the 2D group (09:13 ± 03:10 min) while there was no difference between 3D and VR (p = 0.419). The VR environment was evaluated as most useful for identification of anatomic anomalies, risk and target structures and for the transfer of anatomical and pathological information to the intraoperative situation in the questionnaire.

CONCLUSIONS

A VR environment with 3D visualization using a HMD is useful as a surgical training tool to accurately and quickly determine liver anatomy and tumor involvement in surgery.

Potential of Three-Dimensional Portable Document Format for Article Publication in Craniofacial Surgery

The portable document format (PDF) is a globally recognized standard file format of Word documents. The PDF supports the embedding of three-dimensional objects. This three-dimensional PDF can contribute to new ways of presenting and disseminating medical information in future medical journals.

Clinical Applications of Mixed Reality and 3D Printing in Congenital Heart Disease

Understanding the anatomical features and generation of realistic three-dimensional (3D) visualization of congenital heart disease (CHD) is always challenging due to the complexity and wide spectrum of CHD. Emerging technologies, including 3D printing and mixed reality (MR), have the potential to overcome these limitations based on 2D and 3D reconstructions of the standard DICOM (Digital Imaging and Communications in Medicine) images. However, very little research has been conducted with regard to the clinical value of these two novel technologies in CHD. This study aims to investigate the usefulness and clinical value of MR and 3D printing in assisting diagnosis, medical education, pre-operative planning, and intraoperative guidance of CHD surgeries through evaluations from a group of cardiac specialists and physicians. Two cardiac computed tomography angiography scans that demonstrate CHD of different complexities (atrial septal defect and double outlet right ventricle) were selected and converted into 3D-printed heart models (3DPHM) and MR models. Thirty-four cardiac specialists and physicians were recruited. The results showed that the MR models were ranked as the best modality amongst the three, and were significantly better than DICOM images in demonstrating complex CHD lesions (mean difference (MD) = 0.76, p = 0.01), in enhancing depth perception (MD = 1.09, p = 0.00), in portraying spatial relationship between cardiac structures (MD = 1.15, p = 0.00), as a learning tool of the pathology (MD = 0.91, p = 0.00), and in facilitating pre-operative planning (MD = 0.87, p = 0.02). The 3DPHM were ranked as the best modality and significantly better than DICOM images in facilitating communication with patients (MD = 0.99, p = 0.00). In conclusion, both MR models and 3DPHM have their own strengths in different aspects, and they are superior to standard DICOM images in the visualization and management of CHD.

Embedding interactive, three-dimensional content in portable document format to deliver gross anatomy information and knowledge

The Portable Document Format (PDF) is likely the most widely used digital file format for scholarly and scientific electronic publishing. Since format specification version 1.6, three-dimensional (3D) models in Universal 3D (U3D) format can be embedded into PDF files. The present study demonstrates a repertoire of graphic strategies and modes of presentation that exploit the potentials of 3D models embedded in PDF to deliver anatomical information and knowledge. Three-dimensional models and scenes representing anatomical structures generated by 3D surface scanning or by segmentation from either clinical imaging data or cadaver sectional images were converted into U3D format and then embedded into PDF files using both freely and commercially available software. The relevant steps and required software tools are described. Built-in tools in Adobe Acrobat and JavaScript scripting both were used to pre-configure user interaction with 3D contents. Eight successive proof-of-concept examples of increasing complexity are presented and provided as supplementary material, including both unannotated and annotated 3D specimens, use of bitmap-textures, guided navigation through predetermined 3D scenes, 3D animation, and interactive navigation through tri-planar sectional human cadaver images. Three-dimensional contents embedded in PDF files are generally comparable to multimedia and dedicated 3D software in terms of quality, flexibility, and convenience, and offer new unprecedented opportunities to deliver anatomical information and knowledge.

IMHOTEP: cross-professional evaluation of a three-dimensional virtual reality system for interactive surgical operation planning, tumor board discussion and immersive training for complex liver surgery in a head-mounted display

BACKGROUND

Virtual reality (VR) with head-mounted displays (HMD) may improve medical training and patient care by improving display and integration of different types of information. The aim of this study was to evaluate among different healthcare professions the potential of an interactive and immersive VR environment for liver surgery that integrates all relevant patient data from different sources needed for planning and training of procedures.

METHODS

3D-models of the liver, other abdominal organs, vessels, and tumors of a sample patient with multiple hepatic masses were created. 3D-models, clinical patient data, and other imaging data were visualized in a dedicated VR environment with an HMD (IMHOTEP). Users could interact with the data using head movements and a computer mouse. Structures of interest could be selected and viewed individually or grouped. IMHOTEP was evaluated in the context of preoperative planning and training of liver surgery and for the potential of broader surgical application. A standardized questionnaire was voluntarily answered by four groups (students, nurses, resident and attending surgeons).

RESULTS

In the evaluation by 158 participants (57 medical students, 35 resident surgeons, 13 attending surgeons and 53 nurses), 89.9% found the VR system agreeable to work with. Participants generally agreed that complex cases in particular could be assessed better (94.3%) and faster (84.8%) with VR than with traditional 2D display methods. The highest potential was seen in student training (87.3%), resident training (84.6%), and clinical routine use (80.3%). Least potential was seen in nursing training (54.8%).

CONCLUSIONS

The present study demonstrates that using VR with HMD to integrate all available patient data for the preoperative planning of hepatic resections is a viable concept. VR with HMD promises great potential to improve medical training and operation planning and thereby to achieve improvement in patient care.

Case Report: Virtual and Interactive 3D Vascular Reconstruction Before Planned Pancreatic Head Resection and Complex Vascular Anatomy: A Bench-To-Bedside Transfer of New Visualization Techniques in Pancreatic Surgery

Introduction: Bühler's anastomosis (or Bühler's arcade) is an embryonic relic and represents an arterio-arterial connection between the superior mesenteric artery and the celiac trunk. It can be found as a variety in 1–2% of patients.

Case Presentation: We present a case of a patient with metatastatic squamous cell carcinoma of the lung. The patient was in stable disease for 4 years under palliative therapy (most recently second-line therapy with Nevolumab). In 2019, a locally advanced adenocarcinoma of the papilla vateri was diagnosed, additionally. The patient also underwent right hemicolectomy and patch plasty of the celiac trunk and superior mesenteric artery due to colonic ischemia and arteriosclerotic disease with 50–70% stenosis of the superior mesenteric artery several years ago. Due to a complex vascular prehistory, the standardized preoperative imaging was supplemented by two independent vascular reconstructions (a CT angiogram and a reconstruction based on the CT) for the planning of a pylorus-preserving pancreatic head resection and reconstruction according to Traverso-Longmire. In addition, a 3D print was produced. Both, the reconstruction based on the CT scan and the 3D print were created for off-label use as a part of a research project (VIVATOP: Versatile Immersive Virtual and Augmented Tangible OP).

Discussion: In the standardized CT scan and in the clinical CT-angiography, there were no obvious surgically relevant anatomical variations. A Bühler anastomosis was detected in a digital, virtual and interactive 3D-reconstruction. In addition, in the 3D print of the abdominal site the anastomosis was seen as well. Intraoperatively, the presence of Bühler's anastomosis was confirmed. This information had a significant impact on the intraoperative approach. Retrospectively, the vessel variant could be surmised in the axial projection of the CT scan, if one knew what to look for.

Conclusion: For the conduction of a safe surgical procedure, it is imperative that rare anatomical variations are known preoperatively. Increasing digitalization in surgical and perioperative preparation holds great potential for better planning and improved patient safety. Research and cooperation projects such as the VIVATOP project are instrumental for the development of new visualization techniques, which are able to enhance the understanding of complex anatomical relations.

Communicating 3D data-interactive 3D PDF documents for expert reports and scientific publications in the field of forensic medicine

INTRODUCTION

Modern forensic investigations increasingly revert to 3D imaging techniques, such as computed tomography, magnetic resonance imaging, and 3D surface imaging. Findings are therefore often based on 3D data sets; however, this information is commonly reported and communicated within 2D imagery. The use of interactive 3D PDFs is already established in the scientific community but has yet to be implemented in the field of forensic medicine.

METHODS AND MATERIALS

Three example cases were chosen to serve as exemplary data for the most commonly applied imaging techniques in postmortem imaging. 3D surface models were created from postmortem magnetic resonance imaging (PMMR), postmortem computed tomography (PMCT), and 3D surface imaging data sets.

RESULTS

PMMR revealed a space-occupying subdural hemorrhage that led to ipsilateral compression of the brain tissue of the right hemisphere. PMCT displayed a defect in the skull on the left side of the temporal bone. 3D surface imaging data displayed a patterned discoloration on the inside of the left forearm.

DISCUSSION

Interactive 3D PDFs offer the possibility to communicate 3D information to the reader while maintaining all the benefits of a regular 2D PDF. With Adobe Acrobat, the reader can interactively navigate through 3D data sets and create sufficient depth cues to generate a realistic 3D perception of the data.

CONCLUSION

The interactive 3D PDF is a useful extension of standard 2D PDFs and has the potential to communicate 3D data to the reader in a more complete, more comprehensible, and less subjective manner than 2D PDFs.

Three-Dimensional Portable Document Format (3D PDF) in Clinical Communication and Biomedical Sciences: Systematic Review of Applications, Tools, and Protocols

Background

The Portable Document Format (PDF) is the standard file format for the communication of biomedical information via the internet and for electronic scholarly publishing. Although PDF allows for the embedding of three-dimensional (3D) objects and although this technology has great potential for the communication of such data, it is not broadly used by the scientific community or by clinicians.

Objective

The objective of this review was to provide an overview of existing publications that apply 3D PDF technology and the protocols and tools for the creation of model files and 3D PDFs for scholarly purposes to demonstrate the possibilities and the ways to use this technology.

Methods

A systematic literature review was performed using PubMed and Google Scholar. Articles searched for were in English, peer-reviewed with biomedical reference, published since 2005 in a journal or presented at a conference or scientific meeting. Ineligible articles were removed after screening. The found literature was categorized into articles that (1) applied 3D PDF for visualization, (2) showed ways to use 3D PDF, and (3) provided tools or protocols for the creation of 3D PDFs or necessary models. Finally, the latter category was analyzed in detail to provide an overview of the state of the art.

Results

The search retrieved a total of 902 items. Screening identified 200 in-scope publications, 13 covering the use of 3D PDF for medical purposes. Only one article described a clinical routine use case; all others were pure research articles. The disciplines that were covered beside medicine were many. In most cases, either animal or human anatomies were visualized. A method, protocol, software, library, or other tool for the creation of 3D PDFs or model files was described in 19 articles. Most of these tools required advanced programming skills and/or the installation of further software packages. Only one software application presented an all-in-one solution with a graphical user interface.

Conclusions

The use of 3D PDF for visualization purposes in clinical communication and in biomedical publications is still not in common use, although both the necessary technique and suitable tools are available, and there are many arguments in favor of this technique. The potential of 3D PDF usage should be disseminated in the clinical and biomedical community. Furthermore, easy-to-use, standalone, and free-of-charge software tools for the creation of 3D PDFs should be developed.

Composing only by thought: Novel application of the P300 brain-computer interface

The P300 event-related potential is a well-known pattern in the electroencephalogram (EEG). This kind of brain signal is used for many different brain-computer interface (BCI) applications, e.g., spellers, environmental controllers, web browsers, or for painting. In recent times, BCI systems are mature enough to leave the laboratories to be used by the end-users, namely severely disabled people. Therefore, new challenges arise and the systems should be implemented and evaluated according to user-centered design (USD) guidelines. We developed and implemented a new system that utilizes the P300 pattern to compose music. Our Brain Composing system consists of three parts: the EEG acquisition device, the P300-based BCI, and the music composing software. Seventeen musical participants and one professional composer performed a copy-spelling, a copy-composing, and a free-composing task with the system. According to the USD guidelines, we investigated the efficiency, the effectiveness and subjective criteria in terms of satisfaction, enjoyment, frustration, and attractiveness. The musical participants group achieved high average accuracies: 88.24% (copy-spelling), 88.58% (copy-composing), and 76.51% (free-composing). The professional composer achieved also high accuracies: 100% (copy-spelling), 93.62% (copy-composing), and 98.20% (free-composing). General results regarding the subjective criteria evaluation were that the participants enjoyed the usage of the Brain Composing system and were highly satisfied with the system. Showing very positive results with healthy people in this study, this was the first step towards a music composing system for severely disabled people.

The effect of intraoperative fluoroscopy on the accuracy of femoral tunnel placement in single-bundle anatomic ACL reconstruction

PURPOSE

The purpose of the current study was to investigate the potential effect of intraoperative fluoroscopy on the accuracy of femoral tunnel placement in anatomic ACL reconstruction, using an ideal anatomic point as reference and evaluating postoperative tunnel placement based on 3D CT.

METHODS

An experienced ACL surgeon, using the anatomic approach for femoral tunnel placement, relying on intraarticular landmarks and remnants of the torn ACL-and novel to the fluoroscopic assist-was introduced to its use. A prospective series of patients was included where group 1 (without fluoroscopy) and group 2 (with fluoroscopy) both had postoperative CT scans so that femoral tunnel position could be evaluated and compared to an ideal tunnel centre based on anatomic studies by using the Bernard and Hertel grid.

RESULTS

Group 2, where fluoroscopy was used, had a mean femoral tunnel that was closer to the ideal anatomic centre than group 1. In the Bernard and Hertel grid, the distance in the high-low axis (y-axis) was found significantly closer (P = 0.001), whilst the deep-shallow axis (x-axis) and a total absolute distance were not significantly closer to the ideal described anatomic centre.

CONCLUSIONS

Intraoperative fluoroscopy was found effective as an aid for placing the femoral tunnel in a more accurate position, as compared to a desired anatomic centre. Although the concept of the "one-size-fits-all" approach for tunnel placement is debatable, the avoidance of grossly misplaced tunnels is the benefit of using fluoroscopy during ACL reconstruction. The authors hold that fluoroscopy is readily available, safe and easy to use and therefore a good aid in the anatomic approach for graft tunnel placement, for example, in a learning situation, in revision cases and when performing low volumes of such surgery.

LEVEL OF EVIDENCE

III.

Computer-assisted surgery planning in children with complex liver tumors identifies variability of the classical Couinaud classification

BACKGROUND

In complex malignant pediatric liver tumors there is an ongoing discussion regarding surgical strategy; for example, primary organ transplantation versus extended resection in hepatoblastoma involving 3 or 4 sectors of the liver. We evaluated the possible role of computer-assisted surgery planning in children with complex hepatic tumors.

METHODS

Between May 2004 and March 2016, 24 Children with complex liver tumors underwent standard multislice helical CT scan or MRI scan at our institution. Imaging data were processed using the software assistant LiverAnalyzer (Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany). Results were provided as Portable Document Format (PDF) with embedded interactive 3-dimensional surface mesh models.

RESULTS

Median age of patients was 33months. Diagnoses were hepatoblastoma (n=14), sarcoma (n=3), benign parenchyma alteration (n=2), as well as hepatocellular carcinoma, rhabdoid tumor, focal nodular hyperplasia, hemangioendothelioma, or multiple hepatic metastases of a pancreas carcinoma (each n=1). Volumetry of liver segments identified remarkable variations and substantial aberrances from the Couinaud classification. Computer-assisted surgery planning was used to determine surgical strategies in 20/24 children; this was especially relevant in tumors affecting 3 or 4 liver sectors. Primary liver transplantation could be avoided in 12 of 14 hepaoblastoma patients who theoretically were candidates for this approach.

CONCLUSIONS

Computer-assisted surgery planning substantially contributed to the decision for surgical strategies in children with complex hepatic tumors. This tool possibly allows determination of specific surgical procedures such as extended surgical resection instead of primary transplantation in certain conditions.

The Scientist as Illustrator

Proficiency in art and illustration was once considered an essential skill for biologists, because text alone often could not suffice to describe observations of biological systems. With modern imaging technology, it is no longer necessary to illustrate what we can see by eye. However, in molecular and cellular biology, our understanding of biological processes is dependent on our ability to synthesize diverse data to generate a hypothesis. Creating visual models of these hypotheses is important for generating new ideas and for communicating to our peers and to the public. Here, I discuss the benefits of creating visual models in molecular and cellular biology and consider steps to enable researchers to become more effective visual communicators.

[Preoperative imaging/operation planning for liver surgery]

The currently established standard for planning liver surgery is multistage contrast media-enhanced multidetector computed tomography (CM-CT), which as a rule enables an appropriate resection planning, e.g. a precise identification and localization of primary and secondary liver tumors as well as the anatomical relation to extrahepatic and/or intrahepatic vascular and biliary structures. Furthermore, CM-CT enables the measurement of tumor volume, total liver volume and residual liver volume after resection. Under the condition of normal liver function a residual liver volume of 25 % is nowadays considered sufficient and safe. Recent studies in patients with liver metastases of colorectal cancer showed a clear staging advantage of contrast media-enhanced magnetic resonance imaging (CM-MRI) versus CM-CT. In addition, most recent data showed that the use of liver-specific MRI contrast media further increases the sensitivity and specificity of detection of liver metastases. This imaging technology seems to lead closer to the ideal "one stop shopping" diagnostic tool in preoperative planning of liver resection.

Towards an easier creation of three-dimensional data for embedding into scholarly 3D PDF (Portable Document Format) files

The Portable Document Format (PDF) allows for embedding three-dimensional (3D) models and is therefore particularly suitable to communicate respective data, especially as regards scholarly articles. The generation of the necessary model data, however, is still challenging, especially for inexperienced users. This prevents an unrestrained proliferation of 3D PDF usage in scholarly communication. This article introduces a new solution for the creation of three of types of 3D geometry (point clouds, polylines and triangle meshes), that is based on MeVisLab, a framework for biomedical image processing. This solution enables even novice users to generate the model data files without requiring programming skills and without the need for an intensive training by simply using it as a conversion tool. Advanced users can benefit from the full capability of MeVisLab to generate and export the model data as part of an overall processing chain. Although MeVisLab is primarily designed for handling biomedical image data, the new module is not restricted to this domain. It can be used for all scientific disciplines.