IMHOTEP: cross-professional evaluation of a three-dimensional virtual reality system for interactive surgical operation planning, tumor board discussion and immersive training for complex liver surgery in a head-mounted display

[Virtual reality in liver surgery-Planning, advanced training, testing]

Anatomical structures of the liver could be reconstructed three dimensionally from preoperative cross-sectional imaging for over 20 years. This three-dimensional (3D) representation not only optimizes the preoperative planning options but also the communication of specific operation-relevant structures can be improved using 3D models. In addition to a plastic and interactive visualization of 3D organ models, the disruptive technology of virtual reality (VR) can also provide a possibility for structured training and further education regarding surgical anatomy of the liver and operation planning. This makes examinations in virtual reality also seem tangible. The latest prototypes even enable patient-specific simulation of intraoperative sonography. This review article provides an overview of the current status of available technologies for training and further education using VR exemplified by liver surgery.

Surgical planning in virtual reality: a systematic review

Purpose

Virtual reality (VR) technology has emerged as a promising tool for physicians, offering the ability to assess anatomical data in 3D with visuospatial interaction qualities. The last decade has witnessed a remarkable increase in the number of studies focusing on the application of VR to assess patient-specific image data. This systematic review aims to provide an up-to-date overview of the latest research on VR in the field of surgical planning.

Approach

A comprehensive literature search was conducted based on the preferred reporting items for systematic reviews and meta-analyses covering the period from April 1, 2021 to May 10, 2023. It includes research articles reporting on preoperative surgical planning using patient-specific medical images in virtual reality using head-mounted displays. The review summarizes the current state of research in this field, identifying key findings, technologies, study designs, methods, and potential directions for future research.

Results

The selected studies show a positive impact on surgical decision-making and anatomy understanding compared to other visualization modalities. A substantial number of studies are reporting anecdotal evidence and case-specific outcomes. Notably, surgical planning using VR led to more frequent changes in surgical plans compared to planning with other visualization methods when surgeons reassessed their initial plans. VR demonstrated benefits in reducing planning time and improving spatial localization of pathologies.

Conclusions

Results show that the application of VR for surgical planning is still in an experimental stage but is gradually advancing toward clinical use. The diverse study designs, methodologies, and varying reporting hinder a comprehensive analysis. Some findings lack statistical evidence and rely on subjective assumptions. To strengthen evaluation, future research should focus on refining study designs, improving technical reporting, defining visual and technical proficiency requirements, and enhancing VR software usability and design. Addressing these areas could pave the way for an effective implementation of VR in clinical settings.

Virtual Reality Rehabilitation and Artificial Intelligence in Healthcare Technology

The benefit of virtual rehabilitation is that it helps the patient increase their engagement and motivation. Another advantage is that it allows patient specific. A third utility is that the therapist can make the sessions more efficient and productive. A feature of virtual reality (VR) rehabilitation is that it is possible to create virtual environments which are more realistic than those in a video game and in which the patients can perform exercises. As a result, the patients are more immersed and motivated to avoid the boredom from which patients in standard therapy usually suffer. The features of artificial intelligence (AI) in biomedicine are the optimisation of diagnostics, treatment, and patient monitoring. AI allows for the analysis to have the potential to detect subtle deviations. In this chapter, the application of virtual reality and artificial intelligence in healthcare was discussed.

Comparing a virtual reality head-mounted display to on-screen three-dimensional visualization and two-dimensional computed tomography data for training in decision making in hepatic surgery: a randomized controlled study

OBJECTIVE

Evaluation of the benefits of a virtual reality (VR) environment with a head-mounted display (HMD) for decision-making in liver surgery.

BACKGROUND

Training in liver surgery involves appraising radiologic images and considering the patient's clinical information. Accurate assessment of 2D-tomography images is complex and requires considerable experience, and often the images are divorced from the clinical information. We present a comprehensive and interactive tool for visualizing operation planning data in a VR environment using a head-mounted-display and compare it to 3D visualization and 2D-tomography.

METHODS

Ninety medical students were randomized into three groups (1:1:1 ratio). All participants analyzed three liver surgery patient cases with increasing difficulty. The cases were analyzed using 2D-tomography data (group "2D"), a 3D visualization on a 2D display (group "3D") or within a VR environment (group "VR"). The VR environment was displayed using the "Oculus Rift ™" HMD technology. Participants answered 11 questions on anatomy, tumor involvement and surgical decision-making and 18 evaluative questions (Likert scale).

RESULTS

Sum of correct answers were significantly higher in the 3D (7.1 ± 1.4, p < 0.001) and VR (7.1 ± 1.4, p < 0.001) groups than the 2D group (5.4 ± 1.4) while there was no difference between 3D and VR (p = 0.987). Times to answer in the 3D (6:44 ± 02:22 min, p < 0.001) and VR (6:24 ± 02:43 min, p < 0.001) groups were significantly faster than the 2D group (09:13 ± 03:10 min) while there was no difference between 3D and VR (p = 0.419). The VR environment was evaluated as most useful for identification of anatomic anomalies, risk and target structures and for the transfer of anatomical and pathological information to the intraoperative situation in the questionnaire.

CONCLUSIONS

A VR environment with 3D visualization using a HMD is useful as a surgical training tool to accurately and quickly determine liver anatomy and tumor involvement in surgery.

Exploring mobile mixed reality for critical thinking in nursing and healthcare education: A systematic review

BACKGROUND

The shortage of nursing and healthcare clinical placements has prompted the investigation of ways to supplement authentic learning. Mobile mixed reality has become increasingly available, however, the affordances and design principles for the facilitation of critical thinking are yet to be explored.

OBJECTIVE

To examine how mobile mixed reality facilitates critical thinking in nursing and healthcare higher education.

DESIGN

Systematic review.

REVIEW METHODS

A search in seven databases (MEDLINE, PsychINFO, AMED, ERIC, Scopus, Cochrane, and Web of Science) was conducted with 3488 titles and abstracts screened. The quality of the included studies was evaluated using the Mixed Methods Assessment Tool (MMAT).

RESULTS

A total of 12 studies with 1108 participants were included. The breadth of healthcare disciplines was limited to five disciplines that utilised bespoke scenarios on head-mounted displays. Most scenarios were emergency or critical response, with limited time for pre-brief, debrief, or overall user time. Only two studies directly measured critical thinking, with others including indirect reference to diagnoses, interpretation, analysis, or evaluation of healthcare scenarios. Affordances and design principles for the future development of mobile mixed reality for critical thinking in nursing and healthcare higher education are identified.

CONCLUSIONS

While some pedagogical affordances of mobile mixed reality can be identified in a narrow number of healthcare disciplines, there remain to be limited valid measures of critical thinking used to quantify effectiveness. Future studies would benefit from considering scenarios beyond emergency and critical responses, including longitudinal studies that reflect the development of critical thinking over time, and exploration of co-designed scenarios with and by nursing and healthcare students.

Optimizing Surgical Efficiency in Complex Spine Surgery Using Virtual Reality as a Communication Technology to Promote a Shared Mental Model: A Case Series and Review

BACKGROUND AND OBJECTIVES

Virtual reality (VR) is an emerging technology that can be used to promote a shared mental model among a surgical team. We present a case series demonstrating the use of 3-dimensional (3D) VR models to visually communicate procedural steps to a surgical team to promote a common operating objective. We also review the literature on existing uses of VR for preoperative communication and planning in spine surgery.

METHODS

Narrations of 3 to 4-minute walkthroughs were created in a VR visualization platform, converted, and distributed to team members through text and email the night before surgical intervention. A VR huddle was held immediately before the intervention to refine surgical goals. After the intervention, the participating team members' perceptions on the value of the tool were assessed using a survey that used a 5-point Likert scale. MEDLINE, Google Scholar, and Dimensions AI databases were queried from July 2010 to October 2022 to examine existing literature on preoperative VR use to plan spine surgery.

RESULTS

Three illustrative cases are presented with accompanying video. Postoperative survey results demonstrate a positive experience among surgical team members after reviewing preoperative plans created with patient-specific 3D VR models. Respondents felt that preoperative VR video review was "moderately useful" or more useful in improving their understanding of the operational sequence (71%, 5/7), in enhancing their ability to understand their role (86%, 6/7), and in improving the safety or efficiency of the case (86%, 6/7).

CONCLUSION

We present a proof of concept of a novel preoperative communication tool used to create a shared mental model of a common operating objective for surgical team members using narrated 3D VR models. Initial survey results demonstrate positive feedback among respondents. There is a paucity of literature investigating VR technology as a means for preoperative surgical communication in spine surgery.

ETHICS

Institutional review board approval (IRB-300009785) was obtained before this study.

Optimizing Surgical Performance Using Preoperative Virtual Reality Planning: A Systematic Review

BACKGROUND

Surgery is often a complex process that requires detailed 3-dimensional anatomical knowledge and rigorous interplay between team members to attain ideal operational efficiency or "flow." Virtual Reality (VR) represents a technology by which to rehearse complex plans and communicate precise steps to a surgical team prior to entering the operating room. The objective of this study was to evaluate the use of VR for preoperative surgical team planning and interdisciplinary communication across all surgical specialties.

METHODS

A systematic review of the literature was performed examining existing research on VR use for preoperative surgical team planning and interdisciplinary communication across all surgical fields in order to optimize surgical efficiency. MEDLINE, SCOPUS, CINAHL databases were searched from inception to July 31, 2022 using standardized search clauses. A qualitative data synthesis was performed with particular attention to preoperative planning, surgical efficiency optimization, and interdisciplinary collaboration/communication techniques determined a priori. Preferred Reporting Items for Systematic Review and Meta-Analysis guidelines were followed. All included studies were appraised for their quality using the Medical Education Research Study Quality Instrument (MERSQI) tool.

RESULTS

One thousand and ninety-three non-duplicated articles with abstract and full text availability were identified. Thirteen articles that examined preoperative VR-based planning techniques for optimization of surgical efficiency and/or interdisciplinary communication fulfilled inclusion and exclusion criteria. These studies had a low-to-medium methodological quality with a MERSQI mean score of 10.04 out of 18 (standard deviation 3.61).

CONCLUSIONS

This review demonstrates that time spent rehearsing and visualizing patient-specific anatomical relationships in VR may improve operative efficiency and communication across multiple surgical specialties.

What factors influence continuous usage intention of head-mounted display-based virtual reality content?: a cross-sectional survey

PURPOSE

The aim of this study was to explore the continuous usage intention of head-mounted display-based virtual reality (HMD-based VR) content among college students. The study also sought to understand how this intention is influenced by factors related to personal cognition, social aspects, VR content, and HMD-related elements.

METHODS

This descriptive correlational study used a self-report questionnaire to survey 217 students from two universities in Korea who had prior experience with HMD-based VR content.

RESULTS

The mean score for continuous usage intention of HMD-based VR content was 2.59±0.57 points (range, 1-5 points). Regarding the average frequency of HMD-based VR content usage, 64.5% of participants reported using it 1 to 2 times, while 91.7% indicated a total HMD-based VR usage period of less than 6 months. Factors such as personal cognition, VR content, social aspects, and HMD-related elements had explanatory power of 35.1%, 10.7%, 4.4%, and 2.5%, respectively, for the continuous usage intention of HMD-based VR content. Additionally, engagement (β=.45, p<.001), influential others (β=.37, p<.001), environmental support (β=-.18, p=.030), and cyber sickness (β=-.21, p=.001) were identified as having a significant influence.

CONCLUSION

When developing HMD-based VR content, strategies to improve users' personal cognition should be included. Additionally, it is necessary to develop strategies that enhance enjoyment and interest in the content, while also facilitating ongoing social support. Furthermore, coping strategies should be devised that take into account cyber sickness, a potential side effect of these devices.

The Role of Immersive Virtual Reality and Augmented Reality in Medical Communication: A Scoping Review

Communication between clinicians and patients and communication within clinical teams is widely recognized as a tool through which improved patient outcomes can be achieved. As emerging technologies, there is a notable lack of commentary on the role of immersive virtual reality (VR) and augmented reality (AR) in enhancing medical communication. This scoping review aims to map the current landscape of literature on this topic and highlights gaps in the evidence to inform future endeavors. A comprehensive search strategy was conducted across 3 databases (PubMed, Web of Science, and Embase), yielding 1000 articles, of which 623 were individually screened for relevance. Ultimately, 22 articles were selected for inclusion and review. Similarities across the cohort of studies included small sample sizes, observational study design, use of questionnaires, and more VR studies than AR. The majority of studies found these technologies to improve medical communication, although user tolerability limitations were identified. More studies are required, presenting more robust findings, in order to draw more definitive conclusions and stronger recommendations for use of immersive VR/AR in clinical environments.

Toward the validation of VR-HMDs for medical education: a systematic literature review

The latest technological advancements in the domain of virtual reality (VR) have created new opportunities to use VR as a training platform for medical students and practitioners more broadly. Despite the growing interest in the use of VR as a training tool, a commonly identified gap in VR-training for medical education is the confidence in the long-term validity of the applications. A systematic literature review was undertaken to explore the extent of VR (in particular head-mounted displays) applications for medical training with an additional focus on validation measures. The papers included in this review discussed empirical case studies of specific applications; however, these were mostly concerned with human-computer interaction and were polarized between demonstrating that a conceptual technology solution was feasible for simulation or looked at specific areas of VR usability with little discussion on validation measures for long-term training effectiveness and outcomes. The review uncovered a wide range of ad hoc applications and studies in terms of technology vendors, environments, tasks, envisaged users and effectiveness of learning outcomes. This presents decision-making challenges for those seeking to adopt, implement and embed such systems in teaching practice. The authors of this paper then take a wider socio-technical systems perspective to understand how the holistic training system can be engineered and validated effectively as fit for purpose, through distillation of a generic set of requirements from the literature review to aid design specification and implementation, and to drive more informed and traceable validation of these types of systems. In this review, we have identified 92 requirement statements in 11 key areas against which a VR-HMD training system could be validated; these were grouped into design considerations, learning mechanisms and implementation considerations.

Immersive Virtual Reality for Patient-Specific Preoperative Planning: A Systematic Review

Background. Immersive virtual reality (iVR) facilitates surgical decision-making by enabling surgeons to interact with complex anatomic structures in realistic 3-dimensional environments. With emerging interest in its applications, its effects on patients and providers should be clarified. This systematic review examines the current literature on iVR for patient-specific preoperative planning. Materials and Methods. A literature search was performed on five databases for publications from January 1, 2000 through March 21, 2021. Primary studies on the use of iVR simulators by surgeons at any level of training for patient-specific preoperative planning were eligible. Two reviewers independently screened titles, abstracts, and full texts, extracted data, and assessed quality using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD). Results were qualitatively synthesized, and descriptive statistics were calculated. Results. The systematic search yielded 2,555 studies in total, with 24 full-texts subsequently included for qualitative synthesis, representing 264 medical personnel and 460 patients. Neurosurgery was the most frequently represented discipline (10/24; 42%). Preoperative iVR did not significantly improve patient-specific outcomes of operative time, blood loss, complications, and length of stay, but may decrease fluoroscopy time. In contrast, iVR improved surgeon-specific outcomes of surgical strategy, anatomy visualization, and confidence. Validity, reliability, and feasibility of patient-specific iVR models were assessed. The mean QATSDD score of included studies was 32.9%. Conclusions. Immersive VR improves surgeon experiences of preoperative planning, with minimal evidence for impact on short-term patient outcomes. Future work should focus on high-quality studies investigating long-term patient outcomes, and utility of preoperative iVR for trainees.

Magnetic resonance cholangiopancreatography enhanced by virtual reality as a novel tool to improve the understanding of biliary anatomy and the teaching of surgical trainees

Objective

The novel picture archiving and communication system (PACS), compatible with virtual reality (VR) software, displays cross-sectional images in VR. VR magnetic resonance cholangiopancreatography (MRCP) was tested to improve the anatomical understanding and intraoperative performance of minimally invasive cholecystectomy (CHE) in surgical trainees.

Design

We used an immersive VR environment to display volumetric MRCP data (Specto VRTM). First, we evaluated the tolerability and comprehensibility of anatomy with a validated simulator sickness questionnaire (SSQ) and examined anatomical landmarks. Second, we compared conventional MRCP and VR MRCP by matching three-dimensional (3D) printed models and identifying and measuring common bile duct stones (CBDS) using VR MRCP. Third, surgical trainees prepared for CHE with either conventional MRCP or VR MRCP, and we measured perioperative parameters and surgical performance (validated GOALS score).

Setting

The study was conducted out at Clarunis, University Center for Gastrointestinal and Liver Disease, Basel, Switzerland.

Participants

For the first and second study step, doctors from all specialties and years of experience could participate. In the third study step, exclusively surgical trainees were included. Of 74 participating clinicians, 34, 27, and 13 contributed data to the first, second, and third study phases, respectively.

Results

All participants determined the relevant biliary structures with VR MRCP. The median SSQ score was 0.75 (IQR: 0, 3.5), indicating good tolerability. Participants selected the corresponding 3D printed model faster and more reliably when previously studying VR MRCP compared to conventional MRCP: We obtained a median of 90 s (IQR: 55, 150) and 72.7% correct answers with VR MRCP versus 150 s (IQR: 100, 208) and 49.6% correct answers with conventional MRCP, respectively (p &lt; 0.001). CBDS was correctly identified in 90.5% of VR MRCP cases. The median GOALS score was higher after preparation with VR MRCP than with conventional MRCP for CHE: 16 (IQR: 13, 22) and 11 (IQR: 11, 18), respectively (p = 0.27).

Conclusions

VR MRCP allows for a faster, more accurate understanding of displayed anatomy than conventional MRCP and potentially leads to improved surgical performance in CHE in surgical trainees.

Leberchirurgie 4.0 - OP-Planung, Volumetrie, Navigation und Virtuelle Realität

Durch die Optimierung der konservativen Behandlung, die Verbesserung der bildgebenden Verfahren und die Weiterentwicklung der Operationstechniken haben sich das operative Spektrum sowie der Maßstab für die Resektabilität in Bezug auf die Leberchirurgie in den letzten Jahrzehnten deutlich verändert.

Dank zahlreicher technischer Entwicklungen, insbesondere der 3-dimensionalen Segmentierung, kann heutzutage die präoperative Planung und die Orientierung während der Operation selbst, vor allem bei komplexen Eingriffen, unter Berücksichtigung der patientenspezifischen Anatomie erleichtert werden.

Neue Technologien wie 3-D-Druck, virtuelle und augmentierte Realität bieten zusätzliche Darstellungsmöglichkeiten für die individuelle Anatomie. Verschiedene intraoperative Navigationsmöglichkeiten sollen die präoperative Planung im Operationssaal verfügbar machen, um so die Patientensicherheit zu erhöhen.

Dieser Übersichtsartikel soll einen Überblick über den gegenwärtigen Stand der verfügbaren Technologien sowie einen Ausblick in den Operationssaal der Zukunft geben.

Three-dimensional visualization and virtual reality simulation role in hepatic surgery: Further research warranted

Artificial intelligence (AI) is the study of algorithms that enable machines to analyze and execute cognitive activities including problem solving, object and word recognition, reduce the inevitable errors to improve the diagnostic accuracy, and decision-making. Hepatobiliary procedures are technically complex and the use of AI in perioperative management can improve patient outcomes as discussed below. Three-dimensional (3D) reconstruction of images obtained via ultrasound, computed tomography scan or magnetic resonance imaging, can help surgeons better visualize the surgical sites with added depth perception. Pre-operative 3D planning is associated with lesser operative time and intraoperative complications. Also, a more accurate assessment is noted, which leads to fewer operative complications. Images can be converted into physical models with 3D printing technology, which can be of educational value to students and trainees. 3D images can be combined to provide 3D visualization, which is used for preoperative navigation, allowing for more precise localization of tumors and vessels. Nevertheless, AI enables surgeons to provide better, personalized care for each patient.

Virtual Simulation in Undergraduate Medical Education: A Scoping Review of Recent Practice

Virtual simulation (VS) as an emerging interactive pedagogical strategy has been paid more and more attentions in the undergraduate medical education. Because of the fast development of modern computer simulation technologies, more and more advanced and emerging VS-based instructional practices are constantly increasing to promote medical education in diverse forms. In order to describe an overview of the current trends in VS-based medical teaching and learning, this scoping review presented a worldwide analysis of 92 recently published articles of VS in the undergraduate medical teaching and learning. The results indicated that 98% of included articles were from Europe, North America, and Asia, suggesting a possible inequity in digital medical education. Half (52%) studies reported the immersive virtual reality (VR) application. Evidence for educational effectiveness of VS in medical students' knowledge or skills was sufficient as per Kirkpatrick's model of outcome evaluation. Recently, VS has been widely integrated in surgical procedural training, emergency and pediatric emergency medicine training, teaching of basic medical sciences, medical radiation and imaging, puncture or catheterization training, interprofessional medical education, and other case-based learning experiences. Some challenges, such as accessibility of VS instructional resources, lack of infrastructure, "decoupling" users from reality, as well as how to increase students' motivation and engagement, should be addressed.

Investigating the utility of VR for spatial understanding in surgical planning: evaluation of head-mounted to desktop display

Recent technological advances have made Virtual Reality (VR) attractive in both research and real world applications such as training, rehabilitation, and gaming. Although these other fields benefited from VR technology, it remains unclear whether VR contributes to better spatial understanding and training in the context of surgical planning. In this study, we evaluated the use of VR by comparing the recall of spatial information in two learning conditions: a head-mounted display (HMD) and a desktop screen (DT). Specifically, we explored (a) a scene understanding and then (b) a direction estimation task using two 3D models (i.e., a liver and a pyramid). In the scene understanding task, participants had to navigate the rendered the 3D models by means of rotation, zoom and transparency in order to substantially identify the spatial relationships among its internal objects. In the subsequent direction estimation task, participants had to point at a previously identified target object, i.e., internal sphere, on a materialized 3D-printed version of the model using a tracked pointing tool. Results showed that the learning condition (HMD or DT) did not influence participants' memory and confidence ratings of the models. In contrast, the model type, that is, whether the model to be recalled was a liver or a pyramid significantly affected participants' memory about the internal structure of the model. Furthermore, localizing the internal position of the target sphere was also unaffected by participants' previous experience of the model via HMD or DT. Overall, results provide novel insights on the use of VR in a surgical planning scenario and have paramount implications in medical learning by shedding light on the mental model we make to recall spatial structures.

Using virtual 3D-models in surgical planning: workflow of an immersive virtual reality application in liver surgery

Purpose

Three-dimensional (3D) surgical planning is widely accepted in liver surgery. Currently, the 3D reconstructions are usually presented as 3D PDF data on regular monitors. 3D-printed liver models are sometimes used for education and planning.

Methods

We developed an immersive virtual reality (VR) application that enables the presentation of preoperative 3D models. The 3D reconstructions are exported as STL files and easily imported into the application, which creates the virtual model automatically. The presentation is possible in “OpenVR”-ready VR headsets. To interact with the 3D liver model, VR controllers are used. Scaling is possible, as well as changing the opacity from invisible over transparent to fully opaque. In addition, the surgeon can draw potential resection lines on the surface of the liver. All these functions can be used in a single or multi-user mode.

Results

Five highly experienced HPB surgeons of our department evaluated the VR application after using it for the very first time and considered it helpful according to the “System Usability Scale” (SUS) with a score of 76.6%. Especially with the subitem “necessary learning effort,” it was shown that the application is easy to use.

Conclusion

We introduce an immersive, interactive presentation of medical volume data for preoperative 3D liver surgery planning. The application is easy to use and may have advantages over 3D PDF and 3D print in preoperative liver surgery planning. Prospective trials are needed to evaluate the optimal presentation mode of 3D liver models.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00423-021-02127-7.