Virtual and augmented reality for biomedical applications

From static web to metaverse: reinventing medical education in the post-pandemic era

The World Wide Web and the advancement of computer technology in the 1960s and 1990s respectively set the ground for a substantial and simultaneous change in many facets of our life, including medicine, health care, and medical education. The traditional didactic approach has shifted towards more dynamic and interactive methods, leveraging technologies such as simulation tools, virtual reality, and online platforms. At the forefront is the remarkable evolution that has revolutionized how medical knowledge is accessed, disseminated, and integrated into pedagogical practices. The COVID-19 pandemic also led to rapid and large-scale adoption of e-learning and digital resources in medical education because of widespread lockdowns, social distancing measures, and the closure of medical schools and healthcare training programs. This review paper examines the evolution of medical education from the Flexnerian era to the modern digital age, closely examining the influence of the evolving WWW and its shift from Education 1.0 to Education 4.0. This evolution has been further accentuated by the transition from the static landscapes of Web 2D to the immersive realms of Web 3D, especially considering the growing notion of the metaverse. The application of the metaverse is an interconnected, virtual shared space that includes virtual reality (VR), augmented reality (AR), and mixed reality (MR) to create a fertile ground for simulation-based training, collaborative learning, and experiential skill acquisition for competency development. This review includes the multifaceted applications of the metaverse in medical education, outlining both its benefits and challenges. Through insightful case studies and examples, it highlights the innovative potential of the metaverse as a platform for immersive learning experiences. Moreover, the review addresses the role of emerging technologies in shaping the post-pandemic future of medical education, ultimately culminating in a series of recommendations tailored for medical institutions aiming to successfully capitalize on revolutionary changes.

Bone-Targeted Supramolecular Nanoagonist Assembled by Accurate Ratiometric Herbal-Derived Therapeutics for Osteoporosis Reversal

Developing novel strategies for defeating osteoporosis has become a world-wide challenge with the aging of the population. In this work, novel supramolecular nanoagonists (NAs), constructed from alkaloids and phenolic acids, emerge as a carrier-free nanotherapy for efficacious osteoporosis treatment. These precision nanoagonists are formed through the self-assembly of berberine (BER) and chlorogenic acid (CGA), utilizing noncovalent electrostatic, π-π, and hydrophobic interactions. This assembly results in a 100% drug loading capacity and stable nanostructure. Furthermore, the resulting weights and proportions of CGA and BER within the NAs are meticulously controlled with strong consistency when the CGA/BER assembly feed ratio is altered from 1:1 to 1:4. As anticipated, our NAs themselves could passively target osteoporotic bone tissues following prolonged blood circulation, modulate Wnt signaling, regulate osteogenic differentiation, and ameliorate bone loss in ovariectomy-induced osteoporotic mice. We hope this work will open a new strategy to design efficient herbal-derived Wnt NAs for dealing with intractable osteoporosis.

3D visualization technology for Learning human anatomy among medical students and residents: a meta- and regression analysis

BACKGROUND

3D visualization technology applies computers and other devices to create a realistic virtual world for individuals with various sensory experiences such as 3D vision, touch, and smell to gain a more effective understanding of the relationships between real spatial structures and organizations. The purpose of this study was to comprehensively evaluate the effectiveness of 3D visualization technology in human anatomy teaching/training and explore the potential factors that affect the training effects to better guide the teaching of classroom/laboratory anatomy.

METHODS

We conducted a meta-analysis of randomized controlled studies on teaching human anatomy using 3D visualization technology. We extensively searched three authoritative databases, PubMed, Web of Science, and Embase; the main outcomes were the participants' test scores and satisfaction, while the secondary outcomes were time consumption and enjoyment. Heterogeneity by I² was statistically determined because I²> 50%; therefore, a random-effects model was employed, using data processing software such as RevMan, Stata, and VOSviewer to process data, apply standardized mean difference and 95% confidence interval, and subgroup analysis to evaluate test results, and then conduct research through sensitivity analysis and meta-regression analysis.

RESULTS

Thirty-nine randomized controlled trials (2,959 participants) were screened and included in this study. The system analysis of the main results showed that compared with other methods, including data from all regions 3D visualization technology moderately improved test scores as well as satisfaction and enjoyment; however, the time that students took to complete the test was not significantly reduced. Meta-regression analysis also showed that regional factorsaffected test scores, whereas other factors had no significant impact. When the literature from China was excluded, the satisfaction and happiness of the 3D virtual-reality group were statistically significant compared to those of the traditional group; however, the test results and time consumption were not statistically significant.

CONCLUSION

3D visualization technology is an effective way to improve learners' satisfaction with and enjoyment of human anatomical learning, but it cannot reduce the time required for testers to complete the test. 3D visualization technology may struggle to improve the testers' scores. The literature test results from China are more prone to positive results and affected by regional bias.

Performance on a target acquisition task differs between augmented reality and touch screen displays

Target acquisition tasks quantify human motor and perceptual abilities while performing discrete tasks to support interface design and sensorimotor assessments. This study investigated the effects of display, Touchscreen and Augmented Reality (AR), on a standardized 2D multidirectional target acquisition task. Thirty-two participants performed the target acquisition task with both modality types and at two indexes of difficulty. The touchscreen modality yielded improved performance over AR as measured by accuracy, precision, error rates, throughput, and movement time. Throughput using the nominal index of difficulty was 10.12 bits/s for touchscreen and 3.11 bits/s for AR. AR designers can use the results to improve performance when designing AR interfaces by selecting larger buttons when accuracy and efficiency are required and by embedding perception cues to button target surfaces such as depth and proximity cues.

iVR-fNIRS: studying brain functions in a fully immersive virtual environment

Immersive virtual reality (iVR) employs head-mounted displays or cave-like environments to create a sensory-rich virtual experience that simulates the physical presence of a user in a digital space. The technology holds immense promise in neuroscience research and therapy. In particular, virtual reality (VR) technologies facilitate the development of diverse tasks and scenarios closely mirroring real-life situations to stimulate the brain within a controlled and secure setting. It also offers a cost-effective solution in providing a similar sense of interaction to users when conventional stimulation methods are limited or unfeasible. Although combining iVR with traditional brain imaging techniques may be difficult due to signal interference or instrumental issues, recent work has proposed the use of functional near infrared spectroscopy (fNIRS) in conjunction with iVR for versatile brain stimulation paradigms and flexible examination of brain responses. We present a comprehensive review of current research studies employing an iVR-fNIRS setup, covering device types, stimulation approaches, data analysis methods, and major scientific findings. The literature demonstrates a high potential for iVR-fNIRS to explore various types of cognitive, behavioral, and motor functions in a fully immersive VR (iVR) environment. Such studies should set a foundation for adaptive iVR programs for both training (e.g., in novel environments) and clinical therapeutics (e.g., pain, motor and sensory disorders and other psychiatric conditions).

Dissecting human anatomy learning process through anatomical education with augmented reality: AEducAR 2.0, an updated interdisciplinary study

Anatomical education is pivotal for medical students, and innovative technologies like augmented reality (AR) are transforming the field. This study aimed to enhance the interactive features of the AEducAR prototype, an AR tool developed by the University of Bologna, and explore its impact on human anatomy learning process in 130 second-year medical students at the International School of Medicine and Surgery of the University of Bologna. An interdisciplinary team of anatomists, maxillofacial surgeons, biomedical engineers, and educational scientists collaborated to ensure a comprehensive understanding of the study's objectives. Students used the updated version of AEducAR, named AEducAR 2.0, to study three anatomical topics, specifically the orbit zone, facial bones, and mimic muscles. AEducAR 2.0 offered two learning activities: one explorative and one interactive. Following each activity, students took a test to assess learning outcomes. Students also completed an anonymous questionnaire to provide background information and offer their perceptions of the activity. Additionally, 10 students participated in interviews for further insights. The results demonstrated that AEducAR 2.0 effectively facilitated learning and students' engagement. Students totalized high scores in both quizzes and declared to have appreciated the interactive features that were implemented. Moreover, interviews shed light on the interesting topic of blended learning. In particular, the present study suggests that incorporating AR into medical education alongside traditional methods might prove advantageous for students' academic and future professional endeavors. In this light, this study contributes to the growing research emphasizing the potential role of AR in shaping the future of medical education.

Immersive virtual reality-based learning as a supplement for biomedical engineering labs: challenges faced and lessons learned

Introduction

Immersive virtual reality (VR) based laboratory demonstrations have been gaining traction in STEM education as they can provide virtual hands-on experience. VR can also facilitate experiential and visual learning and enhanced retention. However, several optimizations of the implementation, in-depth analyses of advantages and trade-offs of the technology, and assessment of receptivity of modern techniques in STEM education are required to ensure better utilization of VR-based labs.

Methods

In this study, we developed VR-based demonstrations for a biomolecular engineering laboratory and assessed their effectiveness using surveys containing free responses and 5-point Likert scale-based questions. Insta360 Pro2 camera and Meta Quest 2 headsets were used in combination with an in-person lab. A cohort of 53 students watched the experimental demonstration on VR headsets in the lab after a brief lab overview in person and then performed the experiments in the lab.

Results

Only 28.29% of students reported experiencing some form of discomfort after using the advanced VR equipment as opposed to 63.63% of students from the previous cohort. About 40% of the students reported that VR eliminated or reduced auditory and visual distractions from the environment, the length of the videos was appropriate, and they received enough information to understand the tasks.

Discussion

The traditional lab method was found to be more suitable for explaining background information and lab concepts while the VR was found to be suitable for demonstrating lab procedures and tasks. Analyzing open-ended questions revealed several factors and recommendations to overcome the potential challenges and pitfalls of integrating VR with traditional modes of learning. This study provides key insights to help optimize the implementation of immersive VR to effectively supplement in-person learning experiences.

Virtual reality images created on the back and front of a display

To better investigate the biological mechanism of microorganisms, we developed a novel, to the best of our knowledge, virtual reality (VR) microscope that incorporates a head-mounted display (HMD) that creates VR images with a digital microscope. This type of VR microscope can be used with any type of optical microscope. The fabricated microscope is quite different from a common bifocal device because it can create VR images on the back and front of a display. If the VR images are displayed with object (OBJ) images, they are observable in [2 × 2] (back and front VR images and OBJ images; 2 × 2 = 4 images). This feature can provide important information on microscopic OBJs, which can be employed in 3D biological analysis. Furthermore, if a laser light source is added to this microscope, the images can be observed in [3 × 2] (back and front laser VR images, VR images, and OBJ images; 3 × 2 = 6 images). The lasers would also enable optical trapping and tracking, leading to improved biological analysis.

Exergaming in augmented reality is tailor-made for aerobic training and enjoyment among healthy young adults

In recent years, the use of exergaming for physical activity practice has gain in popularity but few is known about the use of augmented reality for physical activity, particularly at moderate to vigorous intensities. The present study examined the use of an exergame in augmented reality for aerobic training in healthy young adults. In a within-subject design, 18 participants (19.8 ± 1.4 years of age) have performed two physical activity sessions playing dodgeball. Indeed, they realized a classical dodgeball session and an exergaming session with an augmented reality version of dodgeball game. Physical loads and intensities were measured with accelerometers, RPE and heart sensors. Enjoyment experienced during the sessions was measured with the short version of the physical activity enjoyment scale questionnaire. Results revealed that both physical load and intensity were appropriate for aerobic training in the two conditions (i.e., classical and exergame in augmented reality) although values were significantly higher in the classical condition. Enjoyment was high in the two conditions with a higher significant value in the classical condition compared to the exergame in augmented reality condition. Put together, these results indicate that an aerobic state can be attained through both physical gameplay and its augmented reality equivalent and was associated to a high level of enjoyment among healthy young adults.

Virtual, augmented, and mixed reality: potential clinical and training applications in pediatrics

BACKGROUND

COVID-19 pandemic has significantly impacted the field of medical training, necessitating innovative approaches to education and practice. During this period, the use of novel technologies like virtual reality (VR), augmented reality (AR), and mixed reality (MR) has become increasingly vital. These technologies offer the advantage of transcending the limitations of time and space, thus enabling medical professionals to access various personalized programs for both education and service delivery. This shift is particularly relevant in the realm of pediatric medicine, where traditional training and clinical methods face unique challenges.

PURPOSE

The primary aim of this study is to explore the application of VR, AR, and MR technologies in pediatric medical settings, with a focus on both clinical applications and the training of pediatric medical professionals. We aim to comprehensively search and review studies that have utilized these technologies in the treatment of pediatric patients and the education of healthcare providers in this field.

METHODS

Peer-reviewed articles published in PubMed, the Cochrane Library, ScienceDirect, Google Scholar, and Scopus from January 1, 2018, to March 1, 2023, were comprehensively searched. The review was conducted according to the PRISMA (Preferred Reporting Items for Systematic review and Meta-Analyses) guidelines. Among the 89 studies, 63 investigated the clinical applications of VR (n=60) or AR (n=3) in pediatric patients, and 25 investigated the applications of VR (n=19), AR (n=5), or MR (n=1) for training medical professionals.

RESULTS

A total of 36 randomized controlled trials (RCTs) for clinical application (n=31) and medical training (n=5) were retrieved. Among the RCTs, 21 reported significant improvements in clinical applications (n=17) and medical training (n=4).

CONCLUSION

Despite a few limitations in conducting research on innovative technology, such research has rapidly expanded, indicating that an increasing number of researchers are involved in pediatric research using these technologies.

An architecture for collaboration in systems biology at the age of the Metaverse

As the current state of the Metaverse is largely driven by corporate interests, which may not align with scientific goals and values, academia should play a more active role in its development. Here, we present the challenges and solutions for building a Metaverse that supports systems biology research and collaboration. Our solution consists of two components: Kosmogora, a server ensuring biological data access, traceability, and integrity in the context of a highly collaborative environment such as a metaverse; and ECellDive, a virtual reality application to explore, interact, and build upon the data managed by Kosmogora. We illustrate the synergy between the two components by visualizing a metabolic network and its flux balance analysis. We also argue that the Metaverse of systems biology will foster closer communication and cooperation between experimentalists and modelers in the field.

Evolution of anxiety management in prostate biopsy under local anesthesia: a narrative review

INTRODUCTION AND METHODS

Prostate biopsy (PB) is an essential step in the diagnosis and active surveillance of prostate cancer (PCa). Transperineal PB (TP-PB) is now the recommended approach and is mostly conducted under local anesthesia. However, this procedure can potentially cause anxiety for patients, given the oncological context and the fear of peri-procedural pain and complications. The objective of this narrative review is to summarize the currently available tools for the management of peri-interventional anxiety during TP-PB, with a particular emphasis on the potential role of virtual reality (VR) in this setting.

RESULTS

In TP-PB, preoperative anxiety can lead to increased pain perception, longer procedure time, and decreased patient satisfaction. Pharmacological and non-pharmacological approaches have been explored to reduce anxiety, such as premedication, deep sedation, education, relaxation techniques, hypnosis, and music therapy, albeit with mixed results. VR has recently emerged in the technological armamentarium for managing pain and anxiety, and the efficiency of this technology has been evaluated in various medical fields, including pediatrics, gastroenterology, urology, gynecology, and psychiatry.

CONCLUSION

Despite the paucity of available data, VR appears to be a safe and effective technique in reducing anxiety in many procedures, even in frail patients. No studies have evaluated the role of VR in TP-PB. Future research should thus explore the optimal way to implement VR technology and any potential benefits for TP-PB patients.

Exploring the potential role for extended reality in Mohs micrographic surgery

Mohs micrographic surgery (MMS) is a cornerstone of dermatological practice. Virtual reality (VR) and augmented reality (AR) technology, initially used for entertainment, have entered healthcare, offering real-time data overlaying a surgeon's view. This paper explores potential applications of VR and AR in MMS, emphasizing their advantages and limitations. We aim to identify research gaps to facilitate innovation in dermatological surgery. We conducted a PubMed search using the following: "augmented reality" OR "virtual reality" AND "Mohs" or "augmented reality" OR "virtual reality" AND "surgery." Inclusion criteria were peer-reviewed articles in English discussing these technologies in medical settings. We excluded non-peer-reviewed sources, non-English articles, and those not addressing these technologies in a medical context. VR alleviates patient anxiety and enhances patient satisfaction while serving as an educational tool. It also aids physicians by providing realistic surgical simulations. On the other hand, AR assists in real-time lesion analysis, optimizing incision planning, and refining margin control during surgery. Both of these technologies offer remote guidance for trainee residents, enabling real-time learning and oversight and facilitating synchronous teleconsultations. These technologies may transform dermatologic surgery, making it more accessible and efficient. However, further research is needed to validate their effectiveness, address potential challenges, and optimize seamless integration. All in all, AR and VR enhance real-world environments with digital data, offering real-time surgical guidance and medical insights. By exploring the potential integration of these technologies in MMS, our study identifies avenues for further research to thoroughly understand the role of these technologies to redefine dermatologic surgery, elevating precision, surgical outcomes, and patient experiences.

Surgical Perspectives of Open vs. Laparoscopic Approaches to Lateral Pancreaticojejunostomy: A Comprehensive Review

Pancreaticojejunostomy, a critical step in pancreatic surgery, has significantly evolved surgical approaches, including open, laparoscopic, and robotic techniques. This comprehensive review explores open surgery's historical success, advantages, and disadvantages, emphasizing surgeons' accrued experience and familiarity with this approach. However, heightened morbidity and prolonged recovery associated with open pancreaticojejunostomy underscore the need for a nuanced evaluation of alternatives. The advent of robotic-assisted surgery introduces a paradigm shift in pancreatic procedures. Enhanced dexterity, facilitated by wristed instruments, allows intricate suturing and precise tissue manipulation crucial in pancreatic surgery. Three-dimensional visualization augments surgeon perception, improving spatial orientation and anastomotic alignment. Moreover, the potential for a reduced learning curve may enhance accessibility, especially for surgeons transitioning from open techniques. Emerging technologies, including advanced imaging modalities and artificial intelligence, present promising avenues for refining both open and minimally invasive approaches. The ongoing pursuit of optimal outcomes mandates a judicious consideration of surgical techniques, incorporating technological advancements to navigate challenges and enhance patient care in pancreaticojejunostomy.

Validation of real-time inside-out tracking and depth realization technologies for augmented reality-based neuronavigation

PURPOSE

Concomitant with the significant advances in computing technology, the utilization of augmented reality-based navigation in clinical applications is being actively researched. In this light, we developed novel object tracking and depth realization technologies to apply augmented reality-based neuronavigation to brain surgery.

METHODS

We developed real-time inside-out tracking based on visual inertial odometry and a visual inertial simultaneous localization and mapping algorithm. The cube quick response marker and depth data obtained from light detection and ranging sensors are used for continuous tracking. For depth realization, order-independent transparency, clipping, and annotation and measurement functions were developed. In this study, the augmented reality model of a brain tumor patient was applied to its life-size three-dimensional (3D) printed model.

RESULTS

Using real-time inside-out tracking, we confirmed that the augmented reality model remained consistent with the 3D printed patient model without flutter, regardless of the movement of the visualization device. The coordination accuracy during real-time inside-out tracking was also validated. The average movement error of the X and Y axes was 0.34 ± 0.21 and 0.04 ± 0.08 mm, respectively. Further, the application of order-independent transparency with multilayer alpha blending and filtered alpha compositing improved the perception of overlapping internal brain structures. Clipping, and annotation and measurement functions were also developed to aid depth perception and worked perfectly during real-time coordination. We named this system METAMEDIP navigation.

CONCLUSIONS

The results validate the efficacy of the real-time inside-out tracking and depth realization technology. With these novel technologies developed for continuous tracking and depth perception in augmented reality environments, we are able to overcome the critical obstacles in the development of clinically applicable augmented reality neuronavigation.

The status of virtual simulation experiments in medical education in China: based on the national virtual simulation experiment teaching Center (iLAB-X)

BACKGROUND

Virtual simulation experiments have been rapidly applied to medical education curricula in recent years. China constructed a national virtual simulation experimental teaching center (iLAB-X), and this platform covered almost all of the virtual simulation experiment curricula of domestic colleges or universities. We aimed to comprehensively assess the characteristics and usages of virtual simulation experiments in medical education based on iLAB-X.

METHODS

A total of 480 virtual simulation experiment courses had been constructed on iLAB-X (https://www.ilab-x.com/) by December 20, 2022, and the curriculum level, type and design were all searched in this platform. We also conducted an evaluation of curriculum usage and online tests, including the page view, frequency of participation, number of participants, duration of experimental learning and passing rate of the experimental test.

RESULTS

The national and provincial high-quality virtual simulation experiment curricula accounted for 33.5% (161/480) and 35.8% (172/480), respectively. The curricula were mainly set as basic practice experiments (46.5%) and synthetic designing experiments (48.8%). Significantly, forensic medicine (100%), public health and preventive medicine (83%) and basic medical sciences (66%) focused on synthetic design experiments. In terms of usage experiments, the average duration of experimental learning was 25 minutes per course, and the average number of participants was just 1257. The average passing (score ≥60) rate of online tests was 80.6%, but the average rate of score ≥ 85 was only 58.5%. In particular, the average page views, the number of participants, the duration of learning and the test passing rate of clinical medicine were relatively low.

CONCLUSIONS

The curriculum design features, construction level and utilization rate varied in different medical majors. Virtual simulation experiments are particularly underutilized in clinical medicine. There is a long way for virtual simulation experiments to go to become a supplement or alternative for traditional medical education in the future.

Applications of Extended Reality in Orthopaedic Surgery

➤ Extended reality is a term that encompasses different modalities, including virtual reality, augmented reality, and mixed reality.➤ Although fully immersive virtual reality has benefits for developing procedural memory and technical skills, augmented and mixed reality are more appropriate modalities for preoperative planning and intraoperative utilization.➤ Current investigations on the role of extended reality in preoperative planning and intraoperative utilization are still in the early stages, but preliminarily show that extended reality technologies can help surgeons to be more accurate and efficient.

Metaverse Wearables for Immersive Digital Healthcare: A Review

The recent exponential growth of metaverse technology has been instrumental in reshaping a myriad of sectors, not least digital healthcare. This comprehensive review critically examines the landscape and future applications of metaverse wearables toward immersive digital healthcare. The key technologies and advancements that have spearheaded the metamorphosis of metaverse wearables are categorized, encapsulating all-encompassed extended reality, such as virtual reality, augmented reality, mixed reality, and other haptic feedback systems. Moreover, the fundamentals of their deployment in assistive healthcare (especially for rehabilitation), medical and nursing education, and remote patient management and treatment are investigated. The potential benefits of integrating metaverse wearables into healthcare paradigms are multifold, encompassing improved patient prognosis, enhanced accessibility to high-quality care, and high standards of practitioner instruction. Nevertheless, these technologies are not without their inherent challenges and untapped opportunities, which span privacy protection, data safeguarding, and innovation in artificial intelligence. In summary, future research trajectories and potential advancements to circumvent these hurdles are also discussed, further augmenting the incorporation of metaverse wearables within healthcare infrastructures in the post-pandemic era.

Application of flipped classroom combined with virtual simulation platform in clinical biochemistry practical course

BACKGROUND

The study explores an innovative teaching mode that integrates Icourse, DingTalk, and online experimental simulation platforms to provide online theoretical and experimental resources for clinical biochemistry practical courses. These platforms, combined with flipped classroom teaching, aim to increase student engagement and benefit in practical courses, ultimately improving the effectiveness of clinical biochemistry practical teaching.

METHODS

In a prospective cohort study, we examined the impact of integrating the Icourse and DingTalk platforms to provide theoretical knowledge resources and clinical cases to 48 medical laboratory science students from the 2019 and 2020 grades. Students were assigned to the experimental group using an overall sampling method, and had access to relevant videos through Icourse before and during class. Using a flipped classroom approach, students actively participated in the design, analysis, and discussion of the experimental technique. For the experimental operation part, students participated in virtual simulation experiments and actual experiments. Overall, the study aimed to evaluate students' theoretical and operational performance after completing the practical course. To collect feedback, we distributed a questionnaire to students in the experimental group. For comparison, we included 42 students from the grades of 2017 and 2018 who received traditional instruction and were evaluated using standard textbooks as the control group.

RESULTS

The experimental group scored significantly higher than the control group on both the theoretical and experimental operational tests (82.45 ± 3.76 vs. 76.36 ± 3.96, P = 0.0126; 92.03 ± 1.62 vs. 81.67 ± 4.19, P < 0.001). The survey revealed that the experimental group preferred the teaching mode that combined the flipped classroom with the virtual simulation platform. This mixed method effectively promoted understanding of basic knowledge (93.8%, 45/48), operative skills (89.6%, 43/48), learning interest (87.5%, 42/48), clinical thinking (85.4%, 41/48), self-learning ability (91.7%, 44/48), and overall satisfaction compared with traditional methods (P < 0.05). This study demonstrates that an innovative teaching approach significantly improves the quality of clinical biochemistry practical courses and promotes students' professional development and self-directed learning habits.

CONCLUSION

Incorporating virtual simulation with flipped classrooms into clinical biochemistry practical teaching is an efficient and well-received alternative to traditional methods.

Brave New Healthcare: A Narrative Review of Digital Healthcare in American Medicine

The digital revolution has had a profound effect on American and global healthcare, which was accelerated by the pandemic and telehealth applications. Digital health also includes popular and more esoteric forms of wearable monitoring systems and interscatter and other wireless technologies that facilitate their telemetry. The rise in artificial intelligence (AI) and machine learning (ML) may serve to improve interpretation from imaging technologies to electrocardiography or electroencephalographic tracings, and new ML techniques may allow these systems to scan data to discern and contextualize patterns that may have evaded human physicians. The necessity of virtual care during the pandemic has morphed into new treatment paradigms, which have gained patient acceptance but still raise issues with respect to privacy laws and credentialing. Augmented and virtual reality tools can facilitate surgical planning and "hands-on" clinical training activities. Patients are working with new frontiers in digital health in the form of "Dr. Google" and patient support websites to learn or share medical information. Patient-facing digital health information is both a blessing and curse, in that it can be a boon to health-literate patients who seek to be more active in their own care. On the other hand, digital health information can lead to false conclusions, catastrophizing, misunderstandings, and "cyberchondria." The role of blockchain, familiar from cryptocurrency, may play a role in future healthcare information and would serve as a disruptive, decentralizing, and potentially beneficial change. These important changes are both exciting and perplexing as clinicians and their patients learn to navigate this new system and how we address the questions it raises, such as medical privacy in a digital age. The goal of this review is to explore the vast range of digital health and how it may impact the healthcare system.

Ethical enactivism for smart and inclusive STEAM learning design

Current global challenges of the 21st century promote STEAM (science, technology, engineering, arts and mathematics) education and digitalization as a means for humans to be the central actors in the construction of a sustainable society that favors a sense of worth and global wellbeing. In this scenario, new educational technology tools and immersive learning affordances (possibilities), offer unprecedented potential for the design of smart and dynamic learning systems and contexts that can enhance learning processes across varied audiences and educational settings. However, current STEAM education practice lacks attention to equipping all citizens with the necessary skills to use digital technologies in an ethical, critical and creative way. This gap calls for attention in design processes, principles and practices that are attentive to ethical considerations and values-based approaches. On the other hand, in its formulation STEAM as an educational approach is framed in four fundamental pillars: creativity, inclusion, citizenship and emerging technologies, which also put attention on the inclusion of disadvantaged and underrepresented social groups during STEAM education design. Following an apparent need to explore ethical and inclusive design in STEAM education, and inspired in the 4E cognition framework, ethical enactivism and embodied and ecosomaesthetics experience design, here we propose a theoretical framework grounded on systems thinking for the design of smart and dynamic STEAM learning systems and settings. The framework is aimed at STEAM educational psychologists, educational technologists, learning designers and educational practitioners who wish to address the global challenges of 21st century education by means of creative, innovative and inclusive education design.

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future.

Extended reality as a health education strategy of adolescents at school: protocol for systematic review and meta-analysis

INTRODUCTION

Extended reality (XR) is the ensemble of interactive experiences based on a computer-simulated environment that encompasses virtual reality and augmented reality and has been proven to be potentially innovative in the field of health education with adolescents. The objective of this study is to present a systematic review and meta-analysis protocol that seeks to evaluate the main effects of interventions that use XR on health parameters (food intake, sound quality and physical activity) of adolescent students.

METHODS AND ANALYSIS

The literature search will be performed in the following databases: MEDLINE, Embase, Scopus, ERIC, ScienceDirect, Web of Science, Cochrane, LILACS, APA and ADOLEC. Intervention studies (clinical trials-randomised or non-randomised) and quasi-experimental studies will be included. The risk of bias will be assessed using the Risk of Bias in Non-randomized Studies of Interventions tool for randomised controlled trials (RCTs), non-RCTs and quasi-experimental trials. Two independent researchers will conduct all the assessments, and any disagreements will be consulted with a third reviewer. Data analysis and synthesis will be performed using RevMan V.5.4.1 software. The study will be conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols guideline.

ETHICS AND DISSEMINATION

Ethical approval and human consent were not required, as this is a protocol for a systematic review and only secondary data will be used. The findings will be published in a journal and presented at conferences. In case of any changes to this protocol, it will be updated in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses website, and the modifications will be explained in the final report of this review.

PROSPERO REGISTRATION NUMBER

CRD42022373876.

Developing virtual and augmented reality applications for science, technology, engineering and math education

The Rhode Island IDeA Network of Biomedical Research Excellence Molecular Informatics Core at the University of Rhode Island Information Technology Services Innovative Learning Technologies developed virtual and augmented reality applications to teach concepts in biomedical science, including pharmacology, medicinal chemistry, cell culture and nanotechnology. The apps were developed as full virtual reality/augmented reality and 3D gaming versions, which do not require virtual reality headsets. Development challenges included creating intuitive user interfaces, text-to-voice functionality, visualization of molecules and implementing complex science concepts. In-app quizzes are used to assess the user's understanding of topics, and user feedback was collected for several apps to improve the experience. The apps were positively reviewed by users and are being implemented into the curriculum at the University of Rhode Island.

[New perspectives in orthopedics : Developments through neurotechnology and metaverse]

The combination of neurotechnology and metaverse holds high potentials for orthopedics, as it offers a broad spectrum of possibilities to overcome the limits of traditional medical care. The vision of a medical metaverse providing the infrastructure as a link for innovative technologies opens up new opportunities for therapy, medical collaborations and practical, personalized training for aspiring physicians. However, risks and challenges, such as security and privacy, health-related issues, acceptance by patients and doctors, as well as technical hurdles and access to the technologies, remain. Hence, future research and development is paramount. Nonetheless, due to technological progress, the exploration of new research areas, and the improved availability of the technologies paired with cost reduction, the prospects for neurotechnology and metaverse in orthopedics are promising.

Extended reality for biomedicine

Extended reality (XR) refers to an umbrella of methods that allows users to be immersed in a three-dimensional (3D) or a 4D (spatial + temporal) virtual environment to different extents, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). While VR allows a user to be fully immersed in a virtual environment, AR and MR overlay virtual objects over the real physical world. The immersion and interaction of XR provide unparalleled opportunities to extend our world beyond conventional lifestyles. While XR has extensive applications in fields such as entertainment and education, its numerous applications in biomedicine create transformative opportunities in both fundamental research and healthcare. This Primer outlines XR technology from instrumentation to software computation methods, delineating the biomedical applications that have been advanced by state-of-the-art techniques. We further describe the technical advances overcoming current limitations in XR and its applications, providing an entry point for professionals and trainees to thrive in this emerging field.

Impact of Virtual Reality Cognitive and Motor Exercises on Brain Health

Innovative technologies of the 21st century have an extremely significant impact on all activities of modern humans. Among them, virtual reality (VR) offers great opportunities for scientific research and public health. The results of research to date both demonstrate the beneficial effects of using virtual worlds, and indicate undesirable effects on bodily functions. This review presents interesting recent findings related to training/exercise in virtual environments and its impact on cognitive and motor functions. It also highlights the importance of VR as an effective tool for assessing and diagnosing these functions both in research and modern medical practice. The findings point to the enormous future potential of these rapidly developing innovative technologies. Of particular importance are applications of virtual reality in basic and clinical neuroscience.

Effects of Fetal Images Produced in Virtual Reality on Maternal-Fetal Attachment: Randomized Controlled Trial

BACKGROUND

Maternal-fetal attachment (MFA) has been reported to be associated with the postpartum mother-infant relationship. Seeing the fetus through ultrasound might influence MFA, and the effect could be increased by more realistic images, such as those generated in virtual reality (VR).

OBJECTIVE

The aim was to determine the effect of fetal images generated in VR on MFA and depressive symptoms through a prenatal-coaching mobile app.

METHODS

This 2-arm parallel randomized controlled trial involved a total of 80 pregnant women. Eligible women were randomly assigned to either a mobile app-only group (n=40) or an app plus VR group (n=40). The VR group experienced their own baby's images generated in VR based on images obtained from fetal ultrasonography. The prenatal-coaching mobile app recommended health behavior for the pregnant women according to gestational age, provided feedback on entered data for maternal weight, blood pressure, and glucose levels, and included a private diary service for fetal ultrasound images. Both groups received the same app, but the VR group also viewed fetal images produced in VR; these images were stored in the app. All participants filled out questionnaires to assess MFA, depressive symptoms, and other basic medical information. The questionnaires were filled out again after the interventions.

RESULTS

Basic demographic data were comparable between the 2 groups. Most of the assessments showed comparable results for the 2 groups, but the mean score to assess interaction with the fetus was significantly higher for the VR group than the control group (0.4 vs 0.1, P=.004). The proportion of participants with an increased score for this category after the intervention was significantly higher in the VR group than the control group (43% vs 13%, P=.005). The feedback questionnaire revealed that scores for the degree of perception of fetal appearance all increased after the intervention in the VR group.

CONCLUSIONS

The use of a mobile app with fetal images in VR significantly increased maternal interaction with the fetus.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04942197; https://clinicaltrials.gov/ct2/show/NCT04942197.

Comparative effectiveness of virtual reality (VR) vs 3D printed models of congenital heart disease in resident and nurse practitioner educational experience

BACKGROUND

Medical trainees frequently note that cardiac anatomy is difficult to conceive within a two dimensional framework. The specific anatomic defects and the subsequent pathophysiology in flow dynamics may become more apparent when framed in three dimensional models. Given the evidence of improved comprehension using such modeling, this study aimed to contribute further to that understanding by comparing Virtual Reality (VR) and 3D printed models (3DP) in medical education.

OBJECTIVES

We sought to systematically compare the perceived subjective effectiveness of Virtual Reality (VR) and 3D printed models (3DP) in the educational experience of residents and nurse practitioners.

METHODS

Trainees and practitioners underwent individual 15-minute teaching sessions in which features of a developmentally typical heart as well as a congenitally diseased heart were demonstrated using both Virtual Reality (VR) and 3D printed models (3DP). Participants then briefly explored each modality before filling out a short survey in which they identified which model (3DP or VR) they felt was more effective in enhancing their understanding of cardiac anatomy and associated pathophysiology. The survey included a binary summative assessment and a series of Likert scale questions addressing usefulness of each model type and degree of comfort with each modality.

RESULTS

Twenty-seven pediatric residents and 3 nurse practitioners explored models of a developmentally typical heart and tetralogy of Fallot pathology. Most participants had minimal prior exposure to VR (1.1 ± 0.4) or 3D printed models (2.1 ± 1.5). Participants endorsed a greater degree of understanding with VR models (8.5 ± 1) compared with 3D Printed models (6.3 ± 1.8) or traditional models of instruction (5.5 ± 1.5) p < 0.001. Most participants felt comfortable with modern technology (7.6 ± 2.1). 87% of participants preferred VR over 3DP.

CONCLUSIONS

Our study shows that, overall, VR was preferred over 3DP models by pediatric residents and nurse practitioners for understanding cardiac anatomy and pathophysiology.

Contribution of the stereoscopic representation of motion-in-depth during visually guided feedback control

Considerable studies have focused on the neural basis of visually guided tracking movement in the frontoparallel plane, whereas the neural process in real-world circumstances regarding the influence of binocular disparity and motion-in-depth (MID) perception is less understood. Although the role of stereoscopic versus monoscopic MID information has been extensively described for visual processing, its influence on top-down regulation for motor execution has not received much attention. Here, we orthogonally varied the visual representation (stereoscopic versus monoscopic) and motion direction (depth motion versus bias depth motion versus frontoparallel motion) during visually guided tracking movements, with simultaneous functional near-infrared spectroscopy recordings. Results show that the stereoscopic representation of MID could lead to more accurate movements, which was supported by specific neural activity pattern. More importantly, we extend prior evidence about the role of frontoparietal network in brain-behavior relationship, showing that occipital area, more specifically, visual area V2/V3 was also robustly involved in the association. Furthermore, by using the stereoscopic representation of MID, it is plausible to detect robust brain-behavior relationship even with small sample size at low executive task demand. Taken together, these findings highlight the importance of the stereoscopic representation of MID for investigating neural correlates of visually guided feedback control.

Digital health: trends, opportunities and challenges in medical devices, pharma and bio-technology

Digital health interventions refer to the use of digital technology and connected devices to improve health outcomes and healthcare delivery. This includes telemedicine, electronic health records, wearable devices, mobile health applications, and other forms of digital health technology. To this end, several research and developmental activities in various fields are gaining momentum. For instance, in the medical devices sector, several smart biomedical materials and medical devices that are digitally enabled are rapidly being developed and introduced into clinical settings. In the pharma and allied sectors, digital health-focused technologies are widely being used through various stages of drug development, viz. computer-aided drug design, computational modeling for predictive toxicology, and big data analytics for clinical trial management. In the biotechnology and bioengineering fields, investigations are rapidly growing focus on digital health, such as omics biology, synthetic biology, systems biology, big data and personalized medicine. Though digital health-focused innovations are expanding the horizons of health in diverse ways, here the development in the fields of medical devices, pharmaceutical technologies and biotech sectors, with emphasis on trends, opportunities and challenges are reviewed. A perspective on the use of digital health in the Indian context is also included.

Development of metaverse for intelligent healthcare

The metaverse integrates physical and virtual realities, enabling humans and their avatars to interact in an environment supported by technologies such as high-speed internet, virtual reality, augmented reality, mixed and extended reality, blockchain, digital twins and artificial intelligence (AI), all enriched by effectively unlimited data. The metaverse recently emerged as social media and entertainment platforms, but extension to healthcare could have a profound impact on clinical practice and human health. As a group of academic, industrial, clinical and regulatory researchers, we identify unique opportunities for metaverse approaches in the healthcare domain. A metaverse of 'medical technology and AI' (MeTAI) can facilitate the development, prototyping, evaluation, regulation, translation and refinement of AI-based medical practice, especially medical imaging-guided diagnosis and therapy. Here, we present metaverse use cases, including virtual comparative scanning, raw data sharing, augmented regulatory science and metaversed medical intervention. We discuss relevant issues on the ecosystem of the MeTAI metaverse including privacy, security and disparity. We also identify specific action items for coordinated efforts to build the MeTAI metaverse for improved healthcare quality, accessibility, cost-effectiveness and patient satisfaction.

Ultra-Reliable Low-Latency Communications: Unmanned Aerial Vehicles Assisted Systems

Ultra-reliable low-latency communication (uRLLC) is a group of fifth-generation and sixth-generation (5G/6G) cellular applications with special requirements regarding latency, reliability, and availability. Most of the announced 5G/6G applications are uRLLC that require an end-to-end latency of milliseconds and ultra-high reliability of communicated data. Such systems face many challenges since traditional networks cannot meet such requirements. Thus, novel network structures and technologies have been introduced to enable such systems. Since uRLLC is a promising paradigm that covers many applications, this work considers reviewing the current state of the art of the uRLLC. This includes the main applications, specifications, and main requirements of ultra-reliable low-latency (uRLL) applications. The design challenges of uRLLC systems are discussed, and promising solutions are introduced. The virtual and augmented realities (VR/AR) are considered the main use case of uRLLC, and the current proposals for VR and AR are discussed. Moreover, unmanned aerial vehicles (UAVs) are introduced as enablers of uRLLC. The current research directions and the existing proposals are discussed.

Bioinformatics and the Metaverse: Are We Ready?

COVID-19 forced humanity to think about new ways of working globally without physically being present with other people, and eXtended Reality (XR) systems (defined as Virtual Reality, Augmented Reality and Mixed Reality) offer a potentially elegant solution. Previously seen as mainly for gaming, commercial and research institutions are investigating XR solutions to solve real world problems from training, simulation, mental health, data analysis, and studying disease progression. More recently large corporations such as Microsoft and Meta have announced they are developing the Metaverse as a new paradigm to interact with the digital world. This article will look at how visualization can leverage the Metaverse in bioinformatics research, the pros and cons of this technology, and what the future may hold.

Snake Robots for Surgical Applications: A Review

Although substantial advancements have been achieved in robot-assisted surgery, the blueprint to existing snake robotics predominantly focuses on the preliminary structural design, control, and human–robot interfaces, with features which have not been particularly explored in the literature. This paper aims to conduct a review of planning and operation concepts of hyper-redundant serpentine robots for surgical use, as well as any future challenges and solutions for better manipulation. Current researchers in the field of the manufacture and navigation of snake robots have faced issues, such as a low dexterity of the end-effectors around delicate organs, state estimation and the lack of depth perception on two-dimensional screens. A wide range of robots have been analysed, such as the i²Snake robot, inspiring the use of force and position feedback, visual servoing and augmented reality (AR). We present the types of actuation methods, robot kinematics, dynamics, sensing, and prospects of AR integration in snake robots, whilst addressing their shortcomings to facilitate the surgeon’s task. For a smoother gait control, validation and optimization algorithms such as deep learning databases are examined to mitigate redundancy in module linkage backlash and accidental self-collision. In essence, we aim to provide an outlook on robot configurations during motion by enhancing their material compositions within anatomical biocompatibility standards.

Feasibility of virtual reality based training for optimising COVID-19 case handling in Uganda

BACKGROUND

Epidemics and pandemics are causing high morbidity and mortality on a still-evolving scale exemplified by the COVID-19 pandemic. Infection prevention and control (IPC) training for frontline health workers is thus essential. However, classroom or hospital ward-based training portends an infection risk due to the in-person interaction of participants. We explored the use of Virtual Reality (VR) simulations for frontline health worker training since it trains participants without exposing them to infections that would arise from in-person training. It does away with the requirement for expensive personal protective equipment (PPE) that has been in acute shortage and improves learning, retention, and recall. This represents the first attempt in deploying VR-based pedagogy in a Ugandan medical education context.

METHODS

We used animated VR-based simulations of bedside and ward-based training scenarios for frontline health workers. The training covered the donning and doffing of PPE, case management of COVID-19 infected individuals, and hand hygiene. It used VR headsets to actualize an immersive experience, via a hybrid of fully-interactive VR and 360° videos. The level of knowledge acquisition between individuals trained using this method was compared to similar cohorts previously trained in a classroom setting. That evaluation was supplemented by a qualitative assessment based on feedback from participants about their experience.

RESULTS

The effort resulted in a COVID-19 IPC curriculum adapted into VR, corresponding VR content, and a pioneer cohort of VR trained frontline health workers. The formalized comparison with classroom-trained cohorts showed relatively better outcomes by way of skills acquired, speed of learning, and rates of information retention (P-value = 4.0e-09). In the qualitative assessment, 90% of the participants rated the method as very good, 58.1% strongly agreed that the activities met the course objectives, and 97.7% strongly indicated willingness to refer the course to colleagues.

CONCLUSION

VR-based COVID-19 IPC training is feasible, effective and achieves enhanced learning while protecting participants from infections within a pandemic setting in Uganda. It is a delivery medium transferable to the contexts of other highly infectious diseases.

A Self-Powered Triboelectric Hybrid Coder for Human-Machine Interaction

Human-machine interfaces have penetrated various academia and industry fields such as smartphones, robotic, virtual reality, and wearable electronics, due to their abundant functional sensors and information interaction methods. Nevertheless, most sensors' complex structural design, monotonous parameter detection capability, and single information coding communication hinder their rapid development. As the frontier of self-powered sensors, the triboelectric nanogenerator (TENG) has multiple working modes and high structural adaptability, which is a potential solution for multi-parameter sensing and miniaturizing of traditional interactive electronic devices. Herein, a self-powered hybrid coder (SHC) based on TENG is reported to encode two action parameters of touch and press, which can be used as a smart interface for human-machine interaction. The top-down hollow structure of the SHC, not only constructs a compositing mode to generate stable touch and press signals but also builds a hybrid coding platform for generating action codes in synergy mode. When a finger touches or presses the SHC, Morse code and Gray code can be transmitted for text information or remote control of electric devices. This self-powered coder is of reference value for designing an alternative human-machine interface and having the potential to contribute to the next generation of highly integrated portable smart electronics.

Quo Vadis Anesthesiologist? The Value Proposition of Future Anesthesiologists Lies in Preserving or Restoring Presurgical Health after Surgical Insult

This Special Issue of the Journal of Clinical Medicine is devoted to anesthesia and perioperative care [...].

Application and Visualization of Human 3D Anatomy Teaching for Healthy People Based on a Hybrid Network Model

With the development of computer technology, information technology, and 3D reconstruction technology of the medical human body, 3D virtual digital human body technology for human health has been widely used in various fields of medicine, especially in teaching students of application and anatomy. Its advantage is that it can view 3D human anatomy models from any angle and can be cut in any direction. In this paper, we propose an improved algorithm based on a hybrid density network and an element-level attention mechanism. The hybrid density network is used to generate feasible hypotheses for multiple 3D poses, solve the ambiguity problem in pose reasoning from 2D to 3D, and improve the performance of the network by adding the AReLU function combined with an element-wise attention mechanism. Teaching students in anatomy makes students' learning more convenient and teachers' teaching explanations more vivid. Comparative experiments show that the accuracy of 3D human pose estimation using a single image input is better than the other two-stage methods.

AnemiaAR: a serious game to support teaching of haematology

Serious games can be suitable tools for educational support in different areas of knowledge, such as Medicine. These applications, combined with technologies like mixed and augmented reality, provide a differentiated user experience that can keep or improve the interest and motivation of students and teachers during the teaching and learning process. In this context, this study aimed to present the development of AnemiaAR, a mixed reality serious game to support the teaching of haematology, helpful for students and professors in the visualisation and presentation of anaemia concepts. Fourteen medical students from the University of Passo Fundo participated in a pilot study to evaluate the application, considering a sociodemographic questionnaire, a questionnaire based on the Technology Acceptance Model, and two modules of the Game Experience Questionnaire. The preliminary results were satisfactory, showing good acceptance and positive experience of the game, besides improvements. The study also pointed out differences in the game evaluation among participants, considering the prior experience in games, the previously attended haematology subject, and the time spent performing the game tasks.

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

A Novel Gesture-Based Control System for Fluorescence Volumetric Data in Virtual Reality

With the development of light microscopy, it is becoming increasingly easy to obtain detailed multicolor fluorescence volumetric data. The need for their appropriate visualization has become an integral part of fluorescence imaging. Virtual reality (VR) technology provides a new way of visualizing multidimensional image data or models so that the entire 3D structure can be intuitively observed, together with different object features or details on or within the object. With the need for imaging advanced volumetric data, demands for the control of virtual object properties are increasing; this happens especially for multicolor objects obtained by fluorescent microscopy. Existing solutions with universal VR controllers or software-based controllers with the need to define sufficient space for the user to manipulate data in VR are not usable in many practical applications. Therefore, we developed a custom gesture-based VR control system with a custom controller connected to the FluoRender visualization environment. A multitouch sensor disk was used for this purpose. Our control system may be a good choice for easier and more comfortable manipulation of virtual objects and their properties, especially using confocal microscopy, which is the most widely used technique for acquiring volumetric fluorescence data so far.