Augmented reality based real-time subcutaneous vein imaging system

Vein segmentation and visualization of upper and lower extremities using convolution neural network

OBJECTIVES

The study focused on developing a reliable real-time venous localization, identification, and visualization framework based upon deep learning (DL) self-parametrized Convolution Neural Network (CNN) algorithm for segmentation of the venous map for both lower and upper limb dataset acquired under unconstrained conditions using near-infrared (NIR) imaging setup, specifically to assist vascular surgeons during venipuncture, vascular surgeries, or Chronic Venous Disease (CVD) treatments.

METHODS

A portable image acquisition setup has been designed to collect venous data (upper and lower extremities) from 72 subjects. A manually annotated image dataset was used to train and compare the performance of existing well-known CNN-based architectures such as ResNet and VGGNet with self-parameterized U-Net, improving automated vein segmentation and visualization.

RESULTS

Experimental results indicated that self-parameterized U-Net performs better at segmenting the unconstrained dataset in comparison with conventional CNN feature-based learning models, with a Dice score of 0.58 and displaying 96.7 % accuracy for real-time vein visualization, making it appropriate to locate veins in real-time under unconstrained conditions.

CONCLUSIONS

Self-parameterized U-Net for vein segmentation and visualization has the potential to reduce risks associated with traditional venipuncture or CVD treatments by outperforming conventional CNN architectures, providing vascular assistance, and improving patient care and treatment outcomes.

Spatial sensitivity distribution assessment and Monte Carlo simulations for needle-based bioimpedance imaging during venipuncture using the finite element method

Despite being among the most common medical procedures, needle insertions suffer from a high error rate. Impedance measurements using electrode-equipped needles offer promise for improved tissue targeting and reduced errors. Impedance visualization usually requires an extensive pre-measured impedance dataset for tissue differentiation and knowledge of the electric fields contributing to the resulting impedances. This work presents two finite element simulation approaches for both problems. The first approach describes the generation of a multitude of impedances with Monte Carlo simulations for both, homogeneous and inhomogeneous tissue to circumvent the need to rely on previously measured data. These datasets could be used for tissue discrimination. The second method describes the simulation of the spatial sensitivity distribution of an electrode layout. Two singularity analysis methods were employed to determine the bulk of the sensitivity within a finite volume, which in turn enables consistent 3D visualization. The modeled electrode layout consists of 12 electrodes radially placed around a hypodermic needle. Electrical excitation was simulated using two neighboring electrodes for current carriage and voltage pickup, which resulted in 12 distinct bipolar excitation states. Both, the impedance simulations and the respective singularity analysis methods were compared with each other. The results show that the statistical spread of impedances is highly dependent on the tissue type and its inhomogeneities. The bounded bulk of sensitivities of both methods are of similar extent and symmetry. Future models should incorporate more detailed tissue properties such as anisotropy or changing material properties due to tissue deformation to gain more accurate predictions.

Exploring the potential role for extended reality in Mohs micrographic surgery

Mohs micrographic surgery (MMS) is a cornerstone of dermatological practice. Virtual reality (VR) and augmented reality (AR) technology, initially used for entertainment, have entered healthcare, offering real-time data overlaying a surgeon's view. This paper explores potential applications of VR and AR in MMS, emphasizing their advantages and limitations. We aim to identify research gaps to facilitate innovation in dermatological surgery. We conducted a PubMed search using the following: "augmented reality" OR "virtual reality" AND "Mohs" or "augmented reality" OR "virtual reality" AND "surgery." Inclusion criteria were peer-reviewed articles in English discussing these technologies in medical settings. We excluded non-peer-reviewed sources, non-English articles, and those not addressing these technologies in a medical context. VR alleviates patient anxiety and enhances patient satisfaction while serving as an educational tool. It also aids physicians by providing realistic surgical simulations. On the other hand, AR assists in real-time lesion analysis, optimizing incision planning, and refining margin control during surgery. Both of these technologies offer remote guidance for trainee residents, enabling real-time learning and oversight and facilitating synchronous teleconsultations. These technologies may transform dermatologic surgery, making it more accessible and efficient. However, further research is needed to validate their effectiveness, address potential challenges, and optimize seamless integration. All in all, AR and VR enhance real-world environments with digital data, offering real-time surgical guidance and medical insights. By exploring the potential integration of these technologies in MMS, our study identifies avenues for further research to thoroughly understand the role of these technologies to redefine dermatologic surgery, elevating precision, surgical outcomes, and patient experiences.

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future.

Quantitative phase analysis and hyperspectral imaging for the automatic identification of veins and blood perfusion maps

INTRODUCTION

Medical workers commonly physically identify subcutaneous veins to locate a suitable vesselto implant a catheter for drug administration or blood sample. The general rule of thumb is to locate a big, clean vein that will allow the medication to readily pass within the intended blood vessel. Peripheral problematic venous access happens when a patient's veins are difficult to palpate because of factors like dark skin tone, edema or excess tissue. The ability to see how the vasculature changes to support the therapeutic methods without damaging the surrounding tissue is another challenge.

MATERIALS AND METHODS

Hyperspectral imaging (HI) is a developing technique with several potential uses in medicine. Using its spectroscopic data, veins and arterioles could be noninvasively detected and discriminated. It is frequently important to use quantitative phase analysis for vein localization. To assess hyperspectral image data for the detection of both veins and peripheral arteries, we suggest using an advanced image processing and classification algorithm based on the phase information related to the index of refraction change and associated scattering. We show that this need may be satisfied using quantitative phase imaging of forearm skin tissue at different depths.

RESULTS

To demonstrate the variations in the diffuse reflectance characteristics between skin surface and veins, phase resolved pictures were successfully produced for twelve volunteers using our imaging methodology. We found that the skin surface details are completely apparent at the unique wavelength of 441 nm. The 500-nm wavelength was the most efficient for grouping peripheral arteries and illuminating the blood perfusion maps. Using our HI experimental setup and our phase imaging procedure on the 600 nm and 650 nm visible spectral pictures, we were able to properly describe the vein map. This spectral area may be utilized as a vein locator marker which could approximately reach till 3 mm depth under skin surface.

CONCLUSIONS

Initial findings suggested that our imaging technique would be able to assist medical examiners in safely assessing the veins and arteriole's locations automatically without exposing the skin to infrared radiation. Meanwhile, our pilot research in this work to determine the best spectral wavelengths for localizing veins and mapping blood perfusion using our phase analysis imaging strategy with the HI camera. By substituting the HI camera with a custom conventional RGB camera that only functions at specific wavelengths during the discovering of blood perfusion locations or prior to intravenous catheterization, a distinctive and efficient system for precise identification may be developed to serve in the field of the vascular therapeutic methods.

Design and Implementation of Embedded-Based Vein Image Processing System with Enhanced Denoising Capabilities

In general, it is very difficult to visually locate blood vessels for intravenous injection or surgery. In addition, if vein detection fails, physical and mental pain occurs to the patient and leads to financial loss in the hospital. In order to prevent this problem, NIR-based vein detection technology is developing. The proposed study combines vein detection and digital hair removal to eliminate body hair, a noise that hinders the accuracy of detection, improving the performance of the entire algorithm by about 10.38% over existing systems. In addition, as a result of performing venous detection of patients without body hair, 5.04% higher performance than the existing system was detected, and the proposed study results were verified. It is expected that the use of devices to which the proposed study is applied will provide more accurate vascular maps in general situations.

Feasibility of 830 nm laser imaging for vein localization in dark skin tissue-mimicking phantoms

Accessing blood vessels by medical professionals has been a challenge in healthcare centers worldwide. The main objective of this work is to investigate the localization of blood vessels in dark skin based on near infrared laser imaging. An 830 nm diode laser was used as a light source to irradiate dark skin mimicking optical phantoms. Phantoms were constructed to simulate dark skin with embedded polymer tubes filled with human blood to mimic subcutaneous veins. Appropriate image processing techniques were also used to enhance the detection and depth resolved differentiation of the vein phantoms. Results show that a linear regression model can represent the relation between the grey level in subcutaneous vein images and the depth of vessels down to 3 mm or deeper (n = 15, R² = 0.88, P < 0.001). The effect of laser power on the system performance is also discussed. Analysis of the collected images demonstrates the feasibility of 830 nm laser imaging for differentiating vein depths under dark skin surface. The proposed method would enhance the localization of invisible subcutaneous veins. This, in turn, would further improve the success rate of related medical procedures such as blood sampling, drawing, in the dark skin population.

Design of interactive augmented reality functions for robotic surgery and evaluation in dry-lab lymphadenectomy

BACKGROUND

Augmented reality (AR) has been widely researched for use in healthcare. Prior AR for robot-assisted minimally invasive surgery has mainly focussed on superimposing preoperative three-dimensional (3D) images onto patient anatomy. This article presents alternative interactive AR tools for robotic surgery.

METHODS

We designed, built and evaluated four voice-controlled functions: viewing a live video of the operating room, viewing two-dimensional preoperative images, measuring 3D distances and warning about out-of-view instruments. This low-cost system was developed on a da Vinci Si, and it can be integrated into surgical robots equipped with a stereo camera and a stereo viewer.

RESULTS

Eight experienced surgeons performed dry-lab lymphadenectomies and reported that the functions improved the procedure. They particularly appreciated the possibility of accessing the patient's medical records on demand, measuring distances intraoperatively and interacting with the functions using voice commands.

CONCLUSIONS

The positive evaluations garnered by these alternative AR functions and interaction methods provide support for further exploration.

Dorsal Hand Vein Image Enhancement Using Fusion of CLAHE and Fuzzy Adaptive Gamma

Enhancement of captured hand vein images is essential for a number of purposes, such as accurate biometric identification and ease of medical intravenous access. This paper presents an improved hand vein image enhancement technique based on weighted average fusion of contrast limited adaptive histogram equalization (CLAHE) and fuzzy adaptive gamma (FAG). The proposed technique is applied using three stages. Firstly, grey level intensities with CLAHE are locally applied to image pixels for contrast enhancement. Secondly, the grey level intensities are then globally transformed into membership planes and modified with FAG operator for the same purposes. Finally, the resultant images from CLAHE and FAG are fused using improved weighted averaging methods for clearer vein patterns. Then, matched filter with first-order derivative Gaussian (MF-FODG) is employed to segment vein patterns. The proposed technique was tested on self-acquired dorsal hand vein images as well as images from the SUAS databases. The performance of the proposed technique is compared with various other image enhancement techniques based on mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index measurement (SSIM). The proposed enhancement technique’s impact on the segmentation process has also been evaluated using sensitivity, accuracy, and dice coefficient. The experimental results show that the proposed enhancement technique can significantly enhance the hand vein patterns and improve the detection of dorsal hand veins.

Local-global active contour model based on tensor-based representation for 3D ultrasound vessel segmentation

Three-dimensional (3D) vessel segmentation can provide full spatial information about an anatomic structure to help physicians gain increased understanding of vascular structures, which plays an utmost role in many medical image-processing and analysis applications. The purpose of this paper aims to develop a 3D vessel-segmentation method that can improve segmentation accuracy in 3D ultrasound (US) images. We propose a 3D tensor-based active contour model method for accurate 3D vessel segmentation. With our method, the contrast-independent multiscale bottom-hat tensor representation and local-global information are captured. This strategy ensures the effective extraction of the boundaries of vessels from inhomogeneous and homogeneous regions without being affected by the noise and low-contrast of the 3D US images. Experimental results in clinical 3D US and public 3D Multiphoton Microscopy datasets are used for quantitative and qualitative comparison with several state-of-the-art vessel segmentation methods. Clinical experiments demonstrate that our method can achieve a smoother and more accurate boundary of the vessel object than competing methods. The mean SE, SP and ACC of the proposed method are: 0.7768 ± 0.0597, 0.9978 ± 0.0013 and 0.9971 ± 0.0015 respectively. Experiments on the public dataset show that our method can segment complex vessels in different medical images with noise and low- contrast.

Augmented reality in anesthesia, pain medicine and critical care: a narrative review

Augmented reality (AR) is the integration of computer-generated information with the user's environment in real time. AR is used in many industries, including healthcare, where it has gained significant popularity. Recent strides in hardware and software engineering have reduced the cost of AR, while significantly improving the experience for users and developers. One of the first applications of AR technology in perioperative medicine has been in the identification of anatomical structures for regional blocks and peripheral or central vascular access. AR has also been implemented in pediatric care to reduce periprocedural anxiety. In this narrative review, we summarize the current role of AR in anesthesiology, pain medicine, and critical care.

FPGA-Based Two-Dimensional Matched Filter Design for Vein Imaging Systems

Venipuncture is a common medical procedure. The use of augmented reality-based assistive devices can improve the first puncture success rate in patients with poor vascular filling. In order to improve the image rendering quality and speed of auxiliary equipment, this study develop a two-dimensional matched filtering algorithm on a Field Programmable Gate Array (FPGA) in a near-infrared vein imaging system, which use parallel processing to offer real-time response and is designed as a small handheld portable device. A customized dorsal hand vein image library with 200 images captured from 120 participants is used to analyze the effects of convolution kernel parameters and exposure time on vascular imaging with different depths, and the correlation model between these parameters and vascular depth are constructed. We use the Tenengrad, variance, Laplace smoothness and standard deviation as evaluation indicators, and compare our algorithm with three other related studies. Experimental results show that the rendering quality of our proposed algorithm is significantly higher than other algorithms. In addition, the rendering speed of our algorithm can reach 66 fps, which is twice faster than the current fastest algorithm.

A recent review and a taxonomy for hard and soft tissue visualization-based mixed reality

BACKGROUND

Mixed reality (MR) visualization is gaining popularity in image-guided surgery (IGS) systems, especially for hard and soft tissue surgeries. However, a few MR systems are implemented in real time. Some factors are limiting MR technology and creating a difficulty in setting up and evaluating the MR system in real environments. Some of these factors include: the end users are not considered, the limitations in the operating room, and the medical images are not fully unified into the operating interventions.

METHODOLOGY

The purpose of this article is to use Data, Visualization processing, and View (DVV) taxonomy to evaluate the current MR systems. DVV includes all the components required to be considered and validated for the MR used in hard and soft tissue surgeries. This taxonomy helps the developers and end users like researchers and surgeons to enhance MR system for the surgical field.

RESULTS

We evaluated, validated, and verified the taxonomy based on system comparison, completeness, and acceptance criteria. Around 24 state-of-the-art solutions that are picked relate to MR visualization, which is then used to demonstrate and validate this taxonomy. The results showed that most of the findings are evaluated and others are validated.

CONCLUSION

The DVV taxonomy acts as a great resource for MR visualization in IGS. State-of-the-art solutions are classified, evaluated, validated, and verified to elaborate the process of MR visualization during surgery. The DVV taxonomy provides the benefits to the end users and future improvements in MR.

Intraoperative Imaging and Image Fusion for Venous Interventions

Advanced imaging for intraoperative evaluation of venous pathologies has played an increasingly significant role in this era of evolving minimally invasive surgical and interventional therapies. The evolution of dedicated venous stents and other novel interventional devices has mandated the need for advanced imaging tools to optimize safe and accurate device deployment. Most venous interventions are typically performed using a combination of standard 2-dimensional (2D) fluoroscopy, digital-subtraction angiography, and intravascular ultrasound imaging techniques. Latest generation computer tomography (CT) and magnetic resonance imaging (MRI) scanners have been shown to provide high-resolution 3D and 4D information about venous vasculature. In addition to morphological imaging, novel MRI techniques such as 3D time-resolved MR venography and 4D flow sequences can provide quantitative information and help visualize intricate flow patterns to better understand complex venous pathologies. Moreover, the high-fidelity information from multiple imaging techniques can be integrated using image fusion to overcome the limitations of current intraoperative imaging techniques. For example, the limitations of standard 2D fluoroscopy and luminal angiography can be compensated for by perivascular and soft-tissue information from MRI during complex venous interventions using image fusion techniques. Intraoperative dynamic evaluation of devices such as venous stents and real-time understanding of changes in flow patterns during venous interventions may be routinely available in future interventional suites with integrated multimodality CT or MR imaging capabilities. The purpose of this review is to discuss the outlook for intraoperative imaging and multimodality image fusion techniques and highlight their value during complex venous interventions.

Real-time dual-modal vein imaging system

PURPOSE

In this paper, we present a vein imaging system to combine reflectance mode visible spectrum images (VIS) with transmission mode near-infrared (NIR) images in real time. Clear vessel localization is achieved in this manner with combined NIR-VIS dual-modal imaging.

METHODS

Transmission and reflectance mode optical instrumentation is used to combine VIS and NIR images. Two methods of displaying the combined images are demonstrated here. We have conducted experiments to determine the system's resolution, alignment accuracy, and depth penetration. Vein counts were taken from the hands of test subjects using the system and compared with vein counts taken by visual analysis.

RESULTS

Results indicate that the system can improve vein detection in the human hand while detecting veins of a diameter < 0.5 mm at any working distance and of a 0.25 mm diameter at an optimal working distance of about 30 cm. The system has also been demonstrated to clearly detect silicone vessels with artificial blood of diameter 2, 1, and 0.5 mm diameter under a tissue depth of 3 mm. In a study involving 25 human subjects, we have demonstrated that vein visibility was significantly increased using our system.

CONCLUSIONS

The results indicate that the device is a high-resolution solution to near-surface venous imaging. This technology can be applied for IV placement, morphological analysis for disease state detection, and biometric analysis.

Hybrid constraint optimization for 3D subcutaneous vein reconstruction by near-infrared images

BACKGROUND AND OBJECTIVE

The development of biometric identification technology and intelligent medication has enabled researchers to analyze subcutaneous veins from near-infrared images. However, the stereo reconstruction of subcutaneous veins has not been well studied, and the results are difficult to utilize in clinical practice.

METHODS

We present a hybrid constraint optimization (HCO) matching algorithm for vein reconstruction to solve the matching failure problems caused by the incomplete segmentation of vein structures captured from different views. This algorithm initially introduces the existence of the epipolar and homography constraints in the subcutaneous vein matching. Then, the HCO matching algorithm of the vascular centerline is established by homography point-to-point matching, homography matrix optimization, and vascular section matching. Finally, the 3D subcutaneous vein is reconstructed on the basis of the principle of triangulation and system calibration parameters.

RESULTS

To validate the performance of the proposed matching method, we designed a series of experiments to evaluate the effectiveness of the hybrid constraint optimization method. The experiments were performed on simulated and real datasets. 42 real vascular images were analyzed on the basis of different matching strategies. Experimental result shows that the matching accuracy increased significantly with the proposed optimization matching method. To calculate the reconstruction accuracy, we reconstructed seven simulated cardboards and measured 10 vascular distances in each simulated cardboard. The average vascular distance error of each simulated image was within 1.0 mm, and the distance errors of 75% feature points were less than 1.5 mm. Also, we printed a 3D simulated vein model to improve the illustration of this system. The reconstruction error extends from -3.58 mm to 1.94 mm with a standard deviation of 0.68 mm and a mean of 0.07 mm.

CONCLUSIONS

The algorithm is validated in terms of homography optimization, matching efficiency, and simulated vascular reconstruction error. The experimental results demonstrate that the veins captured from the left and right views can be accurately matched through the proposed algorithm.

Subcutaneous veins depth measurement using diffuse reflectance images

Intravenous (IV) procedures are often difficult due to the poor visualization of subcutaneous veins. Because existing vein locators lack the ability to assess depth, and also because mis-punctures and poor vascular access remain problematic, we propose an imaging system that employs diffuse reflectance images at three isosbestic wavelengths to measure both the depth and thickness of subcutaneous veins. This paper describes the proposed system as well as proof-of-principle experimental demonstrations. We initially introduce the working principle and structure of the system. All measurements were based on the Monte Carlo (MC) method and accomplished by referring an optical density (OD) ratio to a multi-layer diffuse reflectance model. Results were all validated by comparative ultrasound measurements. Experimental trials included 11 volunteers who were subjected to both ultrasound measurements and the proposed optical process to validate the system's applicability. However, the unreliability of the "thickness" measurement of the vein may be due to the fact that the veins have collapsible walls - so excess pressure by the transducer will give a false thickness.

Multisensory System for the Detection and Localization of Peripheral Subcutaneous Veins

This paper proposes a multisensory system for the detection and localization of peripheral subcutaneous veins, as a first step for achieving automatic robotic insertion of catheters in the near future. The multisensory system is based on the combination of a SWIR (Short-Wave Infrared) camera, a TOF (Time-Of-Flight) camera and a NIR (Near Infrared) lighting source. The associated algorithm consists of two main parts: one devoted to the features extraction from the SWIR image, and another envisaged for the registration of the range data provided by the TOF camera, with the SWIR image and the results of the peripheral veins detection. In this way, the detected subcutaneous veins are mapped onto the 3D reconstructed surface, providing a full representation of the region of interest for the automatic catheter insertion. Several experimental tests were carried out in order to evaluate the capabilities of the presented approach. Preliminary results demonstrate the feasibility of the proposed design and highlight the potential benefits of the solution.

3D reconstruction of cystoscopy videos for comprehensive bladder records

White light endoscopy is widely used for diagnostic imaging of the interior of organs and body cavities, but the inability to correlate individual 2D images with 3D organ morphology limits its utility for quantitative or longitudinal studies of disease physiology or cancer surveillance. As a result, most endoscopy videos, which carry enormous data potential, are used only for real-time guidance and are discarded after collection. We present a computational method to reconstruct and visualize a 3D model of organs from an endoscopic video that captures the shape and surface appearance of the organ. A key aspect of our strategy is the use of advanced computer vision techniques and unmodified, clinical-grade endoscopy hardware with few constraints on the image acquisition protocol, which presents a low barrier to clinical translation. We validate the accuracy and robustness of our reconstruction and co-registration method using cystoscopy videos from tissue-mimicking bladder phantoms and show clinical utility during cystoscopy in the operating room for bladder cancer evaluation. As our method can powerfully augment the visual medical record of the appearance of internal organs, it is broadly applicable to endoscopy and represents a significant advance in cancer surveillance opportunities for big-data cancer research.