Enhanced imaging colonoscopy facilitates dense motion-based 3D reconstruction

Surface Reconstruction of the Pediatric Larynx via Structure from Motion Photogrammetry: A Pilot Study

Endoscopy is the gold standard for characterizing pediatric airway disorders, however, it is limited for quantitative analysis due to lack of three-dimensional (3D) vision and poor stereotactic depth perception. We utilize structure from motion (SfM) photogrammetry, to reconstruct 3D surfaces of pathologic and healthy pediatric larynges from monocular two-dimensional (2D) endoscopy. Models of pediatric subglottic stenosis were 3D printed and airway endoscopies were simulated. 3D surfaces were successfully reconstructed from endoscopic videos of all models using an SfM analysis toolkit. Average subglottic surface error between SfM reconstructed surfaces and 3D printed models was 0.65 mm as measured by Modified Hausdorff Distance. Average volumetric similarity between SfM surfaces and printed models was 0.82 as measured by Jaccard Index. SfM can be used to accurately reconstruct 3D surface renderings of the larynx from 2D endoscopy video. This technique has immense potential for use in quantitative analysis of airway geometry and virtual surgical planning.

Stomach 3D Reconstruction Using Virtual Chromoendoscopic Images

Gastric endoscopy is a golden standard in the clinical process that enables medical practitioners to diagnose various lesions inside a patient’s stomach. If a lesion is found, a success in identifying the location of the found lesion relative to the global view of the stomach will lead to better decision making for the next clinical treatment. Our previous research showed that the lesion localization could be achieved by reconstructing the whole stomach shape from chromoendoscopic indigo carmine (IC) dye-sprayed images using a structure-from-motion (SfM) pipeline. However, spraying the IC dye to the whole stomach requires additional time, which is not desirable for both patients and practitioners. Our objective is to propose an alternative way to achieve whole stomach 3D reconstruction without the need of the IC dye. We generate virtual IC-sprayed (VIC) images based on image-to-image style translation trained on unpaired real no-IC and IC-sprayed images, where we have investigated the effect of input and output color channel selection for generating the VIC images. We validate our reconstruction results by comparing them with the results using real IC-sprayed images and confirm that the obtained stomach 3D structures are comparable to each other. We also propose a local reconstruction technique to obtain a more detailed surface and texture around an interesting region. The proposed method achieves the whole stomach reconstruction without the need of real IC dye using SfM. We have found that translating no-IC green-channel images to IC-sprayed red-channel images gives the best SfM reconstruction result. Clinical impact We offer a method of the frame localization and local 3D reconstruction of a found gastric lesion using standard endoscopy images, leading to better clinical decision.

Stomach 3D Reconstruction Based on Virtual Chromoendoscopic Image Generation

Gastric endoscopy is a standard clinical process that enables medical practitioners to diagnose various lesions inside a patient's stomach. If any lesion is found, it is very important to perceive the location of the lesion relative to the global view of the stomach. Our previous research showed that this could be addressed by reconstructing the whole stomach shape from chromoendoscopic images using a structure-from-motion (SfM) pipeline, in which indigo carmine (IC) blue dye-sprayed images were used to increase feature matches for SfM by enhancing stomach surface's textures. However, spraying the IC dye to the whole stomach requires additional time, labor, and cost, which is not desirable for patients and practitioners. In this paper, we propose an alternative way to achieve whole stomach 3D reconstruction without the need of the IC dye by generating virtual IC-sprayed (VIC) images based on image-to-image style translation trained on unpaired real no-IC and IC-sprayed images. We have specifically investigated the effect of input and output color channel selection for generating the VIC images and found that translating no-IC green-channel images to IC-sprayed red-channel images gives the best SfM reconstruction result.

3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion

Gastric endoscopy is a common clinical practice that enables medical doctors to diagnose the stomach inside a body. In order to identify a gastric lesion's location such as early gastric cancer within the stomach, this work addressed to reconstruct the 3D shape of a whole stomach with color texture information generated from a standard monocular endoscope video. Previous works have tried to reconstruct the 3D structures of various organs from endoscope images. However, they are mainly focused on a partial surface. In this work, we investigated how to enable structure-from-motion (SfM) to reconstruct the whole shape of a stomach from a standard endoscope video. We specifically investigated the combined effect of chromo-endoscopy and color channel selection on SfM. Our study found that 3D reconstruction of the whole stomach can be achieved by using red channel images captured under chromo-endoscopy by spreading indigo carmine (IC) dye on the stomach surface.

Whole Stomach 3D Reconstruction and Frame Localization From Monocular Endoscope Video

Gastric endoscopy is a common clinical practice that enables medical doctors to diagnose various lesions inside a stomach. In order to identify the location of a gastric lesion such as early cancer and a peptic ulcer within the stomach, this work addresses to reconstruct the color-textured 3D model of a whole stomach from a standard monocular endoscope video and localize any selected video frame to the 3D model. We examine how to enable structure-from-motion (SfM) to reconstruct the whole shape of a stomach from endoscope images, which is a challenging task due to the texture-less nature of the stomach surface. We specifically investigate the combined effect of chromo-endoscopy and color channel selection on SfM to increase the number of feature points. We also design a plane fitting-based algorithm for 3D point outliers removal to improve the 3D model quality. We show that whole stomach 3D reconstruction can be achieved (more than 90% of the frames can be reconstructed) by using red channel images captured under chromo-endoscopy by spreading indigo carmine (IC) dye on the stomach surface. In experimental results, we demonstrate the reconstructed 3D models for seven subjects and the application of lesion localization and reconstruction. The methodology and results presented in this paper could offer some valuable reference to other researchers and also could be an excellent tool for gastric surgeons in various computer-aided diagnosis applications.

HTC Vive MeVisLab integration via OpenVR for medical applications

Virtual Reality, an immersive technology that replicates an environment via computer-simulated reality, gets a lot of attention in the entertainment industry. However, VR has also great potential in other areas, like the medical domain, Examples are intervention planning, training and simulation. This is especially of use in medical operations, where an aesthetic outcome is important, like for facial surgeries. Alas, importing medical data into Virtual Reality devices is not necessarily trivial, in particular, when a direct connection to a proprietary application is desired. Moreover, most researcher do not build their medical applications from scratch, but rather leverage platforms like MeVisLab, MITK, OsiriX or 3D Slicer. These platforms have in common that they use libraries like ITK and VTK, and provide a convenient graphical interface. However, ITK and VTK do not support Virtual Reality directly. In this study, the usage of a Virtual Reality device for medical data under the MeVisLab platform is presented. The OpenVR library is integrated into the MeVisLab platform, allowing a direct and uncomplicated usage of the head mounted display HTC Vive inside the MeVisLab platform. Medical data coming from other MeVisLab modules can directly be connected per drag-and-drop to the Virtual Reality module, rendering the data inside the HTC Vive for immersive virtual reality inspection.

How big is this neoplasia? live colonoscopic size measurement using the Infocus-Breakpoint

Colonoscopy is the reference medical examination for early diagnosis and treatment of colonic diseases. This minimally invasive technique allows endoscopists to explore the colon cavity and remove neoplasias - abnormal growths of tissue - which may develop into malignant tumors. The size, shape and appearance of a neoplasia are essential cues for diagnostic. However, the size is difficult to estimate because the absolute scale of the observed tissue is not directly conveyed in the 2D colonoscopic images. An erroneous size estimate may lead to inappropriate treatment. There currently exist no solutions to reproducible neoplasia size measurement adapted to colonoscopy. We propose a colonoscopic size measurement system for neoplasias. By using a simple planar geometry, the key technical problem is reduced to resolving scale. Our core contribution is introducing the Infocus-Breakpoint (IB) that allows us to resolve scale from a regular colonoscopic video. We define the IB as the lower limit of the colonoscope's depth of field. The IB corresponds to a precise colonoscope to tissue distance, called the reference depth, which we calibrate preoperatively. We detect the IB intraoperatively thanks to two novel modules: deformable Blur-Estimating Tracking (BET) and Blur-Model Fitting (BMF). With our system, the endoscopist may interactively measure the length and area of a neoplasia in a 2D colonoscopic image directly. Our system needs no hardware modification to standard monocular colonoscopes, yet reaching a size measurement accuracy of the order of a millimeter, as shown on several phantom and patient datasets.