Hybrid multiscale affine and elastic image registration approach towards wireless capsule endoscope localization

Reducing reading time and assessing disease in capsule endoscopy videos: A deep learning approach

BACKGROUND

The wireless capsule endoscope (CE) is a valuable diagnostic tool in gastroenterology, offering a safe and minimally invasive visualization of the gastrointestinal tract. One of the few drawbacks identified by the gastroenterology community is the time-consuming task of analyzing CE videos.

OBJECTIVES

This article investigates the feasibility of a computer-aided diagnostic method to speed up CE video analysis. We aim to generate a significantly smaller CE video with all the anomalies (i.e., diseases) identified by the medical doctors in the original video.

METHODS

The summarized video consists of the original video frames classified as anomalous by a pre-trained convolutional neural network (CNN). We evaluate our approach on a testing dataset with eight CE videos captured with five CE types and displaying multiple anomalies.

RESULTS

On average, the summarized videos contain 93.33% of the anomalies identified in the original videos. The average playback time of the summarized videos is just 10 min, compared to 58 min for the original videos.

CONCLUSION

Our findings demonstrate the potential of deep learning-aided diagnostic methods to accelerate CE video analysis.

Calibrated analytical model for magnetic localization of wireless capsule endoscope based on onboard sensing

Wireless capsule endoscopes (WCEs) are pill-sized camera-embedded devices that can provide visualization of the gastrointestinal (GI) tract by capturing and transmitting images to an external receiver. Determination of the exact location of the WCE is crucial for the accurate navigation of the WCE through external guidance, tracking of the GI abnormality, and the treatment of the detected disease. Despite the enormous progress in the real-time tracking of the WCE, a well-calibrated analytical model is still missing for the accurate localization of WCEs by the measurements from different onboard sensing units. In this paper, a well-calibrated analytical model for the magnetic localization of the WCE was established by optimizing the magnetic moment in the magnetic dipole model. The Jacobian-based iterative method was employed to solve the position of the WCE. An error model was established and experimentally verified for the analysis and prediction of the localization errors caused by inaccurate measurements from the magnetic field sensor. The assessment of the real-time localization of the WCE was performed via experimental trials using an external permanent magnet (EPM) mounted on a robotic manipulator and a WCE equipped with a 3-axis magnetic field sensor and an inertial measurement unit (IMU). The localization errors were measured under different translational and rotational motion modes and working spaces. The results showed that the selection of workspace (distance relative to the EPM) could lead to different positioning errors. The proposed magnetic localization method holds great potential for the real-time localization of WCEs when performing complex motions during GI diagnosis.

Tracking the Traveled Distance of Capsule Endoscopes along a Gastrointestinal-Tract Model Using Differential Static Magnetic Localization

The traveled distance and orientation of capsule endoscopes for each video frame are not available in commercial systems, but they would be highly relevant for physicians. Furthermore, scientific approaches lack precisely tracking the capsules along curved trajectories within the typical gastrointestinal tract. Recently, we showed that the differential static magnetic localisation method is suitable for the precise absolute localisation of permanent magnets assumed to be integrated into capsule endoscopes. Thus, in the present study, the differential method was employed to track permanent magnets in terms of traveled distance and orientation along a length trajectory of 487.5 mm, representing a model of the winding gastrointestinal tract. Permanent magnets with a diameter of 10 mm and different lengths were used to find a lower boundary for magnet size. Results reveal that the mean relative distance and orientation errors did not exceed 4.3 ± 3.3%, and 2 ± 0.6∘, respectively, when the magnet length was at least 5 mm. Thus, a 5 mm long magnet would be a good compromise between achievable tracking accuracy and magnet volume, which are essential for integration into small commercial capsules. Overall, the proposed tracking accuracy was better than that of the state of the art within a region covering the typical gastrointestinal-tract size.

Joint estimation of depth and motion from a monocular endoscopy image sequence using a multi-loss rebalancing network

Building an in vivo three-dimensional (3D) surface model from a monocular endoscopy is an effective technology to improve the intuitiveness and precision of clinical laparoscopic surgery. This paper proposes a multi-loss rebalancing-based method for joint estimation of depth and motion from a monocular endoscopy image sequence. The feature descriptors are used to provide monitoring signals for the depth estimation network and motion estimation network. The epipolar constraints of the sequence frame is considered in the neighborhood spatial information by depth estimation network to enhance the accuracy of depth estimation. The reprojection information of depth estimation is used to reconstruct the camera motion by motion estimation network with a multi-view relative pose fusion mechanism. The relative response loss, feature consistency loss, and epipolar consistency loss function are defined to improve the robustness and accuracy of the proposed unsupervised learning-based method. Evaluations are implemented on public datasets. The error of motion estimation in three scenes decreased by 42.1%,53.6%, and 50.2%, respectively. And the average error of 3D reconstruction is 6.456 ± 1.798mm. This demonstrates its capability to generate reliable depth estimation and trajectory reconstruction results for endoscopy images and meaningful applications in clinical.

Deep learning for registration of region of interest in consecutive wireless capsule endoscopy frames

BACKGROUND AND OBJECTIVE

Functional gastrointestinal disorders (FGIDs) are reported as worldwide gastrointestinal (GI) diseases. GI motility assessment can assist the diagnosis of patients with intestine motility dysfunction. Wireless capsule endoscopy (WCE) can acquire images in the gastrointestinal (GI) tract including the small intestine where other conventional endoscopes cannot penetrate, and WCE images can reveal GI motility. To generally analyze WCE frames, the high-precision registration of consecutive WCE frames is an absolute necessity. It is difficult and meaningless to register entire WCE frames on a pixel level due to the unpredictable and massive non-rigid deformation between consecutive frames, the low quality of imaging and the complex intestinal environment. Thus, the registration of region of interest (ROI) functioning in a feature level has more significance than entire frame registration.

METHODS

In this paper we present Timecylce-WCE, an end-to-end automatic registration approach of ROIs on WCE images. The clinicians can determine a ROI by drawing a bounding box in any WCE frame to be registered. This proposed approach is based on a deep-learning model of time-consistency in recurrent-registering, skip-registering and self-registering cycle, and it is fully unsupervised without any label. We incorporate the global correlation map with the local correlation map in matching the features, and a novel overall loss function is designed to enable the convergence of the model. As the output, a thin-plate spline (TPS) transformed region in the template frame is highly aligned with the query ROI in a finer-grained level. To the best of our knowledge this is the first time that a deep-learning-based registration method is proposed for WCE imaging motion.

RESULTS

To highlight the effectiveness of the proposed approach, our proposed method is compared with the existing non-deep-learning methods and tested in a validation dataset with labeled matching points. The presented method resulted in the best PCK@10 (Percentage of Correct Key-points, i.e., the predicted and the true joint is within the threshold - 10 pixels) of 66.49%. We also demonstrate that variants of design improved registration accuracy.

CONCLUSIONS

From the experimental analysis, it is clear that our proposed method outperforms the other existing methods. This lays the groundwork for subsequent studies, such as GI motility assessment, and WCE image synthesis.

DINOSARC: Color Features Based on Selective Aggregation of Chromatic Image Components for Wireless Capsule Endoscopy

Wireless Capsule Endoscopy (WCE) is a noninvasive diagnostic technique enabling the inspection of the whole gastrointestinal (GI) tract by capturing and wirelessly transmitting thousands of color images. Proprietary software "stitches" the images into videos for examination by accredited readers. However, the videos produced are of large length and consequently the reading task becomes harder and more prone to human errors. Automating the WCE reading process could contribute in both the reduction of the examination time and the improvement of its diagnostic accuracy. In this paper, we present a novel feature extraction methodology for automated WCE image analysis. It aims at discriminating various kinds of abnormalities from the normal contents of WCE images, in a machine learning-based classification framework. The extraction of the proposed features involves an unsupervised color-based saliency detection scheme which, unlike current approaches, combines both point and region-level saliency information and the estimation of local and global image color descriptors. The salient point detection process involves estimation of DIstaNces On Selective Aggregation of chRomatic image Components (DINOSARC). The descriptors are extracted from superpixels by coevaluating both point and region-level information. The main conclusions of the experiments performed on a publicly available dataset of WCE images are (a) the proposed salient point detection scheme results in significantly less and more relevant salient points; (b) the proposed descriptors are more discriminative than relevant state-of-the-art descriptors, promising a wider adoption of the proposed approach for computer-aided diagnosis in WCE.

A Motion Artifact Reduction Method in Cerebrovascular DSA Sequence Images

Digital Subtraction Angiography (DSA) can be used for diagnosing the pathologies of vascular system including systemic vascular disease, coronary heart disease, arrhythmia, valvular disease and congenital heart disease. Previous studies have provided some image enhancement algorithms for DSA images. However, these studies are not suitable for automated processes in huge amounts of data. Furthermore, few algorithms solved the problems of image contrast corruption after artifact removal. In this paper, we propose a fully automatic method for cerebrovascular DSA sequence images artifact removal based on rigid registration and guided filter. The guided filtering method is applied to fuse the original DSA image and registered DSA image, the results of which preserve clear vessel boundary from the original DSA image and remove the artifacts by the registered procedure. The experimental evaluation with 40 DSA sequence images shows that the proposed method increases the contrast index by 24.1% for improving the quality of DSA images compared with other image enhancement methods, and can be implemented as a fully automatic procedure.

Deep Endoscopic Visual Measurements

Robotic endoscopic systems offer a minimally invasive approach to the examination of internal body structures, and their application is rapidly extending to cover the increasing needs for accurate therapeutic interventions. In this context, it is essential for such systems to be able to perform measurements, such as measuring the distance traveled by a wireless capsule endoscope, so as to determine the location of a lesion in the gastrointestinal tract, or to measure the size of lesions for diagnostic purposes. In this paper, we investigate the feasibility of performing contactless measurements using a computer vision approach based on neural networks. The proposed system integrates a deep convolutional image registration approach and a multilayer feed-forward neural network into a novel architecture. The main advantage of this system, with respect to the state-of-the-art ones, is that it is more generic in the sense that it is 1) unconstrained by specific models, 2) more robust to nonrigid deformations, and 3) adaptable to most of the endoscopic systems and environment, while enabling measurements of enhanced accuracy. The performance of this system is evaluated under ex vivo conditions using a phantom experimental model and a robotically assisted test bench. The results obtained promise a wider applicability and impact in endoscopy in the era of big data.