Automatic and robust segmentation of endoscopic OCT images and optical staining

The LMIT: Light-mediated minimally-invasive theranostics in oncology

Minimally-invasive diagnosis and therapy have gradually become the trend and research hotspot of current medical applications. The integration of intraoperative diagnosis and treatment is a development important direction for real-time detection, minimally-invasive diagnosis and therapy to reduce mortality and improve the quality of life of patients, so called minimally-invasive theranostics (MIT). Light is an important theranostic tool for the treatment of cancerous tissues. Light-mediated minimally-invasive theranostics (LMIT) is a novel evolutionary technology that integrates diagnosis and therapeutics for the less invasive treatment of diseased tissues. Intelligent theranostics would promote precision surgery based on the optical characterization of cancerous tissues. Furthermore, MIT also requires the assistance of smart medical devices or robots. And, optical multimodality lay a solid foundation for intelligent MIT. In this review, we summarize the important state-of-the-arts of optical MIT or LMIT in oncology. Multimodal optical image-guided intelligent treatment is another focus. Intraoperative imaging and real-time analysis-guided optical treatment are also systemically discussed. Finally, the potential challenges and future perspectives of intelligent optical MIT are discussed.

Wavelet attention network for the segmentation of layer structures on OCT images

Automatic segmentation of layered tissue is critical for optical coherence tomography (OCT) image analysis. The development of deep learning techniques provides various solutions to this problem, while most existing methods suffer from topological errors such as outlier prediction and label disconnection. The channel attention mechanism is a powerful technique to address these problems due to its simplicity and robustness. However, it relies on global average pooling (GAP), which only calculates the lowest frequency component and leaves other potentially useful information unexplored. In this study, we use the discrete wavelet transform (DWT) to extract multi-spectral information and propose the wavelet attention network (WATNet) for tissue layer segmentation. The DWT-based attention mechanism enables multi-spectral analysis with no complex frequency-selection process and can be easily embedded to existing frameworks. Furthermore, the various wavelet bases make the WATNet adaptable to different tasks. Experiments on a self-collected esophageal dataset and two public retinal OCT dataset demonstrated that the WATNet achieved better performance compared to several widely used deep networks, confirming the advantages of the proposed method.

Automatic segmentation of hyperreflective dots via focal priors and visual saliency

PURPOSE

Hyperreflective dots (HRDs) can be observed in spectral domain optical coherence tomography (SD-OCT), which can provide a sensitive marker in the treatment decision process. Quantitative analyses of HRDs are the key to make appropriate decisions on observation, treatment, and retreatment. The purpose of this study is to automatically and accurately segment HRDs in SD-OCT B-scans with diabetic retinopathy (DR).

METHODS

The authors propose an automatic segmentation algorithm of HRDs via focal priors and visual saliency. The algorithm is divided into three stages: segmentation of retinal layers, calculation of the multiscale local contrast saliency map, and adaptive threshold segmentation. First, a method based on improved graph search is used to segment retinal layers to obtain the region of interest (ROI) and the reflectivity estimation of the retinal pigment epithelium (RPE) layer; then, the multiscale local contrast saliency map is obtained by using a local contrast measure, which measures the dissimilarity between the current pixels and corresponding neighborhoods; finally, an adaptive threshold is applied to segment HRDs.

RESULTS

Experimental results on 20 SD-OCT B-scans demonstrate that our method is effective for HRDs segmentation. The average dice similarity coefficient (DSC) and detection accuracy are 71.12% and 85.07%, respectively.

CONCLUSIONS

The proposed method can accurately segment HRDs in SD-OCT B-scans with DR and outperforms current state-of-the-art methods. Our method can provide reliable HRDs segmentation to assist ophthalmologists in clinical diagnosis, treatment, disease monitoring, and progression.

Direct Visualization and Quantitative Imaging of Small Airway Anatomy In Vivo Using Deep Learning Assisted Diffractive OCT

OBJECTIVE/BACKGROUND

In vivo imaging and quantification of the microstructures of small airways in three dimensions (3D) allows a better understanding and management of airway diseases, such as asthma and chronic obstructive pulmonary disease (COPD). At present, the resolution and contrast of the currently available conventional optical coherence tomography (OCT) imaging technologies operating at 1300 nm remain challenging to directly visualize the fine microstructures of small airways in vivo.

METHODS

We developed an ultrahigh-resolution diffractive endoscopic OCT at 800 nm to afford a resolving power of 1.7 µm (in tissue) with an improved contrast and a custom deep residual learning based image segmentation framework to perform accurate and automated 3D quantification of airway anatomy.

RESULTS

The 800-nm diffractive OCT enabled the direct delineation of the structural components in the small airway wall in vivo. We further first demonstrated the 3D anatomic quantification of critical tissue compartments of small airways in sheep using the automated segmentation method.

CONCLUSION

The deep learning assisted diffractive OCT provides a unique ability to access the small airways, directly visualize and quantify the important tissue compartments, such as airway smooth muscle, in the airway wall in vivo in 3D.

SIGNIFICANCE

These pilot results suggest a potential technology for calculating volumetric measurements of small airways in patients in vivo.

Automatic lumen segmentation using uniqueness of vascular connected region for intravascular optical coherence tomography

We present an automatic lumen segmentation method using uniqueness of connected region for intravascular optical coherence tomography (IVOCT), which can effectively remove the effect on lumen segmentation caused by blood artifacts. Utilizing the uniqueness of vascular wall on A-lines, we detect the A-lines shared by multiple connected regions, identify connected regions generated by blood artifacts using traversal comparison of connected regions' location, shared ratio and area ratio and then remove all artifacts. We compare these three methods by 216 challenging images with severe blood artifacts selected from clinical 1076 IVOCT images. The metrics of the proposed method are evaluated including Dice index, Jaccard index and accuracy of 94.57%, 90.12%, 98.02%. Compared with automatic lumen segmentation based on the previous morphological feature method and widely used dynamic programming method, the metrics of the proposed method are significantly enhanced, especially in challenging images with severe blood artifacts.

Connectivity-based deep learning approach for segmentation of the epithelium in in vivo human esophageal OCT images

Optical coherence tomography (OCT) is used for diagnosis of esophageal diseases such as Barrett's esophagus. Given the large volume of OCT data acquired, automated analysis is needed. Here we propose a bilateral connectivity-based neural network for in vivo human esophageal OCT layer segmentation. Our method, connectivity-based CE-Net (Bicon-CE), defines layer segmentation as a combination of pixel connectivity modeling and pixel-wise tissue classification. Bicon-CE outperformed other widely used neural networks and reduced common topological prediction issues in tissues from healthy patients and from patients with Barrett's esophagus. This is the first end-to-end learning method developed for automatic segmentation of the epithelium in in vivo human esophageal OCT images.

Tissue self-attention network for the segmentation of optical coherence tomography images on the esophagus

Automatic segmentation of layered tissue is the key to esophageal optical coherence tomography (OCT) image processing. With the advent of deep learning techniques, frameworks based on a fully convolutional network are proved to be effective in classifying pixels on images. However, due to speckle noise and unfavorable imaging conditions, the esophageal tissue relevant to the diagnosis is not always easy to identify. An effective approach to address this problem is extracting more powerful feature maps, which have similar expressions for pixels in the same tissue and show discriminability from those from different tissues. In this study, we proposed a novel framework, called the tissue self-attention network (TSA-Net), which introduces the self-attention mechanism for esophageal OCT image segmentation. The self-attention module in the network is able to capture long-range context dependencies from the image and analyzes the input image in a global view, which helps to cluster pixels in the same tissue and reveal differences of different layers, thus achieving more powerful feature maps for segmentation. Experiments have visually illustrated the effectiveness of the self-attention map, and its advantages over other deep networks were also discussed.

Colon phantoms with cancer lesions for endoscopic characterization with optical coherence tomography

Optical coherence tomography (OCT) is a growing imaging technique for real-time early diagnosis of digestive system diseases. As with other well-established medical imaging modalities, OCT requires validated imaging performance and standardized test methods for performance assessment. A major limitation in the development and testing of new imaging technologies is the lack of models for simultaneous clinical procedure emulation and characterization of healthy and diseased tissues. Currently, the former can be tested in large animal models and the latter can be tested in small animal disease models or excised human biopsy samples. In this study, a 23 cm by 23 cm optical phantom was developed to mimic the thickness and near-infrared optical properties of each anatomical layer of a human colon, as well as the surface topography of colorectal polyps and visual appearance compatible with white light endoscopy.

Adversarial convolutional network for esophageal tissue segmentation on OCT images

Automatic segmentation is important for esophageal OCT image processing, which is able to provide tissue characteristics such as shape and thickness for disease diagnosis. Existing automatical segmentation methods based on deep convolutional networks may not generate accurate segmentation results due to limited training set and various layer shapes. This study proposed a novel adversarial convolutional network (ACN) to segment esophageal OCT images using a convolutional network trained by adversarial learning. The proposed framework includes a generator and a discriminator, both with U-Net alike fully convolutional architecture. The discriminator is a hybrid network that discriminates whether the generated results are real and implements pixel classification at the same time. Leveraging on the adversarial training, the discriminator becomes more powerful. In addition, the adversarial loss is able to encode high order relationships of pixels, thus eliminating the requirements of post-processing. Experiments on segmenting esophageal OCT images from guinea pigs confirmed that the ACN outperforms several deep learning frameworks in pixel classification accuracy and improves the segmentation result. The potential clinical application of ACN for detecting eosinophilic esophagitis (EoE), an esophageal disease, is also presented in the experiment.

Adversarial convolutional network for esophageal tissue segmentation on OCT images

Automatic segmentation is important for esophageal OCT image processing, which is able to provide tissue characteristics such as shape and thickness for disease diagnosis. Existing automatical segmentation methods based on deep convolutional networks may not generate accurate segmentation results due to limited training set and various layer shapes. This study proposed a novel adversarial convolutional network (ACN) to segment esophageal OCT images using a convolutional network trained by adversarial learning. The proposed framework includes a generator and a discriminator, both with U-Net alike fully convolutional architecture. The discriminator is a hybrid network that discriminates whether the generated results are real and implements pixel classification at the same time. Leveraging on the adversarial training, the discriminator becomes more powerful. In addition, the adversarial loss is able to encode high order relationships of pixels, thus eliminating the requirements of post-processing. Experiments on segmenting esophageal OCT images from guinea pigs confirmed that the ACN outperforms several deep learning frameworks in pixel classification accuracy and improves the segmentation result. The potential clinical application of ACN for detecting eosinophilic esophagitis (EoE), an esophageal disease, is also presented in the experiment.

Robust, accurate depth-resolved attenuation characterization in optical coherence tomography

Depth-resolved optical attenuation coefficient is a valuable tissue parameter that complements the intensity-based structural information in optical coherent tomography (OCT) imaging. Herein we systematically analyzed the under- and over-estimation bias of existing depth-resolved methods when applied to real biological tissues, and then proposed a new algorithm that remedies these issues and accommodates general OCT data that contain incomplete decay and noise floor, thereby affording consistent estimation accuracy for practical biological samples of different scattering properties. Compared with other algorithms, our method demonstrates remarkably improved estimation accuracy and numerical robustness, as validated via numerical simulations and on experimental OCT data obtained from both silicone-TiO₂ phantoms and human ventral tongue leukoplakia samples.

Parallel deep neural networks for endoscopic OCT image segmentation

We report parallel-trained deep neural networks for automated endoscopic OCT image segmentation feasible even with a limited training data set. These U-Net-based deep neural networks were trained using a modified dice loss function and manual segmentations of ultrahigh-resolution cross-sectional images collected by an 800 nm OCT endoscopic system. The method was tested on in vivo guinea pig esophagus images. Results showed its robust layer segmentation capability with a boundary error of 1.4 µm insensitive to lay topology disorders. To further illustrate its clinical potential, the method was applied to differentiating in vivo OCT esophagus images from an eosinophilic esophagitis (EOE) model and its control group, and the results clearly demonstrated quantitative changes in the top esophageal layers' thickness in the EOE model.

Automatic three-dimensional segmentation of endoscopic airway OCT images

Automatic delineation and segmentation of airway structures from endoscopic optical coherence tomography (OCT) images improve image analysis efficiency and thus has been of particular interest. Conventional two-dimensional automatic segmentation methods, such as the dynamic programming approach, ensures the edge-continuity in the xz-direction (intra-B-scan), but fails to preserve the surface-continuity when concerning the y-direction (inter-B-scan). To solve this, we present a novel automatic three-dimensional (3D) airway segmentation strategy. Our segmentation scheme includes an artifact-oriented pre-processing pipeline and a modified 3D optimal graph search algorithm incorporating adaptive tissue-curvature adjustment. The proposed algorithm is tested on endoscopic airway OCT image data sets acquired by different swept-source OCT platforms, and on different animal and human models. With our method, the results show continuous surface segmentation performance, which is both robust and accurate.

Fast esophageal layer segmentation in OCT images of guinea pigs based on sparse Bayesian classification and graph search

Endoscopic optical coherence tomography (OCT) devices are capable of generating high-resolution images of esophageal structures at high speed. To make the obtained data easy to interpret and reveal the clinical significance, an automatic segmentation algorithm is needed. This work proposes a fast algorithm combining sparse Bayesian learning and graph search (termed as SBGS) to automatically identify six layer boundaries on esophageal OCT images. The SBGS first extracts features, including multi-scale gradients, averages and Gabor wavelet coefficients, to train the sparse Bayesian classifier, which is used to generate probability maps indicating boundary positions. Given these probability maps, the graph search method is employed to create the final continuous smooth boundaries. The segmentation performance of the proposed SBGS algorithm was verified by esophageal OCT images from healthy guinea pigs and the eosinophilic esophagitis (EoE) models. Experiments confirmed that the SBGS method is able to implement robust esophageal segmentation for all the tested cases. In addition, benefiting from the sparse model of SBGS, the segmentation efficiency is significantly improved compared to other widely used techniques.

Evaluation of segmentation algorithms for optical coherence tomography images of ovarian tissue

Ovarian cancer has the lowest survival rate among all gynecologic cancers predominantly due to late diagnosis. Early detection of ovarian cancer can increase 5-year survival rates from 40% up to 92%, yet no reliable early detection techniques exist. Optical coherence tomography (OCT) is an emerging technique that provides depth-resolved, high-resolution images of biological tissue in real-time and demonstrates great potential for imaging of ovarian tissue. Mouse models are crucial to quantitatively assess the diagnostic potential of OCT for ovarian cancer imaging; however, due to small organ size, the ovaries must first be separated from the image background using the process of segmentation. Manual segmentation is time-intensive, as OCT yields three-dimensional data. Furthermore, speckle noise complicates OCT images, frustrating many processing techniques. While much work has investigated noise-reduction and automated segmentation for retinal OCT imaging, little has considered the application to the ovaries, which exhibit higher variance and inhomogeneity than the retina. To address these challenges, we evaluate a set of algorithms to segment OCT images of mouse ovaries. We examine five preprocessing techniques and seven segmentation algorithms. While all preprocessing methods improve segmentation, Gaussian filtering is most effective, showing an improvement of 32 % ± 1.2 % . Of the segmentation algorithms, active contours performs best, segmenting with an accuracy of 94.8 % ± 1.2 % compared with manual segmentation. Even so, further optimization could lead to maximizing the performance for segmenting OCT images of the ovaries.

Robust layer segmentation of esophageal OCT images based on graph search using edge-enhanced weights

Automatic segmentation of esophageal layers in OCT images is crucial for studying esophageal diseases and computer-assisted diagnosis. This work aims to improve the current techniques to increase the accuracy and robustness for esophageal OCT image segmentation. A two-step edge-enhanced graph search (EEGS) framework is proposed in this study. Firstly, a preprocessing scheme is applied to suppress speckle noise and remove the disturbance in the esophageal structure. Secondly, the image is formulated into a graph and layer boundaries are located by graph search. In this process, we propose an edge-enhanced weight matrix for the graph by combining the vertical gradients with a Canny edge map. Experiments on esophageal OCT images from guinea pigs demonstrate that the EEGS framework is more robust and more accurate than the current segmentation method. It can be potentially useful for the early detection of esophageal diseases.

Automated 3D segmentation of methyl isocyanate-exposed rat trachea using an ultra-thin, fully fiber optic optical coherence endoscopic probe

Development of effective rescue countermeasures for toxic inhalational industrial chemicals, such as methyl isocyanate (MIC), has been an emerging interest. Nonetheless, current methods for studying toxin-induced airway injuries are limited by cost, labor time, or accuracy, and only provide indirect or localized information. Optical Coherence Tomography (OCT) endoscopic probes have previously been used to visualize the 3-D airway structure. However, gathering such information in small animal models, such as rat airways after toxic gas exposure, remains a challenge due to the required probe size necessary for accessing the small, narrow, and partially obstructed tracheas. In this study, we have designed a 0.4 mm miniature endoscopic probe and investigated the structural changes in rat trachea after MIC inhalation. An automated 3D segmentation algorithm was implemented so that anatomical changes, such as tracheal lumen volume and cross-sectional areas, could be quantified. The tracheal region of rats exposed to MIC by inhalation showed significant airway narrowing, especially within the upper trachea, as a result of epithelial detachment and extravascular coagulation within the airway. This imaging and automated reconstruction technique is capable of rapid and minimally-invasive identification of airway obstruction. This method can be applied to large-scale quantitative analysis of in vivo animal models.