Digital labeling for 3D histology: segmenting blood vessels without a vascular contrast agent using deep learning

Recent advances in optical tissue clearing and three-dimensional (3D) fluorescence microscopy have enabled high resolution in situ imaging of intact tissues. Using simply prepared samples, we demonstrate here "digital labeling," a method to segment blood vessels in 3D volumes solely based on the autofluorescence signal and a nuclei stain (DAPI). We trained a deep-learning neural network based on the U-net architecture using a regression loss instead of a commonly used segmentation loss to achieve better detection of small vessels. We achieved high vessel detection accuracy and obtained accurate vascular morphometrics such as vessel length density and orientation. In the future, such digital labeling approach could easily be transferred to other biological structures.

Deep-learning segmentation of fascicles from microCT of the human vagus nerve

Introduction

MicroCT of the three-dimensional fascicular organization of the human vagus nerve provides essential data to inform basic anatomy as well as the development and optimization of neuromodulation therapies. To process the images into usable formats for subsequent analysis and computational modeling, the fascicles must be segmented. Prior segmentations were completed manually due to the complex nature of the images, including variable contrast between tissue types and staining artifacts.

Methods

Here, we developed a U-Net convolutional neural network (CNN) to automate segmentation of fascicles in microCT of human vagus nerve.

Results

The U-Net segmentation of ~500 images spanning one cervical vagus nerve was completed in 24 s, versus ~40 h for manual segmentation, i.e., nearly four orders of magnitude faster. The automated segmentations had a Dice coefficient of 0.87, a measure of pixel-wise accuracy, thus suggesting a rapid and accurate segmentation. While Dice coefficients are a commonly used metric to assess segmentation performance, we also adapted a metric to assess fascicle-wise detection accuracy, which showed that our network accurately detects the majority of fascicles, but may under-detect smaller fascicles.

Discussion

This network and the associated performance metrics set a benchmark, using a standard U-Net CNN, for the application of deep-learning algorithms to segment fascicles from microCT images. The process may be further optimized by refining tissue staining methods, modifying network architecture, and expanding the ground-truth training data. The resulting three-dimensional segmentations of the human vagus nerve will provide unprecedented accuracy to define nerve morphology in computational models for the analysis and design of neuromodulation therapies.

Segmentation of beating embryonic heart structures from 4-D OCT images using deep learning

Optical coherence tomography (OCT) has been used to investigate heart development because of its capability to image both structure and function of beating embryonic hearts. Cardiac structure segmentation is a prerequisite for the quantification of embryonic heart motion and function using OCT. Since manual segmentation is time-consuming and labor-intensive, an automatic method is needed to facilitate high-throughput studies. The purpose of this study is to develop an image-processing pipeline to facilitate the segmentation of beating embryonic heart structures from a 4-D OCT dataset. Sequential OCT images were obtained at multiple planes of a beating quail embryonic heart and reassembled to a 4-D dataset using image-based retrospective gating. Multiple image volumes at different time points were selected as key-volumes, and their cardiac structures including myocardium, cardiac jelly, and lumen, were manually labeled. Registration-based data augmentation was used to synthesize additional labeled image volumes by learning transformations between key-volumes and other unlabeled volumes. The synthesized labeled images were then used to train a fully convolutional network (U-Net) for heart structure segmentation. The proposed deep learning-based pipeline achieved high segmentation accuracy with only two labeled image volumes and reduced the time cost of segmenting one 4-D OCT dataset from a week to two hours. Using this method, one could carry out cohort studies that quantify complex cardiac motion and function in developing hearts.

Automated endocardial cushion segmentation and cellularization quantification in developing hearts using optical coherence tomography

Of all congenital heart defects (CHDs), anomalies in heart valves and septa are among the most common and contribute about fifty percent to the total burden of CHDs. Progenitors to heart valves and septa are endocardial cushions formed in looping hearts through a multi-step process that includes localized expansion of cardiac jelly, endothelial-to-mesenchymal transition, cell migration and proliferation. To characterize the development of endocardial cushions, previous studies manually measured cushion size or cushion cell density from images obtained using histology, immunohistochemistry, or optical coherence tomography (OCT). Manual methods are time-consuming and labor-intensive, impeding their applications in cohort studies that require large sample sizes. This study presents an automated strategy to rapidly characterize the anatomy of endocardial cushions from OCT images. A two-step deep learning technique was used to detect the location of the heart and segment endocardial cushions. The acellular and cellular cushion regions were then segregated by K-means clustering. The proposed method can quantify cushion development by measuring the cushion volume and cellularized fraction, and also map 3D spatial organization of the acellular and cellular cushion regions. The application of this method to study the developing looping hearts allowed us to discover a spatial asymmetry of the acellular cardiac jelly in endocardial cushions during these critical stages, which has not been reported before.

Nuclei Detection for 3D Microscopy With a Fully Convolutional Regression Network

Advances in three-dimensional microscopy and tissue clearing are enabling whole-organ imaging with single-cell resolution. Fast and reliable image processing tools are needed to analyze the resulting image volumes, including automated cell detection, cell counting and cell analytics. Deep learning approaches have shown promising results in two- and three-dimensional nuclei detection tasks, however detecting overlapping or non-spherical nuclei of different sizes and shapes in the presence of a blurring point spread function remains challenging and often leads to incorrect nuclei merging and splitting. Here we present a new regression-based fully convolutional network that located a thousand nuclei centroids with high accuracy in under a minute when combined with V-net, a popular three-dimensional semantic-segmentation architecture. High nuclei detection F1-scores of 95.3% and 92.5% were obtained in two different whole quail embryonic hearts, a tissue type difficult to segment because of its high cell density, and heterogeneous and elliptical nuclei. Similar high scores were obtained in the mouse brain stem, demonstrating that this approach is highly transferable to nuclei of different shapes and intensities. Finally, spatial statistics were performed on the resulting centroids. The spatial distribution of nuclei obtained by our approach most resembles the spatial distribution of manually identified nuclei, indicating that this approach could serve in future spatial analyses of cell organization.

Three-dimensional alignment of microvasculature and cardiomyocytes in the developing ventricle

While major coronary artery development and pathologies affecting them have been extensively studied, understanding the development and organization of the coronary microvasculature beyond the earliest developmental stages requires new tools. Without techniques to image the coronary microvasculature over the whole heart, it is likely we are underestimating the microvasculature’s impact on normal development and diseases. We present a new imaging and analysis toolset to visualize the coronary microvasculature in intact embryonic hearts and quantify vessel organization. The fluorescent dyes DiI and DAPI were used to stain the coronary vasculature and cardiomyocyte nuclei in quail embryo hearts during rapid growth and morphogenesis of the left ventricular wall. Vessel and cardiomyocytes orientation were automatically extracted and quantified, and vessel density was calculated. The coronary microvasculature was found to follow the known helical organization of cardiomyocytes in the ventricular wall. Vessel density in the left ventricle did not change during and after compaction. This quantitative and automated approach will enable future cohort studies to understand the microvasculature’s role in diseases such as hypertrophic cardiomyopathy where misalignment of cardiomyocytes has been observed in utero.

Semi-automated shear stress measurements in developing embryonic hearts

Blood-induced shear stress influences gene expression. Abnormal shear stress patterns on the endocardium of the early-stage heart tube can lead to congenital heart defects. To have a better understanding of these mechanisms, it is essential to include shear stress measurements in longitudinal cohort studies of cardiac development. Previously reported approaches are computationally expensive and nonpractical when assessing many animals. Here, we introduce a new approach to estimate shear stress that does not rely on recording 4D image sets and extensive post processing. Our method uses two adjacent optical coherence tomography frames (B-scans) where lumen geometry and flow direction are determined from the structural data and the velocity is measured from the Doppler OCT signal. We validated our shear stress estimate by flow phantom experiments and applied it to live quail embryo hearts where observed shear stress patterns were similar to previous studies.

Photothermal Optical Coherence Tomography of Anti-Angiogenic Treatment in the Mouse Retina Using Gold Nanorods as Contrast Agents

Purpose

Optical coherence tomography (OCT) is widely used for ocular imaging in clinical and research settings. OCT natively provides structural information based on the reflectivity of the tissues it images. We demonstrate the utility of photothermal OCT (PTOCT) imaging of gold nanorods (GNR) in the mouse retina in vivo in the laser-induced choroidal neovascularization (LCNV) model to provide additional image contrast within the lesion.

Methods

Wild-type C57BL/6 mice were imaged following the intravenous injection of ICAM2-targeted or untargeted GNR. Mice were also imaged following the injection of ICAM2-targeted GNR with or without the additional ocular delivery of a neutralizing monoclonal anti-vascular endothelial growth factor (anti-VEGF) antibody.

Results

Mice cohorts injected with untargeted or ICAM2-targeted GNR demonstrated increased lesion-associated photothermal signal during subsequent imaging relative to phosphate-buffered saline (PBS)-injected controls. Additionally, intravitreal injection of anti-VEGF antibody caused a detectable reduction in the extent of anatomic laser damage and lesion-associated photothermal signal density in mice treated in the LCNV model and injected with ICAM2-targeted GNR.

Conclusions

These experiments demonstrate the ability of PTOCT imaging of GNR to detect anti-VEGF-induced changes in the mouse retina using the LCNV model.

Translational Relevance

This study shows that PTOCT imaging of GNR in the LCNV model can be used to detect clinically relevant, anti-VEGF-induced changes that are not visible using standard OCT systems. In the future this technology could be used to aid in early detection of disease, monitoring disease progress, and assessing its response to therapies.

Imaging retinal melanin: a review of current technologies

The retinal pigment epithelium (RPE) is essential to the health of the retina and the proper functioning of the photoreceptors. The RPE is rich in melanosomes, which contain the pigment melanin. Changes in RPE pigmentation are seen with normal aging and in diseases such as albinism and age-related macular degeneration. However, most techniques used to this day to detect and quantify ocular melanin are performed ex vivo and are destructive to the tissue. There is a need for in vivo imaging of melanin both at the clinical and pre-clinical level to study how pigmentation changes can inform disease progression. In this manuscript, we review in vivo imaging techniques such as fundus photography, fundus reflectometry, near-infrared autofluorescence imaging, photoacoustic imaging, and functional optical coherence tomography that specifically detect melanin in the retina. These methods use different contrast mechanisms to detect melanin and provide images with different resolutions and field-of-views, making them complementary to each other.

Imaging Melanin Distribution in the Zebrafish Retina Using Photothermal Optical Coherence Tomography

Purpose

To demonstrate and validate that photothermal optical coherence tomography (PT-OCT) can image melanin in the retinal pigment epithelium (RPE) and can observe light-driven melanosome translocation in the zebrafish retina.

Methods

A commercial spectral domain OCT system was modified to perform both OCT and PT-OCT. Four adult tyrosinase-mosaic zebrafish with varying levels of melanin expression across their retinas were imaged, and the PT-OCT signal for pigmented and nonpigmented regions were compared. Wild-type dark-adapted (n = 11 fish) and light-adapted (n = 10 fish) zebrafish were also imaged with OCT and PT-OCT. Longitudinal reflectivity and absorption profiles were generated from B-scans to compare the melanin distribution between the two groups.

Results

A significant increase in PT-OCT signal (P < 0.0001, Student's t-test) was observed in pigmented regions of interest (ROI) compared to nonpigmented ROIs in the tyrosinase-mosaic zebrafish, which confirms the PT-OCT signal is specific to melanin in the eye. A significant increase in PT-OCT signal intensity (P < 0.0001, Student's t-test) was also detected in the light-adapted wild-type zebrafish group compared to the dark-adapted group. Additionally, light-adapted zebrafish display more distinct melanin banding patterns than do dark-adapted zebrafish in PT-OCT B-scans.

Conclusions

PT-OCT can detect different levels of melanin absorption and characterize pigment distribution in the zebrafish retina, including intracellular changes due to light-driven melanosome translocation within the RPE.

Translational Relevance

PT-OCT could quantify changes in pigmentation that occur in retinal diseases. The functional information provided by PT-OCT may also enable a better understanding of the anatomical features within conventional OCT images.

Diagnosis of immunomarkers in vivo via multiplexed surface enhanced Raman spectroscopy with gold nanostars

In this work, we demonstrate the targeted diagnosis of immunomarker programmed death ligand 1 (PD-L1) and simultaneous detection of epidermal growth factor receptor (EGFR) in breast cancer tumors in vivo using gold nanostars (AuNS) with multiplexed surface enhanced Raman spectroscopy (SERS). Real-time longitudinal tracking with SERS demonstrated maximum accumulation of AuNS occurred 6 h post intravenous (IV) delivery, enabling detection of both biomarkers simultaneously. Raman signal correlating to both PD-L1 and EGFR decreased by ∼30% in control tumors where receptors were pre-blocked prior to AuNS delivery, indicating both the sensitivity and specificity of SERS in distinguishing tumors with different levels of PD-L1 and EGFR expression. Our in vivo study was combined with the first demonstration of ex vivo SERS spatial maps of whole tumor lesions that provided both a qualitative and quantitative assessment of biomarker status with near cellular-level resolution. High resolution SERS maps also provided an overview of AuNS distribution in tumors which correlated well with the vascular density. Mass spectrometry showed AuNS accumulation in tumor and liver, and clearance via spleen, and electron microscopy revealed AuNS were endocytosed in tumors, Kupffer cells in the liver, and macrophages in the spleen. This study demonstrates that SERS-based diagnosis mediated by AuNS provides an accurate measure of multiple biomarkers both in vivo and ex vivo, which will ultimately enable a clinically-translatable platform for patient-tailored immunotherapies and combination treatments.

Photothermal optical coherence tomography of indocyanine green in ex vivo eyes

Indocyanine green (ICG) is routinely used during surgery to stain the inner limiting membrane (ILM) and provide contrast on white light surgical microscopy. While translation of optical coherence tomography (OCT) for intraoperative imaging during ophthalmic surgery has enhanced visualization, the ILM remains difficult to distinguish from underlying retinal structures and ICG does not provide additional OCT contrast. We present photothermal OCT (PT-OCT) for high-specificity detection of ICG on retinal OCT images. We demonstrate our technique by performing an ILM peel in ex vivo eyes using low ICG concentrations and laser powers. These results establish the feasibility of PT-OCT for intraoperative guidance during retinal surgery.

Depth-resolved analytical model and correction algorithm for photothermal optical coherence tomography

Photothermal OCT (PT-OCT) is an emerging molecular imaging technique that occupies a spatial imaging regime between microscopy and whole body imaging. PT-OCT would benefit from a theoretical model to optimize imaging parameters and test image processing algorithms. We propose the first analytical PT-OCT model to replicate an experimental A-scan in homogeneous and layered samples. We also propose the PT-CLEAN algorithm to reduce phase-accumulation and shadowing, two artifacts found in PT-OCT images, and demonstrate it on phantoms and in vivo mouse tumors.

Photothermal optical lock-in optical coherence tomography for in vivo imaging

Photothermal OCT (PTOCT) provides high sensitivity to molecular targets in tissue, and occupies a spatial imaging regime that is attractive for small animal imaging. However, current implementations of PTOCT require extensive temporal sampling, resulting in slow frame rates and a large data burden that limit its in vivo utility. To address these limitations, we have implemented optical lock-in techniques for photothermal optical lock-in OCT (poli-OCT), and demonstrated the in vivo imaging capabilities of this approach. The poli-OCT signal was assessed in tissue-mimicking phantoms containing indocyanine green (ICG), an FDA approved small molecule that has not been previously imaged in vivo with PTOCT. Then, the effects of in vivo blood flow and motion artifact were assessed and attenuated, and in vivo poli-OCT was demonstrated with both ICG and gold nanorods as contrast agents. Experiments revealed that poli-OCT signals agreed with optical lock-in theory and the bio-heat equation, and the system exhibited shot noise limited performance. In phantoms containing biologically relevant concentrations of ICG (1 µg/ml), the poli-OCT signal was significantly greater than control phantoms (p<0.05), demonstrating sensitivity to small molecules. Finally, in vivo poli-OCT of ICG identified the lymphatic vessels in a mouse ear, and also identified low concentrations (200 pM) of gold nanorods in subcutaneous injections at frame rates ten times faster than previously reported. This work illustrates that future in vivo molecular imaging studies could benefit from the improved acquisition and analysis times enabled by poli-OCT.