Enabling Histopathological Annotations on Immunofluorescent Images through Virtualization of Hematoxylin and Eosin

High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers

Precision medicine is critically dependent on better methods for diagnosing and staging disease and predicting drug response. Histopathology using hematoxylin and eosin (H&E)-stained tissue (not genomics) remains the primary diagnostic method in cancer. Recently developed highly multiplexed tissue imaging methods promise to enhance research studies and clinical practice with precise, spatially resolved single-cell data. Here, we describe the 'Orion' platform for collecting H&E and high-plex immunofluorescence images from the same cells in a whole-slide format suitable for diagnosis. Using a retrospective cohort of 74 colorectal cancer resections, we show that immunofluorescence and H&E images provide human experts and machine learning algorithms with complementary information that can be used to generate interpretable, multiplexed image-based models predictive of progression-free survival. Combining models of immune infiltration and tumor-intrinsic features achieves a 10- to 20-fold discrimination between rapid and slow (or no) progression, demonstrating the ability of multimodal tissue imaging to generate high-performance biomarkers.

The effect of neural network architecture on virtual H&E staining: Systematic assessment of histological feasibility

Conventional histopathology has relied on chemical staining for over a century. The staining process makes tissue sections visible to the human eye through a tedious and labor-intensive procedure that alters the tissue irreversibly, preventing repeated use of the sample. Deep learning-based virtual staining can potentially alleviate these shortcomings. Here, we used standard brightfield microscopy on unstained tissue sections and studied the impact of increased network capacity on the resulting virtually stained H&E images. Using the generative adversarial neural network model pix2pix as a baseline, we observed that replacing simple convolutions with dense convolution units increased the structural similarity score, peak signal-to-noise ratio, and nuclei reproduction accuracy. We also demonstrated highly accurate reproduction of histology, especially with increased network capacity, and demonstrated applicability to several tissues. We show that network architecture optimization can improve the image translation accuracy of virtual H&E staining, highlighting the potential of virtual staining in streamlining histopathological analysis.

Significance of broad-spectrum racemases for the viability and pathogenicity of Aeromonas hydrophila

Aim: To investigate the function of broad-spectrum racemases in Aeromonas hydrophila (BsrA). Results: The A. hydrophila gene encoding BsrA (bsr) mutants (AHΔbsr) exhibited a significant decrease in growth, motility, extracellular protease production and biofilm formation compared with the wild-type. Furthermore, bsr gene knockout instigated cell wall damage compared with the wild-type strains. The survival rate and replication capability in the blood and organs of the AHΔbsr-infected mice were significantly decreased. The degree of tissue injury in the AHΔbsr-infected group was lower than that of the wild-type-infected group. Moreover, there was a significant decrease in the expression of 12 AHΔbsr virulence genes. Conclusion: The bsr gene is essential for the viability and virulence of A. hydrophila.

Development of Planar Illumination Strategies for Solving Mysteries in the Sub-Cellular Realm

Optical microscopy has vastly expanded the frontiers of structural and functional biology, due to the non-invasive probing of dynamic volumes in vivo. However, traditional widefield microscopy illuminating the entire field of view (FOV) is adversely affected by out-of-focus light scatter. Consequently, standard upright or inverted microscopes are inept in sampling diffraction-limited volumes smaller than the optical system's point spread function (PSF). Over the last few decades, several planar and structured (sinusoidal) illumination modalities have offered unprecedented access to sub-cellular organelles and 4D (3D + time) image acquisition. Furthermore, these optical sectioning systems remain unaffected by the size of biological samples, providing high signal-to-noise (SNR) ratios for objective lenses (OLs) with long working distances (WDs). This review aims to guide biologists regarding planar illumination strategies, capable of harnessing sub-micron spatial resolution with a millimeter depth of penetration.

Creating Virtual Hematoxylin and Eosin Images using Samples Imaged on a Commercial CODEX Platform

Multiparametric fluorescence imaging through CODEX allows the simultaneous imaging of many biomarkers in a single tissue section. While the digital fluorescence data thus obtained can provide highly specific characterizations of individual cells and microenvironments, the images obtained are different from those usually interpreted by pathologists (i.e., hematoxylin and eosin [H&E] slides and 3,3′-diaminobenzidine-stained immunohistochemistry slides). Having the fluorescence data plus coregistered H&E or similar data could facilitate the adoption of multiparametric imaging into regular workflows, as well as facilitate the transfer of algorithms and machine learning previously developed around H&E slides. Since commercial CODEX instruments do not produce H&E-like images by themselves, we developed a staining protocol and associated image processing to make “virtual H&E” images that can be incorporated into the CODEX workflow. While there are many ways to achieve virtual H&E images, including the use of a fluorescent nuclear stain and tissue autofluorescence to simulate eosin staining, we opted to combine fluorescent nuclear staining (through 4′,6-diamidino-2-phenylindole) with actual eosin staining. We also output images derived from fluorescent nuclear staining and autofluorescence images for additional evaluation.

Seamless Virtual Whole Slide Image Synthesis and Validation Using Perceptual Embedding Consistency

Stain virtualization is an application with growing interest in digital pathology allowing simulation of stained tissue images thus saving lab and tissue resources. Thanks to the success of Generative Adversarial Networks (GANs) and the progress of unsupervised learning, unsupervised style transfer GANs have been successfully used to generate realistic, clinically meaningful and interpretable images. The large size of high resolution Whole Slide Images (WSIs) presents an additional computational challenge. This makes tilewise processing necessary during training and inference of deep learning networks. Instance normalization has a substantial positive effect in style transfer GAN applications but with tilewise inference, it has the tendency to cause a tiling artifact in reconstructed WSIs. In this paper we propose a novel perceptual embedding consistency (PEC) loss forcing the network to learn color, contrast and brightness invariant features in the latent space and hence substantially reducing the aforementioned tiling artifact. Our approach results in more seamless reconstruction of the virtual WSIs. We validate our method quantitatively by comparing the virtually generated images to their corresponding consecutive real stained images. We compare our results to state-of-the-art unsupervised style transfer methods and to the measures obtained from consecutive real stained tissue slide images. We demonstrate our hypothesis about the effect of the PEC loss by comparing model robustness to color, contrast and brightness perturbations and visualizing bottleneck embeddings. We validate the robustness of the bottleneck feature maps by measuring their sensitivity to the different perturbations and using them in a tumor segmentation task. Additionally, we propose a preliminary validation of the virtual staining application by comparing interpretation of 2 pathologists on real and virtual tiles and inter-pathologist agreement.

Use of Deep Learning to Develop and Analyze Computational Hematoxylin and Eosin Staining of Prostate Core Biopsy Images for Tumor Diagnosis

IMPORTANCE

Histopathological diagnoses of tumors from tissue biopsy after hematoxylin and eosin (H&E) dye staining is the criterion standard for oncological care, but H&E staining requires trained operators, dyes and reagents, and precious tissue samples that cannot be reused.

OBJECTIVES

To use deep learning algorithms to develop models that perform accurate computational H&E staining of native nonstained prostate core biopsy images and to develop methods for interpretation of H&E staining deep learning models and analysis of computationally stained images by computer vision and clinical approaches.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study used hundreds of thousands of native nonstained RGB (red, green, and blue channel) whole slide image (WSI) patches of prostate core tissue biopsies obtained from excess tissue material from prostate core biopsies performed in the course of routine clinical care between January 7, 2014, and January 7, 2017, at Brigham and Women's Hospital, Boston, Massachusetts. Biopsies were registered with their H&E-stained versions. Conditional generative adversarial neural networks (cGANs) that automate conversion of native nonstained RGB WSI to computational H&E-stained images were then trained. Deidentified whole slide images of prostate core biopsy and medical record data were transferred to Massachusetts Institute of Technology, Cambridge, for computational research. Results were shared with physicians for clinical evaluations. Data were analyzed from July 2018 to February 2019.

MAIN OUTCOMES AND MEASURES

Methods for detailed computer vision image analytics, visualization of trained cGAN model outputs, and clinical evaluation of virtually stained images were developed. The main outcome was interpretable deep learning models and computational H&E-stained images that achieved high performance in these metrics.

RESULTS

Among 38 patients who provided samples, single core biopsy images were extracted from each whole slide, resulting in 102 individual nonstained and H&E dye-stained image pairs that were compared with matched computationally stained and unstained images. Calculations showed high similarities between computationally and H&E dye-stained images, with a mean (SD) structural similarity index (SSIM) of 0.902 (0.026), Pearson correlation coefficient (PCC) of 0.962 (0.096), and peak signal to noise ratio (PSNR) of 22.821 (1.232) dB. A second cGAN performed accurate computational destaining of H&E-stained images back to their native nonstained form, with a mean (SD) SSIM of 0.900 (0.030), PCC of 0.963 (0.011), and PSNR of 25.646 (1.943) dB compared with native nonstained images. A single blind prospective study computed approximately 95% pixel-by-pixel overlap among prostate tumor annotations provided by 5 board certified pathologists on computationally stained images, compared with those on H&E dye-stained images. This study also used the first visualization and explanation of neural network kernel activation maps during H&E staining and destaining of RGB images by cGANs. High similarities between kernel activation maps of computationally and H&E-stained images (mean-squared errors <0.0005) provide additional mathematical and mechanistic validation of the staining system.

CONCLUSIONS AND RELEVANCE

These findings suggest that computational H&E staining of native unlabeled RGB images of prostate core biopsy could reproduce Gleason grade tumor signatures that were easily assessed and validated by clinicians. Methods for benchmarking, visualization, and clinical validation of deep learning models and virtually H&E-stained images communicated in this study have wide applications in clinical informatics and oncology research. Clinical researchers may use these systems for early indications of possible abnormalities in native nonstained tissue biopsies prior to histopathological workflows.

Proteome Imaging: From Classic to Modern Mass Spectrometry-Based Molecular Histology

In order to overcome the limitations of classic imaging in Histology during the actually era of multiomics, the multi-color "molecular microscope" by its emerging "molecular pictures" offers quantitative and spatial information about thousands of molecular profiles without labeling of potential targets. Healthy and diseased human tissues, as well as those of diverse invertebrate and vertebrate animal models, including genetically engineered species and cultured cells, can be easily analyzed by histology-directed MALDI imaging mass spectrometry. The aims of this review are to discuss a range of proteomic information emerging from MALDI mass spectrometry imaging comparative to classic histology, histochemistry and immunohistochemistry, with applications in biology and medicine, concerning the detection and distribution of structural proteins and biological active molecules, such as antimicrobial peptides and proteins, allergens, neurotransmitters and hormones, enzymes, growth factors, toxins and others. The molecular imaging is very well suited for discovery and validation of candidate protein biomarkers in neuroproteomics, oncoproteomics, aging and age-related diseases, parasitoproteomics, forensic, and ecotoxicology. Additionally, in situ proteome imaging may help to elucidate the physiological and pathological mechanisms involved in developmental biology, reproductive research, amyloidogenesis, tumorigenesis, wound healing, neural network regeneration, matrix mineralization, apoptosis and oxidative stress, pain tolerance, cell cycle and transformation under oncogenic stress, tumor heterogeneity, behavior and aggressiveness, drugs bioaccumulation and biotransformation, organism's reaction against environmental penetrating xenobiotics, immune signaling, assessment of integrity and functionality of tissue barriers, behavioral biology, and molecular origins of diseases. MALDI MSI is certainly a valuable tool for personalized medicine and "Eco-Evo-Devo" integrative biology in the current context of global environmental challenges.