Generalized Fixation Invariant Nuclei Detection Through Domain Adaptation Based Deep Learning

Learning with limited target data to detect cells in cross-modality images

Deep neural networks have achieved excellent cell or nucleus quantification performance in microscopy images, but they often suffer from performance degradation when applied to cross-modality imaging data. Unsupervised domain adaptation (UDA) based on generative adversarial networks (GANs) has recently improved the performance of cross-modality medical image quantification. However, current GAN-based UDA methods typically require abundant target data for model training, which is often very expensive or even impossible to obtain for real applications. In this paper, we study a more realistic yet challenging UDA situation, where (unlabeled) target training data is limited and previous work seldom delves into cell identification. We first enhance a dual GAN with task-specific modeling, which provides additional supervision signals to assist with generator learning. We explore both single-directional and bidirectional task-augmented GANs for domain adaptation. Then, we further improve the GAN by introducing a differentiable, stochastic data augmentation module to explicitly reduce discriminator overfitting. We examine source-, target-, and dual-domain data augmentation for GAN enhancement, as well as joint task and data augmentation in a unified GAN-based UDA framework. We evaluate the framework for cell detection on multiple public and in-house microscopy image datasets, which are acquired with different imaging modalities, staining protocols and/or tissue preparations. The experiments demonstrate that our method significantly boosts performance when compared with the reference baseline, and it is superior to or on par with fully supervised models that are trained with real target annotations. In addition, our method outperforms recent state-of-the-art UDA approaches by a large margin on different datasets.

Deformation equivariant cross-modality image synthesis with paired non-aligned training data

Cross-modality image synthesis is an active research topic with multiple medical clinically relevant applications. Recently, methods allowing training with paired but misaligned data have started to emerge. However, no robust and well-performing methods applicable to a wide range of real world data sets exist. In this work, we propose a generic solution to the problem of cross-modality image synthesis with paired but non-aligned data by introducing new deformation equivariance encouraging loss functions. The method consists of joint training of an image synthesis network together with separate registration networks and allows adversarial training conditioned on the input even with misaligned data. The work lowers the bar for new clinical applications by allowing effortless training of cross-modality image synthesis networks for more difficult data sets.

Artificial intelligence and frozen section histopathology: A systematic review

Frozen sections are a useful pathologic tool, but variable image quality may impede the use of artificial intelligence and machine learning in their interpretation. We aimed to identify the current research on machine learning models trained or tested on frozen section images. We searched PubMed and Web of Science for articles presenting new machine learning models published in any year. Eighteen papers met all inclusion criteria. All papers presented at least one novel model trained or tested on frozen section images. Overall, convolutional neural networks tended to have the best performance. When physicians were able to view the output of the model, they tended to perform better than either the model or physicians alone at the tested task. Models trained on frozen sections performed well when tested on other slide preparations, but models trained on only formalin-fixed tissue performed significantly worse across other modalities. This suggests not only that machine learning can be applied to frozen section image processing, but also use of frozen section images may increase model generalizability. Additionally, expert physicians working in concert with artificial intelligence may be the future of frozen section histopathology.

The effect of neural network architecture on virtual H&E staining: Systematic assessment of histological feasibility

Conventional histopathology has relied on chemical staining for over a century. The staining process makes tissue sections visible to the human eye through a tedious and labor-intensive procedure that alters the tissue irreversibly, preventing repeated use of the sample. Deep learning-based virtual staining can potentially alleviate these shortcomings. Here, we used standard brightfield microscopy on unstained tissue sections and studied the impact of increased network capacity on the resulting virtually stained H&E images. Using the generative adversarial neural network model pix2pix as a baseline, we observed that replacing simple convolutions with dense convolution units increased the structural similarity score, peak signal-to-noise ratio, and nuclei reproduction accuracy. We also demonstrated highly accurate reproduction of histology, especially with increased network capacity, and demonstrated applicability to several tissues. We show that network architecture optimization can improve the image translation accuracy of virtual H&E staining, highlighting the potential of virtual staining in streamlining histopathological analysis.

Nuclear size rectification: A potential new therapeutic approach to reduce metastasis in cancer

Research on metastasis has recently regained considerable interest with the hope that single cell technologies might reveal the most critical changes that support tumor spread. However, it is possible that part of the answer has been visible through the microscope for close to 200 years. Changes in nuclear size characteristically occur in many cancer types when the cells metastasize. This was initially discarded as contributing to the metastatic spread because, depending on tumor types, both increases and decreases in nuclear size could correlate with increased metastasis. However, recent work on nuclear mechanics and the connectivity between chromatin, the nucleoskeleton, and the cytoskeleton indicate that changes in this connectivity can have profound impacts on cell mobility and invasiveness. Critically, a recent study found that reversing tumor type-dependent nuclear size changes correlated with reduced cell migration and invasion. Accordingly, it seems appropriate to now revisit possible contributory roles of nuclear size changes to metastasis.

Deep learning -- promises for 3D nuclear imaging: a guide for biologists

For the past century, the nucleus has been the focus of extensive investigations in cell biology. However, many questions remain about how its shape and size are regulated during development, in different tissues, or during disease and aging. To track these changes, microscopy has long been the tool of choice. Image analysis has revolutionized this field of research by providing computational tools that can be used to translate qualitative images into quantitative parameters. Many tools have been designed to delimit objects in 2D and, eventually, in 3D in order to define their shapes, their number or their position in nuclear space. Today, the field is driven by deep-learning methods, most of which take advantage of convolutional neural networks. These techniques are remarkably adapted to biomedical images when trained using large datasets and powerful computer graphics cards. To promote these innovative and promising methods to cell biologists, this Review summarizes the main concepts and terminologies of deep learning. Special emphasis is placed on the availability of these methods. We highlight why the quality and characteristics of training image datasets are important and where to find them, as well as how to create, store and share image datasets. Finally, we describe deep-learning methods well-suited for 3D analysis of nuclei and classify them according to their level of usability for biologists. Out of more than 150 published methods, we identify fewer than 12 that biologists can use, and we explain why this is the case. Based on this experience, we propose best practices to share deep-learning methods with biologists.