Author Correction: U-Net: deep learning for cell counting, detection, and morphometry

C1M2: a universal algorithm for 3D instance segmentation, annotation, and quantification of irregular cells

Cell instance segmentation is a fundamental task for many biological applications, especially for packed cells in three-dimensional (3D) microscope images that can fully display cellular morphology. Image processing algorithms based on neural networks and feature engineering have enabled great progress in two-dimensional (2D) instance segmentation. However, current methods cannot achieve high segmentation accuracy for irregular cells in 3D images. In this study, we introduce a universal, morphology-based 3D instance segmentation algorithm called Crop Once Merge Twice (C1M2), which can segment cells from a wide range of image types and does not require nucleus images. C1M2 can be extended to quantify the fluorescence intensity of fluorescent proteins and antibodies and automatically annotate their expression levels in individual cells. Our results suggest that C1M2 can serve as a tissue cytometry for 3D histopathological assays by quantifying fluorescence intensity with spatial localization and morphological information.

Predicting Recurrence in Pancreatic Ductal Adenocarcinoma after Radical Surgery Using an AX-Unet Pancreas Segmentation Model and Dynamic Nomogram

This study aims to investigate the reliability of radiomic features extracted from contrast-enhanced computer tomography (CT) by AX-Unet, a pancreas segmentation model, to analyse the recurrence of pancreatic ductal adenocarcinoma (PDAC) after radical surgery. In this study, we trained an AX-Unet model to extract the radiomic features from preoperative contrast-enhanced CT images on a training set of 205 PDAC patients. Then we evaluated the segmentation ability of AX-Unet and the relationship between radiomic features and clinical characteristics on an independent testing set of 64 patients with clear prognoses. The lasso regression analysis was used to screen for variables of interest affecting patients' post-operative recurrence, and the Cox proportional risk model regression analysis was used to screen for risk factors and create a nomogram prediction model. The proposed model achieved an accuracy of 85.9% for pancreas segmentation, meeting the requirements of most clinical applications. Radiomic features were found to be significantly correlated with clinical characteristics such as lymph node metastasis, resectability status, and abnormally elevated serum carbohydrate antigen 19-9 (CA 19-9) levels. Specifically, variance and entropy were associated with the recurrence rate (p < 0.05). The AUC for the nomogram predicting whether the patient recurred after surgery was 0.92 (95% CI: 0.78-0.99) and the C index was 0.62 (95% CI: 0.48-0.78). The AX-Unet pancreas segmentation model shows promise in analysing recurrence risk factors after radical surgery for PDAC. Additionally, our findings suggest that a dynamic nomogram model based on AX-Unet can provide pancreatic oncologists with more accurate prognostic assessments for their patients.

Deep-learning-enabled brain hemodynamic mapping using resting-state fMRI

Cerebrovascular disease is a leading cause of death globally. Prevention and early intervention are known to be the most effective forms of its management. Non-invasive imaging methods hold great promises for early stratification, but at present lack the sensitivity for personalized prognosis. Resting-state functional magnetic resonance imaging (rs-fMRI), a powerful tool previously used for mapping neural activity, is available in most hospitals. Here we show that rs-fMRI can be used to map cerebral hemodynamic function and delineate impairment. By exploiting time variations in breathing pattern during rs-fMRI, deep learning enables reproducible mapping of cerebrovascular reactivity (CVR) and bolus arrival time (BAT) of the human brain using resting-state CO2 fluctuations as a natural "contrast media". The deep-learning network is trained with CVR and BAT maps obtained with a reference method of CO2-inhalation MRI, which includes data from young and older healthy subjects and patients with Moyamoya disease and brain tumors. We demonstrate the performance of deep-learning cerebrovascular mapping in the detection of vascular abnormalities, evaluation of revascularization effects, and vascular alterations in normal aging. In addition, cerebrovascular maps obtained with the proposed method exhibit excellent reproducibility in both healthy volunteers and stroke patients. Deep-learning resting-state vascular imaging has the potential to become a useful tool in clinical cerebrovascular imaging.

A deep learning framework for quantitative analysis of actin microridges

Microridges are evolutionarily conserved actin-rich protrusions present on the apical surface of squamous epithelial cells. In zebrafish epidermal cells, microridges form self-evolving patterns due to the underlying actomyosin network dynamics. However, their morphological and dynamic characteristics have remained poorly understood owing to a lack of computational methods. We achieved ~95% pixel-level accuracy with a deep learning microridge segmentation strategy enabling quantitative insights into their bio-physical-mechanical characteristics. From the segmented images, we estimated an effective microridge persistence length of ~6.1 μm. We discovered the presence of mechanical fluctuations and found relatively greater stresses stored within patterns of yolk than flank, indicating distinct regulation of their actomyosin networks. Furthermore, spontaneous formations and positional fluctuations of actin clusters within microridges were associated with pattern rearrangements over short length/time-scales. Our framework allows large-scale spatiotemporal analysis of microridges during epithelial development and probing of their responses to chemical and genetic perturbations to unravel the underlying patterning mechanisms.

Tumour infiltrating lymphocytes and survival after adjuvant chemotherapy in patients with gastric cancer: post-hoc analysis of the CLASSIC trial

BACKGROUND

Only a subset of gastric cancer (GC) patients with stage II-III benefits from chemotherapy after surgery. Tumour infiltrating lymphocytes per area (TIL density) has been suggested as a potential predictive biomarker of chemotherapy benefit.

METHODS

We quantified TIL density in digital images of haematoxylin-eosin (HE) stained tissue using deep learning in 307 GC patients of the Yonsei Cancer Center (YCC) (193 surgery+adjuvant chemotherapy [S + C], 114 surgery alone [S]) and 629 CLASSIC trial GC patients (325 S + C and 304 S). The relationship between TIL density, disease-free survival (DFS) and clinicopathological variables was analysed.

RESULTS

YCC S patients and CLASSIC S patients with high TIL density had longer DFS than S patients with low TIL density (P = 0.007 and P = 0.013, respectively). Furthermore, CLASSIC patients with low TIL density had longer DFS if treated with S + C compared to S (P = 0.003). No significant relationship of TIL density with other clinicopathological variables was found.

CONCLUSION

This is the first study to suggest TIL density automatically quantified in routine HE stained tissue sections as a novel, clinically useful biomarker to identify stage II-III GC patients deriving benefit from adjuvant chemotherapy. Validation of our results in a prospective study is warranted.

Towards a comprehensive approach for characterizing cell activity in bright-field microscopic images

When studying physical cellular response observed by light microscopy, variations in cell behavior are difficult to quantitatively measure and are often only discussed on a subjective level. Hence, cell properties are described qualitatively based on a researcher’s impressions. In this study, we aim to define a comprehensive approach to estimate the physical cell activity based on migration and morphology based on statistical analysis of a cell population within a predefined field of view and timespan. We present quantitative measurements of the influence of drugs such as cytochalasin D and taxol on human neuroblastoma, SH-SY5Y cell populations. Both chemicals are well known to interact with the cytoskeleton and affect the cell morphology and motility. Being able to compute the physical properties of each cell for a given observation time, requires precise localization of each cell even when in an adhesive state, where cells are not visually differentiable. Also, the risk of confusion through contaminants is desired to be minimized. In relation to the cell detection process, we have developed a customized encoder-decoder based deep learning cell detection and tracking procedure. Further, we discuss the accuracy of our approach to quantify cell activity and its viability in regard to the cell detection accuracy.

Application value of a deep learning method based on a 3D V-Net convolutional neural network in the recognition and segmentation of the auditory ossicles

Objective

To explore the feasibility of a deep learning three-dimensional (3D) V-Net convolutional neural network to construct high-resolution computed tomography (HRCT)-based auditory ossicle structure recognition and segmentation models.

Methods

The temporal bone HRCT images of 158 patients were collected retrospectively, and the malleus, incus, and stapes were manually segmented. The 3D V-Net and U-Net convolutional neural networks were selected as the deep learning methods for segmenting the auditory ossicles. The temporal bone images were randomized into a training set (126 cases), a test set (16 cases), and a validation set (16 cases). Taking the results of manual segmentation as a control, the segmentation results of each model were compared.

Results

The Dice similarity coefficients (DSCs) of the malleus, incus, and stapes, which were automatically segmented with a 3D V-Net convolutional neural network and manually segmented from the HRCT images, were 0.920 ± 0.014, 0.925 ± 0.014, and 0.835 ± 0.035, respectively. The average surface distance (ASD) was 0.257 ± 0.054, 0.236 ± 0.047, and 0.258 ± 0.077, respectively. The Hausdorff distance (HD) 95 was 1.016 ± 0.080, 1.000 ± 0.000, and 1.027 ± 0.102, respectively. The DSCs of the malleus, incus, and stapes, which were automatically segmented using the 3D U-Net convolutional neural network and manually segmented from the HRCT images, were 0.876 ± 0.025, 0.889 ± 0.023, and 0.758 ± 0.044, respectively. The ASD was 0.439 ± 0.208, 0.361 ± 0.077, and 0.433 ± 0.108, respectively. The HD 95 was 1.361 ± 0.872, 1.174 ± 0.350, and 1.455 ± 0.618, respectively. As these results demonstrated, there was a statistically significant difference between the two groups (P < 0.001).

Conclusion

The 3D V-Net convolutional neural network yielded automatic recognition and segmentation of the auditory ossicles and produced similar accuracy to manual segmentation results.

Code-Free Development and Deployment of Deep Segmentation Models for Digital Pathology

Application of deep learning on histopathological whole slide images (WSIs) holds promise of improving diagnostic efficiency and reproducibility but is largely dependent on the ability to write computer code or purchase commercial solutions. We present a code-free pipeline utilizing free-to-use, open-source software (QuPath, DeepMIB, and FastPathology) for creating and deploying deep learning-based segmentation models for computational pathology. We demonstrate the pipeline on a use case of separating epithelium from stroma in colonic mucosa. A dataset of 251 annotated WSIs, comprising 140 hematoxylin-eosin (HE)-stained and 111 CD3 immunostained colon biopsy WSIs, were developed through active learning using the pipeline. On a hold-out test set of 36 HE and 21 CD3-stained WSIs a mean intersection over union score of 95.5 and 95.3% was achieved on epithelium segmentation. We demonstrate pathologist-level segmentation accuracy and clinical acceptable runtime performance and show that pathologists without programming experience can create near state-of-the-art segmentation solutions for histopathological WSIs using only free-to-use software. The study further demonstrates the strength of open-source solutions in its ability to create generalizable, open pipelines, of which trained models and predictions can seamlessly be exported in open formats and thereby used in external solutions. All scripts, trained models, a video tutorial, and the full dataset of 251 WSIs with ~31 k epithelium annotations are made openly available at https://github.com/andreped/NoCodeSeg to accelerate research in the field.

Computational Methods for Single-Cell Imaging and Omics Data Integration

Integrating single cell omics and single cell imaging allows for a more effective characterisation of the underlying mechanisms that drive a phenotype at the tissue level, creating a comprehensive profile at the cellular level. Although the use of imaging data is well established in biomedical research, its primary application has been to observe phenotypes at the tissue or organ level, often using medical imaging techniques such as MRI, CT, and PET. These imaging technologies complement omics-based data in biomedical research because they are helpful for identifying associations between genotype and phenotype, along with functional changes occurring at the tissue level. Single cell imaging can act as an intermediary between these levels. Meanwhile new technologies continue to arrive that can be used to interrogate the genome of single cells and its related omics datasets. As these two areas, single cell imaging and single cell omics, each advance independently with the development of novel techniques, the opportunity to integrate these data types becomes more and more attractive. This review outlines some of the technologies and methods currently available for generating, processing, and analysing single-cell omics- and imaging data, and how they could be integrated to further our understanding of complex biological phenomena like ageing. We include an emphasis on machine learning algorithms because of their ability to identify complex patterns in large multidimensional data.

Recent Progress in the Correlative Structured Illumination Microscopy

The super-resolution imaging technique of structured illumination microscopy (SIM) enables the mixing of high-frequency information into the optical transmission domain via light-source modulation, thus breaking the optical diffraction limit. Correlative SIM, which combines other techniques with SIM, offers more versatility or higher imaging resolution than traditional SIM. In this review, we first briefly introduce the imaging mechanism and development trends of conventional SIM. Then, the principles and recent developments of correlative SIM techniques are reviewed. Finally, the future development directions of SIM and its correlative microscopies are presented.

Helminth egg analysis platform (HEAP): An opened platform for microscopic helminth egg identification and quantification based on the integration of deep learning architectures

BACKGROUND

Millions of people throughout the world suffer from parasite infections. Traditionally, technicians use manual eye inspection of microscopic specimens to perform a parasite examination. However, manual operations have limitations that hinder the ability to obtain precise egg counts and cause inefficient identification of infected parasites on co-infections. The technician requirements for handling a large number of microscopic examinations in countries that have limited medical resources are substantial. We developed the helminth egg analysis platform (HEAP) as a user-friendly microscopic helminth eggs identification and quantification platform to assist medical technicians during parasite infection examination.

METHODS

Multiple deep learning strategies including SSD (Single Shot MultiBox Detector), U-net, and Faster R-CNN (Faster Region-based Convolutional Neural Network) are integrated to identify the same specimen allowing users to choose the best predictions. An image binning and egg-in-edge algorithm based on pixel density detection was developed to increase the performance. Computers with different operation systems can be gathered to lower the computation time using our easy-to-deploy software architecture.

RESULTS

A user-friendly interface is provided to substantially increase the efficiency of manual validation. To adapt to low-cost computers, we architected a distributed computing structure with high flexibilities.

CONCLUSIONS

HEAP serves not only as a prediction service provider but also as a parasitic egg database of microscopic helminth egg image collection, labeling data and pretrained models. All images and labeling resources are free and accessible at http://heap.cgu.edu.tw. HEAP can also be an ideal education and training resource for helminth egg examination.

Deep learning-based molecular morphometrics for kidney biopsies

Morphologic examination of tissue biopsies is essential for histopathological diagnosis. However, accurate and scalable cellular quantification in human samples remains challenging. Here, we present a deep learning-based approach for antigen-specific cellular morphometrics in human kidney biopsies, which combines indirect immunofluorescence imaging with U-Net-based architectures for image-to-image translation and dual segmentation tasks, achieving human-level accuracy. In the kidney, podocyte loss represents a hallmark of glomerular injury and can be estimated in diagnostic biopsies. Thus, we profiled over 27,000 podocytes from 110 human samples, including patients with antineutrophil cytoplasmic antibody-associated glomerulonephritis (ANCA-GN), an immune-mediated disease with aggressive glomerular damage and irreversible loss of kidney function. We identified previously unknown morphometric signatures of podocyte depletion in patients with ANCA-GN, which allowed patient classification and, in combination with routine clinical tools, showed potential for risk stratification. Our approach enables robust and scalable molecular morphometric analysis of human tissues, yielding deeper biological insights into the human kidney pathophysiology.

DeepMIB: User-friendly and open-source software for training of deep learning network for biological image segmentation

We present DeepMIB, a new software package that is capable of training convolutional neural networks for segmentation of multidimensional microscopy datasets on any workstation. We demonstrate its successful application for segmentation of 2D and 3D electron and multicolor light microscopy datasets with isotropic and anisotropic voxels. We distribute DeepMIB as both an open-source multi-platform Matlab code and as compiled standalone application for Windows, MacOS and Linux. It comes in a single package that is simple to install and use as it does not require knowledge of programming. DeepMIB is suitable for everyone interested of bringing a power of deep learning into own image segmentation workflows.

A Novel Squeeze-and-Excitation W-Net for 2D and 3D Building Change Detection with Multi-Source and Multi-Feature Remote Sensing Data

Building Change Detection (BCD) is one of the core issues in earth observation and has received extensive attention in recent years. With the rapid development of earth observation technology, the data source of remote sensing change detection is continuously enriched, which provides the possibility to describe the spatial details of the ground objects more finely and to characterize the ground objects with multiple perspectives and levels. However, due to the different physical mechanisms of multi-source remote sensing data, BCD based on heterogeneous data is a challenge. Previous studies mostly focused on the BCD of homogeneous remote sensing data, while the use of multi-source remote sensing data and considering multiple features to conduct 2D and 3D BCD research is sporadic. In this article, we propose a novel and general squeeze-and-excitation W-Net, which is developed from U-Net and SE-Net. Its unique advantage is that it can not only be used for BCD of homogeneous and heterogeneous remote sensing data respectively but also can input both homogeneous and heterogeneous remote sensing data for 2D or 3D BCD by relying on its bidirectional symmetric end-to-end network architecture. Moreover, from a unique perspective, we use image features that are stable in performance and less affected by radiation differences and temporal changes. We innovatively introduced the squeeze-and-excitation module to explicitly model the interdependence between feature channels so that the response between the feature channels is adaptively recalibrated to improve the information mining ability and detection accuracy of the model. As far as we know, this is the first proposed network architecture that can simultaneously use multi-source and multi-feature remote sensing data for 2D and 3D BCD. The experimental results in two 2D data sets and two challenging 3D data sets demonstrate that the promising performances of the squeeze-and-excitation W-Net outperform several traditional and state-of-the-art approaches. Moreover, both visual and quantitative analyses of the experimental results demonstrate competitive performance in the proposed network. This demonstrates that the proposed network and method are practical, physically justified, and have great potential application value in large-scale 2D and 3D BCD and qualitative and quantitative research.

The ImageJ ecosystem: Open-source software for image visualization, processing, and analysis

For decades, biologists have relied on software to visualize and interpret imaging data. As techniques for acquiring images increase in complexity, resulting in larger multidimensional datasets, imaging software must adapt. ImageJ is an open-source image analysis software platform that has aided researchers with a variety of image analysis applications, driven mainly by engaged and collaborative user and developer communities. The close collaboration between programmers and users has resulted in adaptations to accommodate new challenges in image analysis that address the needs of ImageJ's diverse user base. ImageJ consists of many components, some relevant primarily for developers and a vast collection of user-centric plugins. It is available in many forms, including the widely used Fiji distribution. We refer to this entire ImageJ codebase and community as the ImageJ ecosystem. Here we review the core features of this ecosystem and highlight how ImageJ has responded to imaging technology advancements with new plugins and tools in recent years. These plugins and tools have been developed to address user needs in several areas such as visualization, segmentation, and tracking of biological entities in large, complex datasets. Moreover, new capabilities for deep learning are being added to ImageJ, reflecting a shift in the bioimage analysis community towards exploiting artificial intelligence. These new tools have been facilitated by profound architectural changes to the ImageJ core brought about by the ImageJ2 project. Therefore, we also discuss the contributions of ImageJ2 to enhancing multidimensional image processing and interoperability in the ImageJ ecosystem.