CoNIC Challenge: Pushing the frontiers of nuclear detection, segmentation, classification and counting

Artificial Intelligence-Based Single-Cell Analysis as a Next-Generation Histologic Grading Approach in Colorectal Cancer: Prognostic Role and Tumor Biology Assessment

The management of colorectal carcinoma (CRC) relies on pathological interpretation. Digital pathology approaches allow for development of new potent artificial intelligence-based prognostic parameters. The study aimed to develop an artificial intelligence-based image analysis platform allowing fully automatized, quantitative, and explainable tumor microenvironment analysis and extraction of prognostic information from hematoxylin and eosin-stained whole-slide images of CRC patients. Three well--characterized, multi-institutional patient cohorts were included (patient n = 1438, whole-slide image n > 2400). The developed image analysis platform implements quality control and established algorithms to segment tissue and detect cell types. It enabled systematic analysis of immune infiltrate, assessing its prognostic relevance, intratumoral heterogeneity, and biological concepts across multiple survival end points. Analyzing single-cell types and their combinations reveals independent, prognostic parameters, highlighting significant intratumoral heterogeneity, especially in the biopsy setting, which must be accounted for. A key morphologic concept related to tumor control by the immune system is described, resulting in a capable, independent prognostic parameter (tumor "out of control"). Our findings have direct clinical implications and can be used as a foundation for updating the existing CRC grading systems.

Integration of nuclear morphology and 3D imaging to profile cellular neighborhoods

Nuclear morphology is an indicator of cellular function and disease states, as changes in nuclear size, shape, and texture often reflect underlying disease-related genetic, epigenetic, and microenvironmental alterations. For disease diagnosis, nuclear segmentation performed in 2D hematoxylin and eosin (H&E)-stained tissue sections has long represented the gold standard. However, recent advances in three-dimensional (3D) histology, which provide a more biologically accurate representation of the spatial heterogeneity of human microanatomy, has led to improved understandings of disease pathology. Yet challenges remain in the development of scalable and computationally efficient pipelines for extracting and interpreting nuclear features in 3D space. 2D histology neglects crucial spatial information, such as 3D connectivity, morphology, and rare events missed by sparser sampling. Here, through extension of the CODA platform, we integrate 3D imaging with nuclear segmentation to analyze nuclear morphological features in human tissue. Analysis of 3D tissue microenvironments uncovered critical changes in 3D morphometric heterogeneity. Additionally, it enables the spatial characterization of immune cell distribution in relation to tissue structures, such as variations in leukocyte density near pancreatic ducts and blood vessels of different sizes. This approach provides a more comprehensive understanding of tissue and nuclear structures, revealing spatial patterns and interactions that are critical for disease progression.

Tissue concepts: Supervised foundation models in computational pathology

Due to the increasing workload of pathologists, the need for automation to support diagnostic tasks and quantitative biomarker evaluation is becoming more and more apparent. Foundation models have the potential to improve generalizability within and across centers and serve as starting points for data efficient development of specialized yet robust AI models. However, the training of foundation models themselves is usually very expensive in terms of data, computation, and time. This paper proposes a supervised training method that drastically reduces these expenses. The proposed method is based on multi-task learning to train a joint encoder, by combining 16 different classification, segmentation, and detection tasks on a total of 912,000 patches. Since the encoder is capable of capturing the properties of the samples, we term it the Tissue Concepts encoder. To evaluate the performance and generalizability of the Tissue Concepts encoder across centers, classification of whole slide images from four of the most prevalent solid cancers - breast, colon, lung, and prostate - was used. The experiments show that the Tissue Concepts model achieve comparable performance to models trained with self-supervision, while requiring only 6% of the amount of training patches. Furthermore, the Tissue Concepts encoder outperforms an ImageNet pre-trained encoder on both in-domain and out-of-domain data. The pre-trained models and will be made available under https://github.com/FraunhoferMEVIS/MedicalMultitaskModeling.

Nuclear pleomorphism in canine cutaneous mast cell tumors: Comparison of reproducibility and prognostic relevance between estimates, manual morphometry, and algorithmic morphometry

Variation in nuclear size and shape is an important criterion of malignancy for many tumor types; however, categorical estimates by pathologists have poor reproducibility. Measurements of nuclear characteristics can improve reproducibility, but current manual methods are time-consuming. The aim of this study was to explore the limitations of estimates and develop alternative morphometric solutions for canine cutaneous mast cell tumors (ccMCTs). We assessed the following nuclear evaluation methods for accuracy, reproducibility, and prognostic utility: (1) anisokaryosis estimates by 11 pathologists; (2) gold standard manual morphometry of at least 100 nuclei; (3) practicable manual morphometry with stratified sampling of 12 nuclei by 9 pathologists; and (4) automated morphometry using deep learning-based segmentation. The study included 96 ccMCTs with available outcome information. Inter-rater reproducibility of anisokaryosis estimates was low (k = 0.226), whereas it was good (intraclass correlation = 0.654) for practicable morphometry of the standard deviation (SD) of nuclear size. As compared with gold standard manual morphometry (area under the ROC curve [AUC] = 0.839, 95% confidence interval [CI] = 0.701-0.977), the prognostic value (tumor-specific survival) of SDs of nuclear area for practicable manual morphometry and automated morphometry were high with an AUC of 0.868 (95% CI = 0.737-0.991) and 0.943 (95% CI = 0.889-0.996), respectively. This study supports the use of manual morphometry with stratified sampling of 12 nuclei and algorithmic morphometry to overcome the poor reproducibility of estimates. Further studies are needed to validate our findings, determine inter-algorithmic reproducibility and algorithmic robustness, and explore tumor heterogeneity of nuclear features in entire tumor sections.

A novel AI-based score for assessing the prognostic value of intra-epithelial lymphocytes in oral epithelial dysplasia

BACKGROUND

Oral epithelial dysplasia (OED) poses a significant clinical challenge due to its potential for malignant transformation and the lack of reliable prognostic markers. Current OED grading systems do not reliably predict transformation and suffer from considerable observer variability. Recent studies have highlighted that peri-epithelial lymphocytes may play an important role in OED malignant transformation, with indication that intra-epithelial lymphocytes (IELs) may also be important.

METHODS

We propose a novel artificial intelligence (AI) based IEL score from Haematoxylin and Eosin (H&E) stained Whole Slide Images (WSIs) of OED tissue slides. We determine the prognostic value of our IEL score on a digital dataset of 219 OED WSIs (acquired using three different scanners), compared to pathologist-led clinical grading.

RESULTS

Our IEL scores demonstrated significant prognostic value (C-index = 0.67, p < 0.001) and were shown to improve both the binary/WHO grading systems in multivariate analyses (p < 0.001). Nuclear analyses confirmed the positive association between higher IEL scores, more severe OED and malignant transformation (p < 0.05).

CONCLUSIONS

This underscores the potential importance of IELs, and by extension our IEL score, as prognostic indicators in OED. Further validation through prospective multi-centric studies is warranted to confirm the clinical utility of IELs.

Proximity adjusted centroid mapping for accurate detection of nuclei in dense 3D cell systems

In the past decade, deep learning algorithms have surpassed the performance of many conventional image segmentation pipelines. Powerful models are now available for segmenting cells and nuclei in diverse 2D image types, but segmentation in 3D cell systems remains challenging due to the high cell density, the heterogenous resolution and contrast across the image volume, and the difficulty in generating reliable and sufficient ground truth data for model training. Reasoning that most image processing applications rely on nuclear segmentation but do not necessarily require an accurate delineation of their shapes, we implemented Proximity Adjusted Centroid MAPping (PAC-MAP), a 3D U-net based method that predicts the position of nuclear centroids and their proximity to other nuclei. We show that our model outperforms existing methods, predominantly by boosting recall, especially in conditions of high cell density. When trained from scratch with limited expert annotations (30 images), PAC-MAP attained an average F1 score of 0.793 for nuclei centroid prediction in dense spheroids. When pretraining using weakly supervised bulk data (>2300 images) followed by finetuning with the available expert annotations, the average F1 score could be significantly improved to 0.816. We demonstrate the utility of our method for quantifying the absolute cell content of spheroids and comprehensively mapping the infiltration pattern of patient-derived glioblastoma cells in cerebral organoids.

Deep learning for predicting prognostic consensus molecular subtypes in cervical cancer from histology images

Cervical cancer remains the fourth most common cancer among women worldwide. This study proposes an end-to-end deep learning framework to predict consensus molecular subtypes (CMS) in HPV-positive cervical squamous cell carcinoma (CSCC) from H&E-stained histology slides. Analysing three CSCC cohorts (n = 545), we show our Digital-CMS scores significantly stratify patients by both disease-specific (TCGA p = 0.0022, Oslo p = 0.0495) and disease-free (TCGA p = 0.0495, Oslo p = 0.0282) survival. In addition, our extensive tumour microenvironment analysis reveals differences between the two CMS subtypes, with CMS-C1 tumours exhibit increased lymphocyte presence, while CMS-C2 tumours show high nuclear pleomorphism, elevated neutrophil-to-lymphocyte ratio, and higher malignancy, correlating with poor prognosis. This study introduces a potentially clinically advantageous Digital-CMS score derived from digitised WSIs of routine H&E-stained tissue sections, offers new insights into TME differences impacting patient prognosis and potential therapeutic targets, and identifies histological patterns serving as potential surrogate markers of the CMS subtypes for clinical application.

A survey on cell nuclei instance segmentation and classification: Leveraging context and attention

Nuclear-derived morphological features and biomarkers provide relevant insights regarding the tumour microenvironment, while also allowing diagnosis and prognosis in specific cancer types. However, manually annotating nuclei from the gigapixel Haematoxylin and Eosin (H&E)-stained Whole Slide Images (WSIs) is a laborious and costly task, meaning automated algorithms for cell nuclei instance segmentation and classification could alleviate the workload of pathologists and clinical researchers and at the same time facilitate the automatic extraction of clinically interpretable features for artificial intelligence (AI) tools. But due to high intra- and inter-class variability of nuclei morphological and chromatic features, as well as H&E-stains susceptibility to artefacts, state-of-the-art algorithms cannot correctly detect and classify instances with the necessary performance. In this work, we hypothesize context and attention inductive biases in artificial neural networks (ANNs) could increase the performance and generalization of algorithms for cell nuclei instance segmentation and classification. To understand the advantages, use-cases, and limitations of context and attention-based mechanisms in instance segmentation and classification, we start by reviewing works in computer vision and medical imaging. We then conduct a thorough survey on context and attention methods for cell nuclei instance segmentation and classification from H&E-stained microscopy imaging, while providing a comprehensive discussion of the challenges being tackled with context and attention. Besides, we illustrate some limitations of current approaches and present ideas for future research. As a case study, we extend both a general (Mask-RCNN) and a customized (HoVer-Net) instance segmentation and classification methods with context- and attention-based mechanisms and perform a comparative analysis on a multicentre dataset for colon nuclei identification and counting. Although pathologists rely on context at multiple levels while paying attention to specific Regions of Interest (RoIs) when analysing and annotating WSIs, our findings suggest translating that domain knowledge into algorithm design is no trivial task, but to fully exploit these mechanisms in ANNs, the scientific understanding of these methods should first be addressed.

GrandQC: A comprehensive solution to quality control problem in digital pathology

Histological slides contain numerous artifacts that can significantly deteriorate the performance of image analysis algorithms. Here we develop the GrandQC tool for tissue and multi-class artifact segmentation. GrandQC allows for high-precision tissue segmentation (Dice score 0.957) and segmentation of tissue without artifacts (Dice score 0.919-0.938 dependent on magnification). Slides from 19 international pathology departments digitized with the most common scanning systems and from The Cancer Genome Atlas dataset were used to establish a QC benchmark, analyzing inter-institutional, intra-institutional, temporal, and inter-scanner slide quality variations. GrandQC improves the performance of downstream image analysis algorithms. We open-source the GrandQC tool, our large manually annotated test dataset, and all QC masks for the entire TCGA cohort to address the problem of QC in digital/computational pathology. GrandQC can be used as a tool to monitor sample preparation and scanning quality in pathology departments and help to track and eliminate major artifact sources.

Keep it accurate and robust: An enhanced nuclei analysis framework

Accurate segmentation and classification of nuclei in histology images is critical but challenging due to nuclei heterogeneity, staining variations, and tissue complexity. Existing methods often struggle with limited dataset variability, with patches extracted from similar whole slide images (WSI), making models prone to falling into local optima. Here we propose a new framework to address this limitation and enable robust nuclear analysis. Our method leverages dual-level ensemble modeling to overcome issues stemming from limited dataset variation. Intra-ensembling applies diverse transformations to individual samples, while inter-ensembling combines networks of different scales. We also introduce enhancements to the HoVer-Net architecture, including updated encoders, nested dense decoding and model regularization strategy. We achieve state-of-the-art results on public benchmarks, including 1st place for nuclear composition prediction and 3rd place for segmentation/classification in the 2022 Colon Nuclei Identification and Counting (CoNIC) Challenge. This success validates our approach for accurate histological nuclei analysis. Extensive experiments and ablation studies provide insights into optimal network design choices and training techniques. In conclusion, this work proposes an improved framework advancing the state-of-the-art in nuclei analysis. We will release our code and models to serve as a toolkit for the community.

Enhancing AI Research for Breast Cancer: A Comprehensive Review of Tumor-Infiltrating Lymphocyte Datasets

The field of immunology is fundamental to our understanding of the intricate dynamics of the tumor microenvironment. In particular, tumor-infiltrating lymphocyte (TIL) assessment emerges as essential aspect in breast cancer cases. To gain comprehensive insights, the quantification of TILs through computer-assisted pathology (CAP) tools has become a prominent approach, employing advanced artificial intelligence models based on deep learning techniques. The successful recognition of TILs requires the models to be trained, a process that demands access to annotated datasets. Unfortunately, this task is hampered not only by the scarcity of such datasets, but also by the time-consuming nature of the annotation phase required to create them. Our review endeavors to examine publicly accessible datasets pertaining to the TIL domain and thereby become a valuable resource for the TIL community. The overall aim of the present review is thus to make it easier to train and validate current and upcoming CAP tools for TIL assessment by inspecting and evaluating existing publicly available online datasets.

The multimodality cell segmentation challenge: toward universal solutions

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multimodality cell segmentation benchmark, comprising more than 1,500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.

NuInsSeg: A fully annotated dataset for nuclei instance segmentation in H&E-stained histological images

In computational pathology, automatic nuclei instance segmentation plays an essential role in whole slide image analysis. While many computerized approaches have been proposed for this task, supervised deep learning (DL) methods have shown superior segmentation performances compared to classical machine learning and image processing techniques. However, these models need fully annotated datasets for training which is challenging to acquire, especially in the medical domain. In this work, we release one of the biggest fully manually annotated datasets of nuclei in Hematoxylin and Eosin (H&E)-stained histological images, called NuInsSeg. This dataset contains 665 image patches with more than 30,000 manually segmented nuclei from 31 human and mouse organs. Moreover, for the first time, we provide additional ambiguous area masks for the entire dataset. These vague areas represent the parts of the images where precise and deterministic manual annotations are impossible, even for human experts. The dataset and detailed step-by-step instructions to generate related segmentation masks are publicly available on the respective repositories.