Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images

AttriMIL: Revisiting attention-based multiple instance learning for whole-slide pathological image classification from a perspective of instance attributes

Multiple instance learning (MIL) is a powerful approach for whole-slide pathological image (WSI) analysis, particularly suited for processing gigapixel-resolution images with slide-level labels. Recent attention-based MIL architectures have significantly advanced weakly supervised WSI classification, facilitating both clinical diagnosis and localization of disease-positive regions. However, these methods often face challenges in differentiating between instances, leading to tissue misidentification and a potential degradation in classification performance. To address these limitations, we propose AttriMIL, an attribute-aware multiple instance learning framework. By dissecting the computational flow of attention-based MIL models, we introduce a multi-branch attribute scoring mechanism that quantifies the pathological attributes of individual instances. Leveraging these quantified attributes, we further establish region-wise and slide-wise attribute constraints to dynamically model instance correlations both within and across slides during training. These constraints encourage the network to capture intrinsic spatial patterns and semantic similarities between image patches, thereby enhancing its ability to distinguish subtle tissue variations and sensitivity to challenging instances. To fully exploit the two constraints, we further develop a pathology adaptive learning technique to optimize pre-trained feature extractors, enabling the model to efficiently gather task-specific features. Extensive experiments on five public datasets demonstrate that AttriMIL consistently outperforms state-of-the-art methods across various dimensions, including bag classification accuracy, generalization ability, and disease-positive region localization. The implementation code is available at https://github.com/MedCAI/AttriMIL.

Deep Learning-Powered Whole Slide Image Analysis in Cancer Pathology

Pathology is the cornerstone of modern cancer care. With the advancement of precision oncology, the demand for histopathologic diagnosis and stratification of patients is increasing as personalized cancer therapy relies on accurate biomarker assessment. Recently, rapid development of whole slide imaging technology has enabled digitalization of traditional histologic slides at high resolution, holding promise to improve both the precision and efficiency of histopathologic evaluation. In particular, deep learning approaches, such as Convolutional Neural Network, Graph Convolutional Network, and Transformer, have shown great promise in enhancing the sensitivity and accuracy of whole slide image (WSI) analysis in cancer pathology because of their ability to handle high-dimensional and complex image data. The integration of deep learning models with WSIs enables us to explore and mine morphologic features beyond the visual perception of pathologists, which can help automate clinical diagnosis, assess histopathologic grade, predict clinical outcomes, and even discover novel morphologic biomarkers. In this review, we present a comprehensive framework for incorporating deep learning with WSIs, highlighting how deep learning-driven WSI analysis advances clinical tasks in cancer care. Furthermore, we critically discuss the opportunities and challenges of translating deep learning-based digital pathology into clinical practice, which should be considered to support personalized treatment of cancer patients.

HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification

Fine-grained classification of whole slide images (WSIs) is essential in precision oncology, enabling precise cancer diagnosis and personalized treatment strategies. The core of this task involves distinguishing subtle morphological variations within the same broad category of gigapixel-resolution images, which presents a significant challenge. While the multi-instance learning (MIL) paradigm alleviates the computational burden of WSIs, existing MIL methods often overlook hierarchical label correlations, treating fine-grained classification as a flat multi-class classification task. To overcome these limitations, we introduce a novel hierarchical multi-instance learning (HMIL) framework. By facilitating on the hierarchical alignment of inherent relationships between different hierarchy of labels at instance and bag level, our approach provides a more structured and informative learning process. Specifically, HMIL incorporates a class-wise attention mechanism that aligns hierarchical information at both the instance and bag levels. Furthermore, we introduce supervised contrastive learning to enhance the discriminative capability for fine-grained classification and a curriculum-based dynamic weighting module to adaptively balance the hierarchical feature during training. Extensive experiments on our large-scale cytology cervical cancer (CCC) dataset and two public histology datasets, BRACS and PANDA, demonstrate the state-of-the-art class-wise and overall performance of our HMIL framework. Our source code is available at https://github.com/ChengJin-git/HMIL.

Prediction of molecular subtypes for endometrial cancer based on hierarchical foundation model

MOTIVATION

Endometrial cancer is a prevalent gynecological malignancy that requires accurate identification of its molecular subtypes for effective diagnosis and treatment. Four molecular subtypes with different clinical outcomes have been identified: POLE mutation, mismatch repair deficient, p53 abnormal, and no specific molecular profile. However, determining these subtypes typically relies on expensive gene sequencing. To overcome this limitation, we propose a novel method that utilizes hematoxylin and eosin-stained whole slide images to predict endometrial cancer molecular subtypes.

RESULTS

Our approach leverages a hierarchical foundation model as a backbone, fine-tuned from the UNI computational pathology foundation model, to extract tissue embedding from different scales. We have achieved promising results through extensive experimentation on the Fudan University Shanghai Cancer Center cohort (N = 364). Our model demonstrates a macro-average AUROC of 0.879 (95% CI, 0.853-0.904) in a five-fold cross-validation. Compared to the current state-of-the-art molecular subtypes prediction for endometrial cancer, our method outperforms in terms of predictive accuracy and computational efficiency. Moreover, our method is highly reproducible, allowing for ease of implementation and widespread adoption. This study aims to address the cost and time constraints associated with traditional gene sequencing techniques. By providing a reliable and accessible alternative to gene sequencing, our method has the potential to revolutionize the field of endometrial cancer diagnosis and improve patient outcomes.

AVAILABILITY AND IMPLEMENTATION

The codes and data used for generating results in this study are available at https://github.com/HaoyuCui/hi-UNI for GitHub and https://doi.org/10.5281/zenodo.14627478 for Zenodo.

Improved DeTraC Binary Coyote Net-Based Multiple Instance Learning for Predicting Lymph Node Metastasis of Breast Cancer From Whole-Slide Pathological Images

BACKGROUND

Early detection of lymph node metastasis in breast cancer is vital for improving treatment outcomes and prognosis.

METHODS

This study introduces an Improved Decompose, Transfer, and Compose Binary Coyote Net-based Multiple Instance Learning (ImDeTraC-BCNet-MIL) method for predicting lymph node metastasis from Whole Slide Images (WSIs) using multiple instance learning. The method involves segmenting WSIs into patches using Otsu and double-dimensional clustering techniques. The developed multiple instance learning approach introduces a paradigm into computational pathology by shaping pathological data and constructing features. ImDeTraC-BCNet-MIL was utilised for feature generation during both training and testing to differentiate lymph node metastasis in WSIs.

RESULTS

The proposed model achieves the highest accuracy of 95.3% and 99.8%, precision values of 98% and 99.8%, and recall rates of 92.9% and 99.8% on the Camelyon16 and Camelyon17 datasets.

CONCLUSIONS

These findings underscore the effectiveness of ImDeTraC-BCNet-MIL in enhancing the early detection of lymph node metastasis in breast cancer.

Cross-Attention-Based Saliency Inference for Predicting Cancer Metastasis on Whole Slide Images

Although multiple instance learning (MIL) methods are widely used for automatic tumor detection on whole slide images (WSI), they suffer from the extreme class imbalance WSIs containing small tumors where the tumor may include only a few isolated cells. For early detection, it is important that MIL algorithms can identify small tumors. Existing studies have attempted to address this issue using attention-based architectures and instance selection-based methodologies but have not produced significant improvements. This paper proposes cross-attention-based salient instance inference MIL (CASiiMIL), which involves a novel saliency-informed attention mechanism to identify small tumors (e.g., breast cancer lymph node micro-metastasis) on WSIs without needing any annotations. In addition to this new attention mechanism, we introduce a negative representation learning algorithm to facilitate the learning of saliency-informed attention weights for improved sensitivity on tumor WSIs. The proposed model outperforms the state-of-the-art MIL methods on two popular tumor metastasis detection datasets. The proposed approach demonstrates great cross-center generalizability, high accuracy in classifying WSIs with small tumor lesions, and excellent interpretability attributed to the saliency-informed attention weights. We expect that the proposed method will pave the way for training algorithms for early tumor detection on large datasets where acquiring fine-grained annotations is is not practical.

Computational methods for metastasis detection in lymph nodes and characterization of the metastasis-free lymph node microarchitecture: A systematic-narrative hybrid review

Background

Histological examination of tumor draining lymph nodes (LNs) plays a vital role in cancer staging and prognostication. However, as soon as a LN is classed as metastasis-free, no further investigation will be performed and thus, potentially clinically relevant information detectable in tumor-free LNs is currently not captured.

Objective

To systematically study and critically assess methods for the analysis of digitized histological LN images described in published research.

Methods

A systematic search was conducted in several public databases up to December 2023 using relevant search terms. Studies using brightfield light microscopy images of hematoxylin and eosin or immunohistochemically stained LN tissue sections aiming to detect and/or segment LNs, their compartments or metastatic tumor using artificial intelligence (AI) were included. Dataset, AI methodology, cancer type, and study objective were compared between articles.

Results

A total of 7201 articles were collected and 73 articles remained for detailed analyses after article screening. Of the remaining articles, 86% aimed at LN metastasis identification, 8% aimed at LN compartment segmentation, and remaining focused on LN contouring. Furthermore, 78% of articles used patch classification and 22% used pixel segmentation models for analyses. Five out of six studies (83%) of metastasis-free LNs were performed on publicly unavailable datasets, making quantitative article comparison impossible.

Conclusions

Multi-scale models mimicking multiple microscopy zooms show promise for computational LN analysis. Large-scale datasets are needed to establish the clinical relevance of analyzing metastasis-free LN in detail. Further research is needed to identify clinically interpretable metrics for LN compartment characterization.

RGS5+ lymphatic endothelial cells facilitate metastasis and acquired drug resistance of breast cancer through oxidative stress-sensing mechanism

AIMS

Oxidative stress reflected by elevated reactive oxygen species (ROS) in the tumor ecosystem, is a hallmark of human cancers. The mechanisms by which oxidative stress regulate the metastatic ecosystem and resistance remain elusive. This study aimed to dissect the oxidative stress-sensing machinery during the evolvement of early dissemination and acquired drug resistance in breast cancer.

METHODS

Here, we constructed single-cell landscape of primary breast tumors and metastatic lymph nodes, and focused on RGS5+ endothelial cell subpopulation in breast cancer metastasis and resistance.

RESULTS

We reported on RGS5 as a master in endothelial cells sensing oxidative stress. RGS5+ endothelial cells facilitated tumor-endothelial adhesion and transendothelial migration of breast cancer cells. Antioxidant suppressed oxidative stress-induced RGS5 expression in endothelial cells, and prevented adhesion and transendothelial migration of cancer cells. RGS5-overexpressed HLECs displayed attenuated glycolysis and oxidative phosphorylation. Drug-resistant HLECs with RGS5 overexpression conferred acquired drug resistance of breast cancer cells. Importantly, genetic knockdown of RGS5 prevented tumor growth and lymph node metastasis.

CONCLUSIONS

Our work demonstrates that RGS5 in lymphatic endothelial cells senses oxidative stress to promote breast cancer lymph node metastasis and resistance, providing a novel insight into a potentially targetable oxidative stress-sensing machinery in breast cancer treatment.

Dual-stream multi-dependency graph neural network enables precise cancer survival analysis

Histopathology image-based survival prediction aims to provide a precise assessment of cancer prognosis and can inform personalized treatment decision-making in order to improve patient outcomes. However, existing methods cannot automatically model the complex correlations between numerous morphologically diverse patches in each whole slide image (WSI), thereby preventing them from achieving a more profound understanding and inference of the patient status. To address this, here we propose a novel deep learning framework, termed dual-stream multi-dependency graph neural network (DM-GNN), to enable precise cancer patient survival analysis. Specifically, DM-GNN is structured with the feature updating and global analysis branches to better model each WSI as two graphs based on morphological affinity and global co-activating dependencies. As these two dependencies depict each WSI from distinct but complementary perspectives, the two designed branches of DM-GNN can jointly achieve the multi-view modeling of complex correlations between the patches. Moreover, DM-GNN is also capable of boosting the utilization of dependency information during graph construction by introducing the affinity-guided attention recalibration module as the readout function. This novel module offers increased robustness against feature perturbation, thereby ensuring more reliable and stable predictions. Extensive benchmarking experiments on five TCGA datasets demonstrate that DM-GNN outperforms other state-of-the-art methods and offers interpretable prediction insights based on the morphological depiction of high-attention patches. Overall, DM-GNN represents a powerful and auxiliary tool for personalized cancer prognosis from histopathology images and has great potential to assist clinicians in making personalized treatment decisions and improving patient outcomes.

Explainable breast cancer molecular expression prediction using multi-task deep-learning based on 3D whole breast ultrasound

OBJECTIVES

To noninvasively estimate three breast cancer biomarkers, estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) and enhance performance and interpretability via multi-task deep learning.

METHODS

The study included 388 breast cancer patients who received the 3D whole breast ultrasound system (3DWBUS) examinations at Xijing Hospital between October 2020 and September 2021. Two predictive models, a single-task and a multi-task, were developed; the former predicts biomarker expression, while the latter combines tumor segmentation with biomarker prediction to enhance interpretability. Performance evaluation included individual and overall prediction metrics, and Delong's test was used for performance comparison. The models' attention regions were visualized using Grad-CAM + + technology.

RESULTS

All patients were randomly split into a training set (n = 240, 62%), a validation set (n = 60, 15%), and a test set (n = 88, 23%). In the individual evaluation of ER, PR, and HER2 expression prediction, the single-task and multi-task models achieved respective AUCs of 0.809 and 0.735 for ER, 0.688 and 0.767 for PR, and 0.626 and 0.697 for HER2, as observed in the test set. In the overall evaluation, the multi-task model demonstrated superior performance in the test set, achieving a higher macro AUC of 0.733, in contrast to 0.708 for the single-task model. The Grad-CAM + + method revealed that the multi-task model exhibited a stronger focus on diseased tissue areas, improving the interpretability of how the model worked.

CONCLUSION

Both models demonstrated impressive performance, with the multi-task model excelling in accuracy and offering improved interpretability on noninvasive 3DWBUS images using Grad-CAM + + technology.

CRITICAL RELEVANCE STATEMENT

The multi-task deep learning model exhibits effective prediction for breast cancer biomarkers, offering direct biomarker identification and improved clinical interpretability, potentially boosting the efficiency of targeted drug screening.

KEY POINTS

Tumoral biomarkers are paramount for determining breast cancer treatment. The multi-task model can improve prediction performance, and improve interpretability in clinical practice. The 3D whole breast ultrasound system-based deep learning models excelled in predicting breast cancer biomarkers.

A universal multiple instance learning framework for whole slide image analysis

BACKGROUND

The emergence of digital whole slide image (WSI) has driven the development of computational pathology. However, obtaining patch-level annotations is challenging and time-consuming due to the high resolution of WSI, which limits the applicability of fully supervised methods. We aim to address the challenges related to patch-level annotations.

METHODS

We propose a universal framework for weakly supervised WSI analysis based on Multiple Instance Learning (MIL). To achieve effective aggregation of instance features, we design a feature aggregation module from multiple dimensions by considering feature distribution, instances correlation and instance-level evaluation. First, we implement instance-level standardization layer and deep projection unit to improve the separation of instances in the feature space. Then, a self-attention mechanism is employed to explore dependencies between instances. Additionally, an instance-level pseudo-label evaluation method is introduced to enhance the available information during the weak supervision process. Finally, a bag-level classifier is used to obtain preliminary WSI classification results. To achieve even more accurate WSI label predictions, we have designed a key instance selection module that strengthens the learning of local features for instances. Combining the results from both modules leads to an improvement in WSI prediction accuracy.

RESULTS

Experiments conducted on Camelyon16, TCGA-NSCLC, SICAPv2, PANDA and classical MIL benchmark datasets demonstrate that our proposed method achieves a competitive performance compared to some recent methods, with maximum improvement of 14.6 % in terms of classification accuracy.

CONCLUSION

Our method can improve the classification accuracy of whole slide images in a weakly supervised way, and more accurately detect lesion areas.

PhiHER2: phenotype-informed weakly supervised model for HER2 status prediction from pathological images

MOTIVATION

Human epidermal growth factor receptor 2 (HER2) status identification enables physicians to assess the prognosis risk and determine the treatment schedule for patients. In clinical practice, pathological slides serve as the gold standard, offering morphological information on cellular structure and tumoral regions. Computational analysis of pathological images has the potential to discover morphological patterns associated with HER2 molecular targets and achieve precise status prediction. However, pathological images are typically equipped with high-resolution attributes, and HER2 expression in breast cancer (BC) images often manifests the intratumoral heterogeneity.

RESULTS

We present a phenotype-informed weakly supervised multiple instance learning architecture (PhiHER2) for the prediction of the HER2 status from pathological images of BC. Specifically, a hierarchical prototype clustering module is designed to identify representative phenotypes across whole slide images. These phenotype embeddings are then integrated into a cross-attention module, enhancing feature interaction and aggregation on instances. This yields a phenotype-based feature space that leverages the intratumoral morphological heterogeneity for HER2 status prediction. Extensive results demonstrate that PhiHER2 captures a better WSI-level representation by the typical phenotype guidance and significantly outperforms existing methods on real-world datasets. Additionally, interpretability analyses of both phenotypes and WSIs provide explicit insights into the heterogeneity of morphological patterns associated with molecular HER2 status.

AVAILABILITY AND IMPLEMENTATION

Our model is available at https://github.com/lyotvincent/PhiHER2.

Global attention based GNN with Bayesian collaborative learning for glomerular lesion recognition

BACKGROUND

Glomerular lesions reflect the onset and progression of renal disease. Pathological diagnoses are widely regarded as the definitive method for recognizing these lesions, as the deviations in histopathological structures closely correlate with impairments in renal function.

METHODS

Deep learning plays a crucial role in streamlining the laborious, challenging, and subjective task of recognizing glomerular lesions by pathologists. However, the current methods treat pathology images as data in regular Euclidean space, limiting their ability to efficiently represent the complex local features and global connections. In response to this challenge, this paper proposes a graph neural network (GNN) that utilizes global attention pooling (GAP) to more effectively extract high-level semantic features from glomerular images. The model incorporates Bayesian collaborative learning (BCL), enhancing node feature fine-tuning and fusion during training. In addition, this paper adds a soft classification head to mitigate the semantic ambiguity associated with a purely hard classification.

RESULTS

This paper conducted extensive experiments on four glomerular datasets, comprising a total of 491 whole slide images (WSIs) and 9030 images. The results demonstrate that the proposed model achieves impressive F1 scores of 81.37%, 90.12%, 87.72%, and 98.68% on four private datasets for glomerular lesion recognition. These scores surpass the performance of the other models used for comparison. Furthermore, this paper employed a publicly available BReAst Carcinoma Subtyping (BRACS) dataset with an 85.61% F1 score to further prove the superiority of the proposed model.

CONCLUSION

The proposed model not only facilitates precise recognition of glomerular lesions but also serves as a potent tool for diagnosing kidney diseases effectively. Furthermore, the framework and training methodology of the GNN can be adeptly applied to address various pathology image classification challenges.

Towards a general-purpose foundation model for computational pathology

Quantitative evaluation of tissue images is crucial for computational pathology (CPath) tasks, requiring the objective characterization of histopathological entities from whole-slide images (WSIs). The high resolution of WSIs and the variability of morphological features present significant challenges, complicating the large-scale annotation of data for high-performance applications. To address this challenge, current efforts have proposed the use of pretrained image encoders through transfer learning from natural image datasets or self-supervised learning on publicly available histopathology datasets, but have not been extensively developed and evaluated across diverse tissue types at scale. We introduce UNI, a general-purpose self-supervised model for pathology, pretrained using more than 100 million images from over 100,000 diagnostic H&E-stained WSIs (>77 TB of data) across 20 major tissue types. The model was evaluated on 34 representative CPath tasks of varying diagnostic difficulty. In addition to outperforming previous state-of-the-art models, we demonstrate new modeling capabilities in CPath such as resolution-agnostic tissue classification, slide classification using few-shot class prototypes, and disease subtyping generalization in classifying up to 108 cancer types in the OncoTree classification system. UNI advances unsupervised representation learning at scale in CPath in terms of both pretraining data and downstream evaluation, enabling data-efficient artificial intelligence models that can generalize and transfer to a wide range of diagnostically challenging tasks and clinical workflows in anatomic pathology.

Construction and validation of a nomogram for predicting the prognosis of patients with lymph node-positive invasive micropapillary carcinoma of the breast: based on SEER database and external validation cohort

Background

Invasive micropapillary carcinoma (IMPC) of the breast is a rare subtype of breast cancer with high incidence of aggressive clinical behavior, lymph node metastasis (LNM) and poor prognosis. In the present study, using the Surveillance, Epidemiology, and End Results (SEER) database, we analyzed the clinicopathological characteristics and prognostic factors of IMPC with LNM, and constructed a prognostic nomogram.

Methods

We retrospectively analyzed data for 487 breast IMPC patients with LNM in the SEER database from January 2010 to December 2015, and randomly divided these patients into a training cohort (70%) and an internal validation cohort (30%) for the construction and internal validation of the nomogram, respectively. In addition, 248 patients diagnosed with IMPC and LNM at the Fourth Hospital of Hebei Medical University from January 2010 to December 2019 were collected as an external validation cohort. Lasso regression, along with Cox regression, was used to screen risk factors. Further more, the discrimination, calibration, and clinical utility of the nomogram were assessed based on the consistency index (C-index), time-dependent receiver operating characteristic (ROC), calibration curve, and decision curve analysis (DCA).

Results

In summary, we identified six variables including molecular subtype of breast cancer, first malignant primary indicator, tumor grade, AJCC stage, radiotherapy and chemotherapy were independent prognostic factors in predicting the prognosis of IMPC patients with LNM (P < 0.05). Based on these factors, a nomogram was constructed for predicting 3- and 5-year overall survival (OS) of patients. The nomogram achieved a C-index of 0.789 (95%CI: 0.759-0.819) in the training cohort, 0.775 (95%CI: 0.731-0.819) in the internal validation cohort, and 0.788 (95%CI: 0.756-0.820) in the external validation cohort. According to the calculated patient risk score, the patients were divided into a high-risk group and a low-risk group, which showed a significant difference in the survival prognosis of the two groups (P<0.0001). The time-dependent ROC curves, calibration curves and DCA curves proved the superiority of the nomogram.

Conclusions

We have successfully constructed a nomogram that could predict 3- and 5-year OS of IMPC patients with LNM and may assist clinicians in decision-making and personalized treatment planning.