Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification

Optimized multiple instance learning for brain tumor classification using weakly supervised contrastive learning

Brain tumors have a great impact on patients' quality of life and accurate histopathological classification of brain tumors is crucial for patients' prognosis. Multi-instance learning (MIL) has become the mainstream method for analyzing whole-slide images (WSIs). However, current MIL-based methods face several issues, including significant redundancy in the input and feature space, insufficient modeling of spatial relations between patches and inadequate representation capability of the feature extractor. To solve these limitations, we propose a new multi-instance learning with weakly supervised contrastive learning for brain tumor classification. Our framework consists of two parts: a cross-detection MIL aggregator (CDMIL) for brain tumor classification and a contrastive learning model based on pseudo-labels (PSCL) for optimizing feature encoder. The CDMIL consists of three modules: an internal patch anchoring module (IPAM), a local structural learning module (LSLM) and a cross-detection module (CDM). Specifically, IPAM utilizes probability distribution to generate representations of anchor samples, while LSLM extracts representations of local structural information between anchor samples. These two representations are effectively fused in CDM. Additionally, we propose a bag-level contrastive loss to interact with different subtypes in the feature space. PSCL uses the samples and pseudo-labels anchored by IPAM to optimize the performance of the feature encoder, which extracts a better feature representation to train CDMIL. We performed benchmark tests on a self-collected dataset and a publicly available dataset. The experiments show that our method has better performance than several existing state-of-the-art methods.

Multiple Instance Learning-Based Prediction of Blood-Brain Barrier Opening Outcomes Induced by Focused Ultrasound

OBJECTIVE

Targeted blood-brain barrier (BBB) opening using focused ultrasound (FUS) and micro/nanobubbles is a promising method for brain drug delivery. This study aims to explore the feasibility of multiple instance learning (MIL) in accurate and fast prediction of FUS BBB opening outcomes.

METHODS

FUS BBB opening experiments are conducted on 52 mice with the infusion of SonoVue microbubbles or custom-made nanobubbles. Acoustic signals collected during the experiments are transformed into frequency domain and used as the dataset. We propose a Simple Transformer-based model for BBB Opening Prediction (SimTBOP). By leveraging the self-attention mechanism, our model considers the contextual relationships between signals from different pulses in a treatment and aggregates this information to predict the BBB opening outcomes. Multiple preprocessing methods are applied to evaluate the performance of the proposed model under various conditions. Additionally, a visualization technique is employed to explain and interpret the model.

RESULTS

The proposed model achieves excellent prediction performance with an accuracy of 96.7%. Excluding absolute intensity information and retaining baseline noise did not affect the model's performance or interpretability. The proposed model trained on SonoVue data generalizes well to nanobubble data and vice versa. Visualization results indicate that the proposed model focuses on pulses with significant signals near the ultra-harmonic frequency.

CONCLUSION

We demonstrate the feasibility of MIL in FUS BBB opening prediction. The proposed Transformer-based model exhibits outstanding performance, interpretability, and cross-agent generalization capability, providing a novel approach for FUS BBB opening prediction with clinical translation potential.

The application of multi-instance learning based on feature reconstruction and cross-mixing in the Gleason grading of prostate cancer from whole-slide images

Background

Prostate cancer is a common malignancy in men, requiring accurate diagnosis and prognosis. The Gleason grading system remains the preferred method of evaluation and is critical to risk stratification and informing treatment strategies. However, analyzing whole-slide image (WSI) is significantly challenging due to high pixel density, tumor heterogeneity, and the difficulty in acquiring precise annotated data. This study developed a weakly supervised multiple instance learning (MIL)-based method for Gleason grading of prostate cancer pathology images, aiming to enhance tumor classification performance and provide more reliable support for clinical risk assessment and treatment strategies.

Methods

This study developed a novel feature reconstruction and cross-mixing-based MIL (FRCM-MIL) method to enhance the accuracy of prostate cancer from WSIs. This method includes a spatial feature reconstruction module based on wavelet transform (SFRM-WT), which combines frequency domain information to extract more diverse features. A cross-attention module (CAM) was included to enhance feature interaction and fusion. Additionally, a confidence query aggregation module (CQAM) was used to consolidate input features and create confidence-enhanced outputs.

Results

The proposed method achieved an accuracy of 81.75% and an area under the curve (AUC) of 94.41% on the Peking Union Medical College Hospital (PUMCH) dataset, along with an accuracy of 67.24% and an AUC of 91.69% on the Prostate Cancer Grade Assessment Challenge (PANDA) dataset, outperforming existing state-of-the-art approaches.

Conclusions

The FRCM-MIL model performs outstandingly in the Gleason grading task for prostate cancer WSIs, effectively distinguishing between different grades. This model has the potential to assist clinicians in formulating personalized chemotherapy and radiotherapy plans, ultimately improving treatment outcomes and demonstrating significant clinical value.

Pan-Cancer Histopathology WSI Pre-Training With Position-Aware Masked Autoencoder

Large-scale pre-training models have promoted the development of histopathology image analysis. However, existing self-supervised methods for histopathology images primarily focus on learning patch features, while there is a notable gap in the availability of pre-training models specifically designed for WSI-level feature learning. In this paper, we propose a novel self-supervised learning framework for pan-cancer WSI-level representation pre-training with the designed position-aware masked autoencoder (PAMA). Meanwhile, we propose the position-aware cross-attention (PACA) module with a kernel reorientation (KRO) strategy and an anchor dropout (AD) mechanism. The KRO strategy can capture the complete semantic structure and eliminate ambiguity in WSIs, and the AD contributes to enhancing the robustness and generalization of the model. We evaluated our method on 7 large-scale datasets from multiple organs for pan-cancer classification tasks. The results have demonstrated the effectiveness and generalization of PAMA in discriminative WSI representation learning and pan-cancer WSI pre-training. The proposed method was also compared with 8 WSI analysis methods. The experimental results have indicated that our proposed PAMA is superior to the state-of-the-art methods. The code and checkpoints are available at https://github.com/WkEEn/PAMA.

M4: Multi-proxy multi-gate mixture of experts network for multiple instance learning in histopathology image analysis

Multiple instance learning (MIL) has been successfully applied for whole slide images (WSIs) analysis in computational pathology, enabling a wide range of prediction tasks from tumor subtyping to inferring genetic mutations and multi-omics biomarkers. However, existing MIL methods predominantly focus on single-task learning, resulting in not only overall low efficiency but also the overlook of inter-task relatedness. To address these issues, we proposed an adapted architecture of Multi-gate Mixture-of-experts with Multi-proxy for Multiple instance learning (M4), and applied this framework for simultaneous prediction of multiple genetic mutations from WSIs. The proposed M4 model has two main innovations: (1) adopting a multi-gate mixture-of-experts strategy for multiple genetic mutation simultaneous prediction on a single WSI; (2) introducing a multi-proxy CNN construction on the expert and gate networks to effectively and efficiently capture patch-patch interactions from WSI. Our model achieved significant improvements across five tested TCGA datasets in comparison to current state-of-the-art single-task methods. The code is available at: https://github.com/Bigyehahaha/M4.

Reimagining cancer tissue classification: a multi-scale framework based on multi-instance learning for whole slide image classification

In cancer pathology diagnosis, analyzing Whole Slide Images (WSI) encounters challenges like invalid data, varying tissue features at different magnifications, and numerous hard samples. Multiple Instance Learning (MIL) is a powerful tool for addressing weakly supervised classification in WSI-based pathology diagnosis. However, existing MIL frameworks cannot simultaneously tackle these issues. To address these challenges, we propose an integrated recognition framework comprising three complementary components: a preprocessing selection method, an Efficient Feature Pyramid Network (EFPN) model for multi-instance learning, and a Similarity Focal Loss. The preprocessing selection method accurately identifies and selects representative image patches, effectively reducing invalid data interference and enhancing subsequent model training efficiency. The EFPN model, inspired by pathologists' diagnostic processes, captures different tissue features in WSI images by constructing a multi-scale feature pyramid, enhancing the model's ability to recognize tumor tissue features. Additionally, the Similarity Focal Loss further improves the model's discriminative power and generalization performance by focusing on hard samples and emphasizing classification boundary information. The test accuracy for binary tumor classification on the CAMELYON16 and two private datasets reached 93.58%, 84.74%, and 99.91%, respectively, all of which outperform existing techniques.

A deep learning approach to perform defect classification of freeze-dried product

Cosmetic inspection of freeze-dried products is an important part of the post-manufacturing quality control process. Traditionally done by human visual inspection, this method poses typical challenges and shortcomings that can be addressed with innovative techniques. While many cosmetic defects can occur, some are considered more critical than others as they can be harmful to the patient or affect the drug's efficacy. With the rise of artificial intelligence and computer vision technology, faster and more reproducible quality control is possible, allowing real-time monitoring on a continuous manufacturing line. In this study, several continuously freeze-dried samples were prepared using formulations and process settings that lead deliberately to specific defects faced in freeze-drying as well as defect-free samples. Two approaches (i.e. patch-based approach and multi-label classification) capable of handling high-resolution images based on Convolutional Neural Networks were developed and compared to select the optimal one. Additional visualization techniques were used to enhance model understanding further. The best approach achieved perfect precision and recall on critical defects, with a prediction time of less than 50 ms to make a decision on the acceptance or rejection of vials generated.

Context aware machine learning techniques for brain tumor classification and detection - A review

Background

Machine learning has tremendous potential in acute medical care, particularly in the field of precise medical diagnosis, prediction, and classification of brain tumors. Malignant gliomas, due to their aggressive growth and dismal prognosis, stand out among various brain tumor types. Recent advancements in understanding the genetic abnormalities that underlie these tumors have shed light on their histo-pathological and biological characteristics, which support in better classification and prognosis.

Objectives

This review aims to predict gene alterations and establish structured correlations among various tumor types, extending the prediction of genetic mutations and structures using the latest machine learning techniques. Specifically, it focuses on multi-modalities of Magnetic Resonance Imaging (MRI) and histopathology, utilizing Convolutional Neural Networks (CNN) for image processing and analysis.

Methods

The review encompasses the most recent developments in MRI, and histology image processing methods across multiple tumor classes, including Glioma, Meningioma, Pituitary, Oligodendroglioma, and Astrocytoma. It identifies challenges in tumor classification, segmentation, datasets, and modalities, employing various neural network architectures. A competitive analysis assesses the performance of CNN. Furthermore it also implies K-MEANS clustering to predict Genetic structure, Genes Clusters prediction and Molecular Alteration of various types and grades of tumors e.g. Glioma, Meningioma, Pituitary, Oligodendroglioma, and Astrocytoma.

Results

CNN and KNN structures, with their ability to extract highlights in image-based information, prove effective in tumor classification and segmentation, surmounting challenges in image analysis. Competitive analysis reveals that CNN and outperform others algorithms on publicly available datasets, suggesting their potential for precise tumor diagnosis and treatment planning.

Conclusion

Machine learning, especially through CNN and SVM algorithms, demonstrates significant potential in the accurate diagnosis and classification of brain tumors based on imaging and histo-pathological data. Further advancements in this area hold promise for improving the accuracy and efficiency of intra-operative tumor diagnosis and treatment.

MacNet: a mobile attention classification network combining convolutional neural network and transformer for the differentiation of cervical cancer

Background

Cervical cancer remains a critical global health issue, responsible for over 600,000 new cases and 300,000 deaths annually. Pathological imaging of cervical cancer is a crucial diagnostic tool. However, distinguishing specific areas of cellular differentiation remains challenging because of the lack of clear boundaries between cells at various stages of differentiation. To address the limitations of conventional clinical and deep learning (DL) methods, we developed a mobile attention classification network (MacNet) with multiscale features, aiming to increase the accuracy of differentiation classification and quantitatively analyze cervical cancer cell differentiation.

Methods

We investigated the application of MacNet for classifying non-background images into 3 stages of cervical cancer differentiation. The feature maps are processed through the Mobile Convolution Neural Network with Mobile Attention (MCMA) module, which integrates mobile convolutional blocks and mobile attention blocks. MacNet harnesses the benefits of the image pyramid structure and self-attention mechanism, enabling multiscale feature extraction and emulation of clinical pathologist analysis. The final prediction is generated by the adaptive fusion module, which aggregates features into a unified output.

Results

Comparative evaluations demonstrated that MacNet outperforms existing models. The proposed method achieved the best classification accuracy of 92.34% among all 7 DL-based models. Specifically, the result achieved by MacNet was 2.62% greater than that of Inception Version 3, 7.9% greater than that of vision transformer, 8.08% greater than that of the visual geometry group network, 3.21% greater than that of Densely Connected Convolutional Network, 2.85% greater than that of shifted window transformer (Swin transformer), 5.4% greater than that of Cross Stage Partial DarkNet, and 5.41% greater than that of Residual Neural Network. At the same time, MacNet also achieved superior results in recall, precision, and F1 score.

Conclusions

We have proposed a lightweight neural network method that innovatively combines attention mechanisms with convolutional neural networks (CNNs) to efficiently utilize multiscale information from histopathological images. This integration enables the precise quantitative display of different stages of differentiation in cervical cancer. By doing so, our method not only enhances diagnostic accuracy but also provides clinicians with a more effective tool for faster and more reliable diagnosis, representing a significant advancement in the field of pathological imaging.

Evaluation of a Task-Specific Self-Supervised Learning Framework in Digital Pathology Relative to Transfer Learning Approaches and Existing Foundation Models

An integral stage in typical digital pathology workflows involves deriving specific features from tiles extracted from a tessellated whole-slide image. Notably, various computer vision neural network architectures, particularly the ImageNet pretrained, have been extensively used in this domain. This study critically analyzes multiple strategies for encoding tiles to understand the extent of transfer learning and identify the most effective approach. The study categorizes neural network performance into 3 weight initialization methods: random, ImageNet-based, and self-supervised learning. Additionally, we propose a framework based on task-specific self-supervised learning, which introduces a shallow feature extraction method, employing a spatial-channel attention block to glean distinctive features optimized for histopathology intricacies. Across 2 different downstream classification tasks (patch classification and weakly supervised whole-slide image classification) with diverse classification data sets, including colorectal cancer histology, Patch Camelyon, prostate cancer detection, The Cancer Genome Atlas, and CIFAR-10, our task-specific self-supervised encoding approach consistently outperforms other convolutional neural network-based encoders. The better performances highlight the potential of task-specific attention-based self-supervised training in tailoring feature extraction for histopathology, indicating a shift from using pretrained models originating outside the histopathology domain. Our study supports the idea that task-specific self-supervised learning allows domain-specific feature extraction, encouraging a more focused analysis.

Aligning knowledge concepts to whole slide images for precise histopathology image analysis

Due to the large size and lack of fine-grained annotation, Whole Slide Images (WSIs) analysis is commonly approached as a Multiple Instance Learning (MIL) problem. However, previous studies only learn from training data, posing a stark contrast to how human clinicians teach each other and reason about histopathologic entities and factors. Here, we present a novel knowledge concept-based MIL framework, named ConcepPath, to fill this gap. Specifically, ConcepPath utilizes GPT-4 to induce reliable disease-specific human expert concepts from medical literature and incorporate them with a group of purely learnable concepts to extract complementary knowledge from training data. In ConcepPath, WSIs are aligned to these linguistic knowledge concepts by utilizing the pathology vision-language model as the basic building component. In the application of lung cancer subtyping, breast cancer HER2 scoring, and gastric cancer immunotherapy-sensitive subtyping tasks, ConcepPath significantly outperformed previous SOTA methods, which lacked the guidance of human expert knowledge.

Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma

In this study, we present a deep-learning-based multimodal classification method for lymphoma diagnosis in digital pathology, which utilizes a whole slide image (WSI) as the primary image data and flow cytometry (FCM) data as auxiliary information. In pathological diagnosis of malignant lymphoma, FCM serves as valuable auxiliary information during the diagnosis process, offering useful insights into predicting the major class (superclass) of subtypes. By incorporating both images and FCM data into the classification process, we can develop a method that mimics the diagnostic process of pathologists, enhancing the explainability. In order to incorporate the hierarchical structure between superclasses and their subclasses, the proposed method utilizes a network structure that effectively combines the mixture of experts (MoE) and multiple instance learning (MIL) techniques, where MIL is widely recognized for its effectiveness in handling WSIs in digital pathology. The MoE network in the proposed method consists of a gating network for superclass classification and multiple expert networks for (sub)class classification, specialized for each superclass. To evaluate the effectiveness of our method, we conducted experiments involving a six-class classification task using 600 lymphoma cases. The proposed method achieved a classification accuracy of 72.3%, surpassing the 69.5% obtained through the straightforward combination of FCM and images, as well as the 70.2% achieved by the method using only images. Moreover, the combination of multiple weights in the MoE and MIL allows for the visualization of specific cellular and tumor regions, resulting in a highly explanatory model that cannot be attained with conventional methods. It is anticipated that by targeting a larger number of classes and increasing the number of expert networks, the proposed method could be effectively applied to the real problem of lymphoma diagnosis.

A Deep Learning Approach for the Identification of the Molecular Subtypes of Pancreatic Ductal Adenocarcinoma Based on Whole Slide Pathology Images

Delayed diagnosis and treatment resistance result in high pancreatic ductal adenocarcinoma (PDAC) mortality rates. Identifying molecular subtypes can improve treatment, but current methods are costly and time-consuming. In this study, deep learning models were used to identify histologic features that classify PDAC molecular subtypes based on routine hematoxylin-eosin-stained histopathologic slides. A total of 97 histopathology slides associated with resectable PDAC from The Cancer Genome Atlas project were used to train a deep learning model and test the performance on 44 needle biopsy material (110 slides) from a local annotated patient cohort. The model achieved balanced accuracy of 96.19% and 83.03% in identifying the classical and basal subtypes of PDAC in The Cancer Genome Atlas and the local cohort, respectively. This study provides a promising method to cost-effectively and rapidly classify PDAC molecular subtypes based on routine hematoxylin-eosin-stained slides, potentially leading to more effective clinical management of this disease.

WISE: Efficient WSI selection for active learning in histopathology

Deep neural network (DNN) models have been applied to a wide variety of medical image analysis tasks, often with the successful performance outcomes that match those of medical doctors. However, given that even minor errors in a model can impact patients' life, it is critical that these models are continuously improved. Hence, active learning (AL) has garnered attention as an effective and sustainable strategy for enhancing DNN models for the medical domain. Extant AL research in histopathology has primarily focused on patch datasets derived from whole-slide images (WSIs), a standard form of cancer diagnostic images obtained from a high-resolution scanner. However, this approach has failed to address the selection of WSIs, which can impede the performance improvement of deep learning models and increase the number of WSIs needed to achieve the target performance. This study introduces a WSI-level AL method, termed WSI-informative selection (WISE). WISE is designed to select informative WSIs using a newly formulated WSI-level class distance metric. This method aims to identify diverse and uncertain cases of WSIs, thereby contributing to model performance enhancement. WISE demonstrates state-of-the-art performance across the Colon and Stomach datasets, collected in the real world, as well as the public DigestPath dataset, significantly reducing the required number of WSIs by more than threefold compared to the one-pool dataset setting, which has been dominantly used in the field.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Learnable Context in Multiple Instance Learning for Whole Slide Image Classification and Segmentation

Multiple instance learning (MIL) has become a cornerstone in whole slide image (WSI) analysis. In this paradigm, a WSI is conceptualized as a bag of instances. Instance features are extracted by a feature extractor, and then a feature aggregator fuses these instance features into a bag representation. In this paper, we advocate that both feature extraction and aggregation can be enhanced by considering the context or correlation between instances. We learn contextual features between instances, and then fuse contextual features with instance features to enhance instance representations. For feature aggregation, we observe performance instability particularly when disease-positive instances are only a minor fraction of the WSI. We introduce a self-attention mechanism to discover correlation among instances and foster more effective bag representations. Through comprehensive testing, we have demonstrated that the proposed method outperforms existing WSI classification methods by 1 to 4% classification accuracy, based on the Camelyon16 and the TCGA-NSCLC datasets. The proposed method also outperforms the most recent weakly supervised WSI segmentation method by 0.6 in terms of the Dice coefficient, based on the Camelyon16 dataset.

Rethinking Multiple Instance Learning for Whole Slide Image Classification: A Bag-Level Classifier is a Good Instance-Level Teacher

Multiple Instance Learning (MIL) has demonstrated promise in Whole Slide Image (WSI) classification. However, a major challenge persists due to the high computational cost associated with processing these gigapixel images. Existing methods generally adopt a two-stage approach, comprising a non-learnable feature embedding stage and a classifier training stage. Though it can greatly reduce memory consumption by using a fixed feature embedder pre-trained on other domains, such a scheme also results in a disparity between the two stages, leading to suboptimal classification accuracy. To address this issue, we propose that a bag-level classifier can be a good instance-level teacher. Based on this idea, we design Iteratively Coupled Multiple Instance Learning (ICMIL) to couple the embedder and the bag classifier at a low cost. ICMIL initially fixes the patch embedder to train the bag classifier, followed by fixing the bag classifier to fine-tune the patch embedder. The refined embedder can then generate better representations in return, leading to a more accurate classifier for the next iteration. To realize more flexible and more effective embedder fine-tuning, we also introduce a teacher-student framework to efficiently distill the category knowledge in the bag classifier to help the instance-level embedder fine-tuning. Intensive experiments were conducted on four distinct datasets to validate the effectiveness of ICMIL. The experimental results consistently demonstrated that our method significantly improves the performance of existing MIL backbones, achieving state-of-the-art results. The code and the organized datasets can be accessed by: https://github.com/Dootmaan/ICMIL/tree/confidence-based.

Cross-modality image translation of 3 Tesla Magnetic Resonance Imaging to 7 Tesla using Generative Adversarial Networks

The rapid advancements in magnetic resonance imaging (MRI) technology have precipitated a new paradigm wherein cross-modality data translation across diverse imaging platforms, field strengths, and different sites is increasingly challenging. This issue is particularly accentuated when transitioning from 3 Tesla (3T) to 7 Tesla (7T) MRI systems. This study proposes a novel solution to these challenges using generative adversarial networks (GANs)-specifically, the CycleGAN architecture-to create synthetic 7T images from 3T data. Employing a dataset of 1112 and 490 unpaired 3T and 7T MR images, respectively, we trained a 2-dimensional (2D) CycleGAN model, evaluating its performance on a paired dataset of 22 participants scanned at 3T and 7T. Independent testing on 22 distinct participants affirmed the model's proficiency in accurately predicting various tissue types, encompassing cerebral spinal fluid, gray matter, and white matter. Our approach provides a reliable and efficient methodology for synthesizing 7T images, achieving a median Dice of 6.82%,7,63%, and 4.85% for Cerebral Spinal Fluid (CSF), Gray Matter (GM), and White Matter (WM), respectively, in the testing dataset, thereby significantly aiding in harmonizing heterogeneous datasets. Furthermore, it delineates the potential of GANs in amplifying the contrast-to-noise ratio (CNR) from 3T, potentially enhancing the diagnostic capability of the images. While acknowledging the risk of model overfitting, our research underscores a promising progression towards harnessing the benefits of 7T MR systems in research investigations while preserving compatibility with existent 3T MR data. This work was previously presented at the ISMRM 2021 conference (Diniz, Helmet, Santini, Aizenstein, & Ibrahim, 2021).

Predicting peritumoral glioblastoma infiltration and subsequent recurrence using deep-learning-based analysis of multi-parametric magnetic resonance imaging

Purpose

Glioblastoma (GBM) is the most common and aggressive primary adult brain tumor. The standard treatment approach is surgical resection to target the enhancing tumor mass, followed by adjuvant chemoradiotherapy. However, malignant cells often extend beyond the enhancing tumor boundaries and infiltrate the peritumoral edema. Traditional supervised machine learning techniques hold potential in predicting tumor infiltration extent but are hindered by the extensive resources needed to generate expertly delineated regions of interest (ROIs) for training models on tissue most and least likely to be infiltrated.

Approach

We developed a method combining expert knowledge and training-based data augmentation to automatically generate numerous training examples, enhancing the accuracy of our model for predicting tumor infiltration through predictive maps. Such maps can be used for targeted supra-total surgical resection and other therapies that might benefit from intensive yet well-targeted treatment of infiltrated tissue. We apply our method to preoperative multi-parametric magnetic resonance imaging (mpMRI) scans from a subset of 229 patients of a multi-institutional consortium (Radiomics Signatures for Precision Diagnostics) and test the model on subsequent scans with pathology-proven recurrence.

Results

Leave-one-site-out cross-validation was used to train and evaluate the tumor infiltration prediction model using initial pre-surgical scans, comparing the generated prediction maps with follow-up mpMRI scans confirming recurrence through post-resection tissue analysis. Performance was measured by voxel-wised odds ratios (ORs) across six institutions: University of Pennsylvania (OR: 9.97), Ohio State University (OR: 14.03), Case Western Reserve University (OR: 8.13), New York University (OR: 16.43), Thomas Jefferson University (OR: 8.22), and Rio Hortega (OR: 19.48).

Conclusions

The proposed model demonstrates that mpMRI analysis using deep learning can predict infiltration in the peri-tumoral brain region for GBM patients without needing to train a model using expert ROI drawings. Results for each institution demonstrate the model's generalizability and reproducibility.

A study of criteria for grading follicular lymphoma using a cell type classifier from pathology images based on complementary-label learning

We propose a criterion for grading follicular lymphoma that is consistent with the intuitive evaluation, which is conducted by experienced pathologists. A criterion for grading follicular lymphoma is defined by the World Health Organization (WHO) based on the number of centroblasts and centrocytes within the field of view. However, the WHO criterion is not often used in clinical practice because it is impractical for pathologists to visually identify the cell type of each cell and count the number of centroblasts and centrocytes. Hence, based on the widespread use of digital pathology, we make it practical to identify and count the cell type by using image processing and then construct a criterion for grading based on the number of cells. Here, the problem is that labeling the cell type is not easy even for experienced pathologists. To alleviate this problem, we build a new dataset for cell type classification, which contains the pathologists' confusion records during labeling, and we construct the cell type classifier using complementary-label learning from this dataset. Then we propose a criterion based on the composition ratio of cell types that is consistent with the pathologists' grading. Our experiments demonstrate that the classifier can accurately identify cell types and the proposed criterion is more consistent with the pathologists' grading than the current WHO criterion.

Neighborhood attention transformer multiple instance learning for whole slide image classification

Introduction

Pathologists rely on whole slide images (WSIs) to diagnose cancer by identifying tumor cells and subtypes. Deep learning models, particularly weakly supervised ones, classify WSIs using image tiles but may overlook false positives and negatives due to the heterogeneous nature of tumors. Both cancerous and healthy cells can proliferate in patterns that extend beyond individual tiles, leading to errors at the tile level that result in inaccurate tumor-level classifications.

Methods

To address this limitation, we introduce NATMIL (Neighborhood Attention Transformer Multiple Instance Learning), which utilizes the Neighborhood Attention Transformer to incorporate contextual dependencies among WSI tiles. NATMIL enhances multiple instance learning by integrating a broader tissue context into the model. Our approach enhances the accuracy of tumor classification by considering the broader tissue context, thus reducing errors associated with isolated tile analysis.

Results

We conducted a quantitative analysis to evaluate NATMIL's performance against other weakly supervised algorithms. When applied to subtyping non-small cell lung cancer (NSCLC) and lymph node (LN) tumors, NATMIL demonstrated superior accuracy. Specifically, NATMIL achieved accuracy values of 89.6% on the Camelyon dataset and 88.1% on the TCGA-LUSC dataset, outperforming existing methods. These results underscore NATMIL's potential as a robust tool for improving the precision of cancer diagnosis using WSIs.

Discussion

Our findings demonstrate that NATMIL significantly improves tumor classification accuracy by reducing errors associated with isolated tile analysis. The integration of contextual dependencies enhances the precision of cancer diagnosis using WSIs, highlighting NATMILs´ potential as a robust tool in pathology.

PathEX: Make good choice for whole slide image extraction

BACKGROUND

The tile-based approach has been widely used for slide-level predictions in whole slide image (WSI) analysis. However, the irregular shapes and variable dimensions of tumor regions pose challenges for the process. To address this issue, we proposed PathEX, a framework that integrates intersection over tile (IoT) and background over tile (BoT) algorithms to extract tile images around boundaries of annotated regions while excluding the blank tile images within these regions.

METHODS

We developed PathEX, which incorporated IoT and BoT into tile extraction, for training a classification model in CAM (239 WSIs) and PAIP (40 WSIs) datasets. By adjusting the IoT and BoT parameters, we generated eight training sets and corresponding models for each dataset. The performance of PathEX was assessed on the testing set comprising 13,076 tile images from 48 WSIs of CAM dataset and 6,391 tile images from 10 WSIs of PAIP dataset.

RESULTS

PathEX could extract tile images around boundaries of annotated region differently by adjusting the IoT parameter, while exclusion of blank tile images within annotated regions achieved by setting the BoT parameter. As adjusting IoT from 0.1 to 1.0, and 1-BoT from 0.0 to 0.5, we got 8 train sets. Experimentation revealed that set C demonstrates potential as the most optimal candidate. Nevertheless, a combination of IoT values ranging from 0.2 to 0.5 and 1-BoT values ranging from 0.2 to 0.5 also yielded favorable outcomes.

CONCLUSIONS

In this study, we proposed PathEX, a framework that integrates IoT and BoT algorithms for tile image extraction at the boundaries of annotated regions while excluding blank tiles within these regions. Researchers can conveniently set the thresholds for IoT and BoT to facilitate tile image extraction in their own studies. The insights gained from this research provide valuable guidance for tile image extraction in digital pathology applications.

Pathology diagnosis of intraoperative frozen thyroid lesions assisted by deep learning

BACKGROUND

Thyroid cancer is a common thyroid malignancy. The majority of thyroid lesion needs intraoperative frozen pathology diagnosis, which provides important information for precision operation. As digital whole slide images (WSIs) develop, deep learning methods for histopathological classification of the thyroid gland (paraffin sections) have achieved outstanding results. Our current study is to clarify whether deep learning assists pathology diagnosis for intraoperative frozen thyroid lesions or not.

METHODS

We propose an artificial intelligence-assisted diagnostic system for frozen thyroid lesions that applies prior knowledge in tandem with a dichotomous judgment of whether the lesion is cancerous or not and a quadratic judgment of the type of cancerous lesion to categorize the frozen thyroid lesions into five categories: papillary thyroid carcinoma, medullary thyroid carcinoma, anaplastic thyroid carcinoma, follicular thyroid tumor, and non-cancerous lesion. We obtained 4409 frozen digital pathology sections (WSI) of thyroid from the First Affiliated Hospital of Sun Yat-sen University (SYSUFH) to train and test the model, and the performance was validated by a six-fold cross validation, 101 papillary microcarcinoma sections of thyroid were used to validate the system's sensitivity, and 1388 WSIs of thyroid were used for the evaluation of the external dataset. The deep learning models were compared in terms of several metrics such as accuracy, F1 score, recall, precision and AUC (Area Under Curve).

RESULTS

We developed the first deep learning-based frozen thyroid diagnostic classifier for histopathological WSI classification of papillary carcinoma, medullary carcinoma, follicular tumor, anaplastic carcinoma, and non-carcinoma lesion. On test slides, the system had an accuracy of 0.9459, a precision of 0.9475, and an AUC of 0.9955. In the papillary carcinoma test slides, the system was able to accurately predict even lesions as small as 2 mm in diameter. Tested with the acceleration component, the cut processing can be performed in 346.12 s and the visual inference prediction results can be obtained in 98.61 s, thus meeting the time requirements for intraoperative diagnosis. Our study employs a deep learning approach for high-precision classification of intraoperative frozen thyroid lesion distribution in the clinical setting, which has potential clinical implications for assisting pathologists and precision surgery of thyroid lesions.

Automated detection of Kaposi sarcoma-associated herpesvirus infected cells in immunohistochemical images of skin biopsies

Immunohistochemical (IHC) staining for the antigen of Kaposi sarcoma-associated herpesvirus (KSHV), latency-associated nuclear antigen (LANA), is helpful in diagnosing Kaposi sarcoma (KS). A challenge, however, lies in distinguishing anti-LANA-positive cells from morphologically similar brown counterparts. In this work, we demonstrate a framework for automated localization and quantification of LANA positivity in whole slide images (WSI) of skin biopsies, leveraging weakly supervised multiple instance learning (MIL) while reducing false positive predictions by introducing a novel morphology-based slide aggregation method. Our framework generates interpretable heatmaps, offering insights into precise anti-LANA-positive cell localization within WSIs and a quantitative value for the percentage of positive tiles, which may assist with histological subtyping. We trained and tested our framework with an anti-LANA-stained KS pathology dataset prepared by pathologists in the United States from skin biopsies of KS-suspected patients investigated in Uganda. We achieved an area under the receiver operating characteristic curve (AUC) of 0.99 with a sensitivity and specificity of 98.15% and 96.00% in predicting anti-LANA-positive WSIs in a test dataset. We believe that the framework can provide promise for automated detection of LANA in skin biopsies, which may be especially impactful in resource-limited areas that lack trained pathologists.

Machine learning quantification of Amyloid-β deposits in the temporal lobe of 131 brain bank cases

Accurate and scalable quantification of amyloid-β (Aβ) pathology is crucial for deeper disease phenotyping and furthering research in Alzheimer Disease (AD). This multidisciplinary study addresses the current limitations on neuropathology by leveraging a machine learning (ML) pipeline to perform a granular quantification of Aβ deposits and assess their distribution in the temporal lobe. Utilizing 131 whole-slide-images from consecutive autopsied cases at the University of California Davis Alzheimer Disease Research Center, our objectives were threefold: (1) Validate an automatic workflow for Aβ deposit quantification in white matter (WM) and gray matter (GM); (2) define the distributions of different Aβ deposit types in GM and WM, and (3) investigate correlates of Aβ deposits with dementia status and the presence of mixed pathology. Our methodology highlights the robustness and efficacy of the ML pipeline, demonstrating proficiency akin to experts' evaluations. We provide comprehensive insights into the quantification and distribution of Aβ deposits in the temporal GM and WM revealing a progressive increase in tandem with the severity of established diagnostic criteria (NIA-AA). We also present correlations of Aβ load with clinical diagnosis as well as presence/absence of mixed pathology. This study introduces a reproducible workflow, showcasing the practical use of ML approaches in the field of neuropathology, and use of the output data for correlative analyses. Acknowledging limitations, such as potential biases in the ML model and current ML classifications, we propose avenues for future research to refine and expand the methodology. We hope to contribute to the broader landscape of neuropathology advancements, ML applications, and precision medicine, paving the way for deep phenotyping of AD brain cases and establishing a foundation for further advancements in neuropathological research.

Multi-scale relational graph convolutional network for multiple instance learning in histopathology images

Graph convolutional neural networks have shown significant potential in natural and histopathology images. However, their use has only been studied in a single magnification or multi-magnification with either homogeneous graphs or only different node types. In order to leverage the multi-magnification information and improve message passing with graph convolutional networks, we handle different embedding spaces at each magnification by introducing the Multi-Scale Relational Graph Convolutional Network (MS-RGCN) as a multiple instance learning method. We model histopathology image patches and their relation with neighboring patches and patches at other scales (i.e., magnifications) as a graph. We define separate message-passing neural networks based on node and edge types to pass the information between different magnification embedding spaces. We experiment on prostate cancer histopathology images to predict the grade groups based on the extracted features from patches. We also compare our MS-RGCN with multiple state-of-the-art methods with evaluations on several source and held-out datasets. Our method outperforms the state-of-the-art on all of the datasets and image types consisting of tissue microarrays, whole-mount slide regions, and whole-slide images. Through an ablation study, we test and show the value of the pertinent design features of the MS-RGCN.

Unsupervised mutual transformer learning for multi-gigapixel Whole Slide Image classification

The classification of gigapixel Whole Slide Images (WSIs) is an important task in the emerging area of computational pathology. There has been a surge of interest in deep learning models for WSI classification with clinical applications such as cancer detection or prediction of cellular mutations. Most supervised methods require expensive and labor-intensive manual annotations by expert pathologists. Weakly supervised Multiple Instance Learning (MIL) methods have recently demonstrated excellent performance; however, they still require large-scale slide-level labeled training datasets that require a careful inspection of each slide by an expert pathologist. In this work, we propose a fully unsupervised WSI classification algorithm based on mutual transformer learning. The instances (i.e., patches) from gigapixel WSIs are transformed into a latent space and then inverse-transformed to the original space. Using the transformation loss, pseudo labels are generated and cleaned using a transformer label cleaner. The proposed transformer-based pseudo-label generator and cleaner modules mutually train each other iteratively in an unsupervised manner. A discriminative learning mechanism is introduced to improve normal versus cancerous instance labeling. In addition to the unsupervised learning, we demonstrate the effectiveness of the proposed framework for weakly supervised learning and cancer subtype classification as downstream analysis. Extensive experiments on four publicly available datasets show better performance of the proposed algorithm compared to the existing state-of-the-art methods.

Diameter-based pseudo labeling for pathological image segmentation

This paper proposes a semi-and-weak supervised pathological image segmentation method that effectively leverages the pre-recorded long-diameter information of a tumor in clinical as weak supervision. By leveraging the tumor diameter, the proposed method can accurately identify candidate tumor regions for pseudo-label selection. The accurate pseudo labels can improve the segmentation performance. The experimental results demonstrate the effectiveness of our method, which achieved the best performance among the comparative methods.

Hard example mining in Multi-Instance Learning for Whole-Slide Image Classification

Multiple instance learning(MIL) has shown superior performance in the classification of whole-slide images(WSIs). The implementation of multiple instance learning for WSI classification typically involves two components, i.e., a feature extractor, which is used to extract features from patches, and an MIL aggregator, responsible for generating WSI features from the patch features, also contributing to the final classification of WSIs. MIL aggregators often employs a specific MIL classification module. To ensure interactive optimization of the feature extractor and the MIL aggregator, existing state-of-the-art methods select patches based on attention scores to optimize the feature extractor. However, they predominantly focus on easy-to-classify instances, leading to inadequate capabilities in discriminating hard-toclassify instances. In this paper, we introduces a novel Multiple Instance Learning method, HPA-MIL (Hard Pseudo-label Assignment), which directly mines hard instances through pseudo-label assignment. Our experiments demonstrate that HPA-MIL achieves an AUC of 0.9523 on the TCGA NSCLC dataset, which outperforms all the existing state-of-the-art methods compared.

Visual Intratumor Heterogeneity and Breast Tumor Progression

High intratumoral heterogeneity is thought to be a poor prognostic indicator. However, the source of heterogeneity may also be important, as genomic heterogeneity is not always reflected in histologic or 'visual' heterogeneity. We aimed to develop a predictor of histologic heterogeneity and evaluate its association with outcomes and molecular heterogeneity. We used VGG16 to train an image classifier to identify unique, patient-specific visual features in 1655 breast tumors (5907 core images) from the Carolina Breast Cancer Study (CBCS). Extracted features for images, as well as the epithelial and stromal image components, were hierarchically clustered, and visual heterogeneity was defined as a greater distance between images from the same patient. We assessed the association between visual heterogeneity, clinical features, and DNA-based molecular heterogeneity using generalized linear models, and we used Cox models to estimate the association between visual heterogeneity and tumor recurrence. Basal-like and ER-negative tumors were more likely to have low visual heterogeneity, as were the tumors from younger and Black women. Less heterogeneous tumors had a higher risk of recurrence (hazard ratio = 1.62, 95% confidence interval = 1.22-2.16), and were more likely to come from patients whose tumors were comprised of only one subclone or had a TP53 mutation. Associations were similar regardless of whether the image was based on stroma, epithelium, or both. Histologic heterogeneity adds complementary information to commonly used molecular indicators, with low heterogeneity predicting worse outcomes. Future work integrating multiple sources of heterogeneity may provide a more comprehensive understanding of tumor progression.

Machine-learning-based integrative -'omics analyses reveal immunologic and metabolic dysregulation in environmental enteric dysfunction

Environmental enteric dysfunction (EED) is a subclinical enteropathy challenging to diagnose due to an overlap of tissue features with other inflammatory enteropathies. EED subjects (n = 52) from Pakistan, controls (n = 25), and a validation EED cohort (n = 30) from Zambia were used to develop a machine-learning-based image analysis classification model. We extracted histologic feature representations from the Pakistan EED model and correlated them to transcriptomics and clinical biomarkers. In-silico metabolic network modeling was used to characterize alterations in metabolic flux between EED and controls and validated using untargeted lipidomics. Genes encoding beta-ureidopropionase, CYP4F3, and epoxide hydrolase 1 correlated to numerous tissue feature representations. Fatty acid and glycerophospholipid metabolism-related reactions showed altered flux. Increased phosphatidylcholine, lysophosphatidylcholine (LPC), and ether-linked LPCs, and decreased ester-linked LPCs were observed in the duodenal lipidome of Pakistan EED subjects, while plasma levels of glycine-conjugated bile acids were significantly increased. Together, these findings elucidate a multi-omic signature of EED.

Multi-Scale Digital Pathology Patch-Level Prostate Cancer Grading Using Deep Learning: Use Case Evaluation of DiagSet Dataset

Prostate cancer remains a prevalent health concern, emphasizing the critical need for early diagnosis and precise treatment strategies to mitigate mortality rates. The accurate prediction of cancer grade is paramount for timely interventions. This paper introduces an approach to prostate cancer grading, framing it as a classification problem. Leveraging ResNet models on multi-scale patch-level digital pathology and the Diagset dataset, the proposed method demonstrates notable success, achieving an accuracy of 0.999 in identifying clinically significant prostate cancer. The study contributes to the evolving landscape of cancer diagnostics, offering a promising avenue for improved grading accuracy and, consequently, more effective treatment planning. By integrating innovative deep learning techniques with comprehensive datasets, our approach represents a step forward in the pursuit of personalized and targeted cancer care.

Development and deployment of a histopathology-based deep learning algorithm for patient prescreening in a clinical trial

Accurate identification of genetic alterations in tumors, such as Fibroblast Growth Factor Receptor, is crucial for treating with targeted therapies; however, molecular testing can delay patient care due to the time and tissue required. Successful development, validation, and deployment of an AI-based, biomarker-detection algorithm could reduce screening cost and accelerate patient recruitment. Here, we develop a deep-learning algorithm using >3000 H&E-stained whole slide images from patients with advanced urothelial cancers, optimized for high sensitivity to avoid ruling out trial-eligible patients. The algorithm is validated on a dataset of 350 patients, achieving an area under the curve of 0.75, specificity of 31.8% at 88.7% sensitivity, and projected 28.7% reduction in molecular testing. We successfully deploy the system in a non-interventional study comprising 89 global study clinical sites and demonstrate its potential to prioritize/deprioritize molecular testing resources and provide substantial cost savings in the drug development and clinical settings.

Advancing Artificial Intelligence Integration Into the Pathology Workflow: Exploring Opportunities in Gastrointestinal Tract Biopsies

This review aims to present a comprehensive overview of the current landscape of artificial intelligence (AI) applications in the analysis of tubular gastrointestinal biopsies. These publications cover a spectrum of conditions, ranging from inflammatory ailments to malignancies. Moving beyond the conventional diagnosis based on hematoxylin and eosin-stained whole-slide images, the review explores additional implications of AI, including its involvement in interpreting immunohistochemical results, molecular subtyping, and the identification of cellular spatial biomarkers. Furthermore, the review examines how AI can contribute to enhancing the quality and control of diagnostic processes, introducing new workflow options, and addressing the limitations and caveats associated with current AI platforms in this context.

Cross-scale multi-instance learning for pathological image diagnosis

Analyzing high resolution whole slide images (WSIs) with regard to information across multiple scales poses a significant challenge in digital pathology. Multi-instance learning (MIL) is a common solution for working with high resolution images by classifying bags of objects (i.e. sets of smaller image patches). However, such processing is typically performed at a single scale (e.g., 20× magnification) of WSIs, disregarding the vital inter-scale information that is key to diagnoses by human pathologists. In this study, we propose a novel cross-scale MIL algorithm to explicitly aggregate inter-scale relationships into a single MIL network for pathological image diagnosis. The contribution of this paper is three-fold: (1) A novel cross-scale MIL (CS-MIL) algorithm that integrates the multi-scale information and the inter-scale relationships is proposed; (2) A toy dataset with scale-specific morphological features is created and released to examine and visualize differential cross-scale attention; (3) Superior performance on both in-house and public datasets is demonstrated by our simple cross-scale MIL strategy. The official implementation is publicly available at https://github.com/hrlblab/CS-MIL.

Computational Pathology for Prediction of Isocitrate Dehydrogenase Gene Mutation from Whole Slide Images in Adult Patients with Diffuse Glioma

Isocitrate dehydrogenase gene (IDH) mutation is one of the most important molecular markers of glioma. Accurate detection of IDH status is a crucial step for integrated diagnosis of adult-type diffuse gliomas. Herein, a clustering-based hybrid of a convolutional neural network and a vision transformer deep learning model was developed to detect IDH mutation status from annotation-free hematoxylin and eosin-stained whole slide pathologic images of 2275 adult patients with diffuse gliomas. For comparison, a pure convolutional neural network, a pure vision transformer, and a classic multiple-instance learning model were also assessed. The hybrid model achieved an area under the receiver operating characteristic curve of 0.973 in the validation set and 0.953 in the external test set, outperforming the other models. The hybrid model's ability in IDH detection between difficult subgroups with different IDH status but shared histologic features, achieving areas under the receiver operating characteristic curve ranging from 0.850 to 0.985 in validation and test sets. These data suggest that the proposed hybrid model has a potential to be used as a computational pathology tool for preliminary rapid detection of IDH mutation from whole slide images in adult patients with diffuse gliomas.

Development of an Interpretable Deep Learning Model for Pathological Tumor Response Assessment After Neoadjuvant Therapy

BACKGROUND

Neoadjuvant therapy followed by surgery has become the standard of care for locally advanced esophageal squamous cell carcinoma (ESCC) and accurate pathological response assessment is critical to assess the therapeutic efficacy. However, it can be laborious and inconsistency between different observers may occur. Hence, we aim to develop an interpretable deep-learning model for efficient pathological response assessment following neoadjuvant therapy in ESCC.

METHODS

This retrospective study analyzed 337 ESCC resection specimens from 2020-2021 at the Pudong-Branch (Cohort 1) and 114 from 2021-2022 at the Puxi-Branch (External Cohort 2) of Fudan University Shanghai Cancer Center. Whole slide images (WSIs) from these two cohorts were generated using different scanning machines to test the ability of the model in handling color variations. Four pathologists independently assessed the pathological response. The senior pathologists annotated tumor beds and residual tumor percentages on WSIs to determine consensus labels. Furthermore, 1850 image patches were randomly extracted from Cohort 1 WSIs and binarily classified for tumor viability. A deep-learning model employing knowledge distillation was developed to automatically classify positive patches for each WSI and estimate the viable residual tumor percentages. Spatial heatmaps were output for model explanations and visualizations.

RESULTS

The approach achieved high concordance with pathologist consensus, with an R^2 of 0.8437, a RAcc_0.1 of 0.7586, a RAcc_0.3 of 0.9885, which were comparable to two senior pathologists (R^2 of 0.9202/0.9619, RAcc_0.1 of 8506/0.9425, RAcc_0.3 of 1.000/1.000) and surpassing two junior pathologists (R^2 of 0.5592/0.5474, RAcc_0.1 of 0.5287/0.5287, RAcc_0.3 of 0.9080/0.9310). Visualizations enabled the localization of residual viable tumor to augment microscopic assessment.

CONCLUSION

This work illustrates deep learning's potential for assisting pathological response assessment. Spatial heatmaps and patch examples provide intuitive explanations of model predictions, engendering clinical trust and adoption (Code and data will be available at https://github.com/WinnieLaugh/ESCC_Percentage once the paper has been conditionally accepted). Integrating interpretable computational pathology could help enhance the efficiency and consistency of tumor response assessment and empower precise oncology treatment decisions.

Teacher-student collaborated multiple instance learning for pan-cancer PDL1 expression prediction from histopathology slides

Programmed cell death ligand 1 (PDL1), as an important biomarker, is quantified by immunohistochemistry (IHC) with few established histopathological patterns. Deep learning aids in histopathological assessment, yet heterogeneity and lacking spatially resolved annotations challenge precise analysis. Here, we present a weakly supervised learning approach using bulk RNA sequencing for PDL1 expression prediction from hematoxylin and eosin (H&E) slides. Our method extends the multiple instance learning paradigm with the teacher-student framework, which assigns dynamic pseudo-labels for intra-slide heterogeneity and retrieves unlabeled instances using temporal ensemble model distillation. The approach, evaluated on 12,299 slides across 20 solid tumor types, achieves a weighted average area under the curve of 0.83 on fresh-frozen and 0.74 on formalin-fixed specimens for 9 tumors with PDL1 as an established biomarker. Our method predicts PDL1 expression patterns, validated by IHC on 20 slides, offering insights into histologies relevant to PDL1. This demonstrates the potential of deep learning in identifying diverse histological patterns for molecular changes from H&E images.

Automated identification of protein expression intensity and classification of protein cellular locations in mouse brain regions from immunofluorescence images

Knowledge of protein expression in mammalian brains at regional and cellular levels can facilitate understanding of protein functions and associated diseases. As the mouse brain is a typical mammalian brain considering cell type and structure, several studies have been conducted to analyze protein expression in mouse brains. However, labeling protein expression using biotechnology is costly and time-consuming. Therefore, automated models that can accurately recognize protein expression are needed. Here, we constructed machine learning models to automatically annotate the protein expression intensity and cellular location in different mouse brain regions from immunofluorescence images. The brain regions and sub-regions were segmented through learning image features using an autoencoder and then performing K-means clustering and registration to align with the anatomical references. The protein expression intensities for those segmented structures were computed on the basis of the statistics of the image pixels, and patch-based weakly supervised methods and multi-instance learning were used to classify the cellular locations. Results demonstrated that the models achieved high accuracy in the expression intensity estimation, and the F1 score of the cellular location prediction was 74.5%. This work established an automated pipeline for analyzing mouse brain images and provided a foundation for further study of protein expression and functions.

Mapping cell-to-tissue graphs across human placenta histology whole slide images using deep learning with HAPPY

Accurate placenta pathology assessment is essential for managing maternal and newborn health, but the placenta's heterogeneity and temporal variability pose challenges for histology analysis. To address this issue, we developed the 'Histology Analysis Pipeline.PY' (HAPPY), a deep learning hierarchical method for quantifying the variability of cells and micro-anatomical tissue structures across placenta histology whole slide images. HAPPY differs from patch-based features or segmentation approaches by following an interpretable biological hierarchy, representing cells and cellular communities within tissues at a single-cell resolution across whole slide images. We present a set of quantitative metrics from healthy term placentas as a baseline for future assessments of placenta health and we show how these metrics deviate in placentas with clinically significant placental infarction. HAPPY's cell and tissue predictions closely replicate those from independent clinical experts and placental biology literature.

Deep learning-based identification of esophageal cancer subtypes through analysis of high-resolution histopathology images

Esophageal cancer (EC) remains a significant health challenge globally, with increasing incidence and high mortality rates. Despite advances in treatment, there remains a need for improved diagnostic methods and understanding of disease progression. This study addresses the significant challenges in the automatic classification of EC, particularly in distinguishing its primary subtypes: adenocarcinoma and squamous cell carcinoma, using histopathology images. Traditional histopathological diagnosis, while being the gold standard, is subject to subjectivity and human error and imposes a substantial burden on pathologists. This study proposes a binary class classification system for detecting EC subtypes in response to these challenges. The system leverages deep learning techniques and tissue-level labels for enhanced accuracy. We utilized 59 high-resolution histopathological images from The Cancer Genome Atlas (TCGA) Esophageal Carcinoma dataset (TCGA-ESCA). These images were preprocessed, segmented into patches, and analyzed using a pre-trained ResNet101 model for feature extraction. For classification, we employed five machine learning classifiers: Support Vector Classifier (SVC), Logistic Regression (LR), Decision Tree (DT), AdaBoost (AD), Random Forest (RF), and a Feed-Forward Neural Network (FFNN). The classifiers were evaluated based on their prediction accuracy on the test dataset, yielding results of 0.88 (SVC and LR), 0.64 (DT and AD), 0.82 (RF), and 0.94 (FFNN). Notably, the FFNN classifier achieved the highest Area Under the Curve (AUC) score of 0.92, indicating its superior performance, followed closely by SVC and LR, with a score of 0.87. This suggested approach holds promising potential as a decision-support tool for pathologists, particularly in regions with limited resources and expertise. The timely and precise detection of EC subtypes through this system can substantially enhance the likelihood of successful treatment, ultimately leading to reduced mortality rates in patients with this aggressive cancer.

A gradient mapping guided explainable deep neural network for extracapsular extension identification in 3D head and neck cancer computed tomography images

BACKGROUND

Diagnosis and treatment management for head and neck squamous cell carcinoma (HNSCC) is guided by routine diagnostic head and neck computed tomography (CT) scans to identify tumor and lymph node features. The extracapsular extension (ECE) is a strong predictor of patients' survival outcomes with HNSCC. It is essential to detect the occurrence of ECE as it changes staging and treatment planning for patients. Current clinical ECE detection relies on visual identification and pathologic confirmation conducted by clinicians. However, manual annotation of the lymph node region is a required data preprocessing step in most of the current machine learning-based ECE diagnosis studies.

PURPOSE

In this paper, we propose a Gradient Mapping Guided Explainable Network (GMGENet) framework to perform ECE identification automatically without requiring annotated lymph node region information.

METHODS

The gradient-weighted class activation mapping (Grad-CAM) technique is applied to guide the deep learning algorithm to focus on the regions that are highly related to ECE. The proposed framework includes an extractor and a classifier. In a joint training process, informative volumes of interest (VOIs) are extracted by the extractor without labeled lymph node region information, and the classifier learns the pattern to classify the extracted VOIs into ECE positive and negative.

RESULTS

In evaluation, the proposed methods are well-trained and tested using cross-validation. GMGENet achieved test accuracy and area under the curve (AUC) of 92.2% and 89.3%, respectively. GMGENetV2 achieved 90.3% accuracy and 91.7% AUC in the test. The results were compared with different existing models and further confirmed and explained by generating ECE probability heatmaps via a Grad-CAM technique. The presence or absence of ECE has been analyzed and correlated with ground truth histopathological findings.

CONCLUSIONS

The proposed deep network can learn meaningful patterns to identify ECE without providing lymph node contours. The introduced ECE heatmaps will contribute to the clinical implementations of the proposed model and reveal unknown features to radiologists. The outcome of this study is expected to promote the implementation of explainable artificial intelligence-assiste ECE detection.

Magnifying Networks for Histopathological Images with Billions of Pixels

Amongst the other benefits conferred by the shift from traditional to digital pathology is the potential to use machine learning for diagnosis, prognosis, and personalization. A major challenge in the realization of this potential emerges from the extremely large size of digitized images, which are often in excess of 100,000 × 100,000 pixels. In this paper, we tackle this challenge head-on by diverging from the existing approaches in the literature-which rely on the splitting of the original images into small patches-and introducing magnifying networks (MagNets). By using an attention mechanism, MagNets identify the regions of the gigapixel image that benefit from an analysis on a finer scale. This process is repeated, resulting in an attention-driven coarse-to-fine analysis of only a small portion of the information contained in the original whole-slide images. Importantly, this is achieved using minimal ground truth annotation, namely, using only global, slide-level labels. The results from our tests on the publicly available Camelyon16 and Camelyon17 datasets demonstrate the effectiveness of MagNets-as well as the proposed optimization framework-in the task of whole-slide image classification. Importantly, MagNets process at least five times fewer patches from each whole-slide image than any of the existing end-to-end approaches.

Multiple instance learning for digital pathology: A review of the state-of-the-art, limitations & future potential

Digital whole slides images contain an enormous amount of information providing a strong motivation for the development of automated image analysis tools. Particularly deep neural networks show high potential with respect to various tasks in the field of digital pathology. However, a limitation is given by the fact that typical deep learning algorithms require (manual) annotations in addition to the large amounts of image data, to enable effective training. Multiple instance learning exhibits a powerful tool for training deep neural networks in a scenario without fully annotated data. These methods are particularly effective in the domain of digital pathology, due to the fact that labels for whole slide images are often captured routinely, whereas labels for patches, regions, or pixels are not. This potential resulted in a considerable number of publications, with the vast majority published in the last four years. Besides the availability of digitized data and a high motivation from the medical perspective, the availability of powerful graphics processing units exhibits an accelerator in this field. In this paper, we provide an overview of widely and effectively used concepts of (deep) multiple instance learning approaches and recent advancements. We also critically discuss remaining challenges as well as future potential.

A multi-class brain tumor grading system based on histopathological images using a hybrid YOLO and RESNET networks

Gliomas are primary brain tumors caused by glial cells. These cancers' classification and grading are crucial for prognosis and treatment planning. Deep learning (DL) can potentially improve the digital pathology investigation of brain tumors. In this paper, we developed a technique for visualizing a predictive tumor grading model on histopathology pictures to help guide doctors by emphasizing characteristics and heterogeneity in forecasts. The proposed technique is a hybrid model based on YOLOv5 and ResNet50. The function of YOLOv5 is to localize and classify the tumor in large histopathological whole slide images (WSIs). The suggested technique incorporates ResNet into the feature extraction of the YOLOv5 framework, and the detection results show that our hybrid network is effective for identifying brain tumors from histopathological images. Next, we estimate the glioma grades using the extreme gradient boosting classifier. The high-dimensional characteristics and nonlinear interactions present in histopathology images are well-handled by this classifier. DL techniques have been used in previous computer-aided diagnosis systems for brain tumor diagnosis. However, by combining the YOLOv5 and ResNet50 architectures into a hybrid model specifically designed for accurate tumor localization and predictive grading within histopathological WSIs, our study presents a new approach that advances the field. By utilizing the advantages of both models, this creative integration goes beyond traditional techniques to produce improved tumor localization accuracy and thorough feature extraction. Additionally, our method ensures stable training dynamics and strong model performance by integrating ResNet50 into the YOLOv5 framework, addressing concerns about gradient explosion. The proposed technique is tested using the cancer genome atlas dataset. During the experiments, our model outperforms the other standard ways on the same dataset. Our results indicate that the proposed hybrid model substantially impacts tumor subtype discrimination between low-grade glioma (LGG) II and LGG III. With 97.2% of accuracy, 97.8% of precision, 98.6% of sensitivity, and the Dice similarity coefficient of 97%, the proposed model performs well in classifying four grades. These results outperform current approaches for identifying LGG from high-grade glioma and provide competitive performance in classifying four categories of glioma in the literature.

MIHIC: a multiplex IHC histopathological image classification dataset for lung cancer immune microenvironment quantification

Background

Immunohistochemistry (IHC) is a widely used laboratory technique for cancer diagnosis, which selectively binds specific antibodies to target proteins in tissue samples and then makes the bound proteins visible through chemical staining. Deep learning approaches have the potential to be employed in quantifying tumor immune micro-environment (TIME) in digitized IHC histological slides. However, it lacks of publicly available IHC datasets explicitly collected for the in-depth TIME analysis.

Method

In this paper, a notable Multiplex IHC Histopathological Image Classification (MIHIC) dataset is created based on manual annotations by pathologists, which is publicly available for exploring deep learning models to quantify variables associated with the TIME in lung cancer. The MIHIC dataset comprises of totally 309,698 multiplex IHC stained histological image patches, encompassing seven distinct tissue types: Alveoli, Immune cells, Necrosis, Stroma, Tumor, Other and Background. By using the MIHIC dataset, we conduct a series of experiments that utilize both convolutional neural networks (CNNs) and transformer models to benchmark IHC stained histological image classifications. We finally quantify lung cancer immune microenvironment variables by using the top-performing model on tissue microarray (TMA) cores, which are subsequently used to predict patients' survival outcomes.

Result

Experiments show that transformer models tend to provide slightly better performances than CNN models in histological image classifications, although both types of models provide the highest accuracy of 0.811 on the testing dataset in MIHIC. The automatically quantified TIME variables, which reflect proportions of immune cells over stroma and tumor over tissue core, show prognostic value for overall survival of lung cancer patients.

Conclusion

To the best of our knowledge, MIHIC is the first publicly available lung cancer IHC histopathological dataset that includes images with 12 different IHC stains, meticulously annotated by multiple pathologists across 7 distinct categories. This dataset holds significant potential for researchers to explore novel techniques for quantifying the TIME and advancing our understanding of the interactions between the immune system and tumors.

Fully automatic mpMRI analysis using deep learning predicts peritumoral glioblastoma infiltration and subsequent recurrence

Glioblastoma (GBM) is most aggressive and common adult brain tumor. The standard treatments typically include maximal surgical resection, followed adjuvant radiotherapy and chemotherapy. However, the efficacy of these treatment is often limited, as tumor often infiltrate into the surrounding brain tissue, often extending beyond the radiologically defined margins. This infiltration contributes to the high recurrence rate and poor prognosis associated with GBM patients, necessitating advanced methods for early and accurate detection of tumor infiltration. Despite the great promise traditional supervised machine learning shows in predicting tumor infiltration beyond resectable margins, these methods are heavily reliant on expert-drawn Regions of Interest (ROIs), which are used to construct multi-variate models of different Magnetic Resonance (MR) signal characteristics associated with tumor infiltration. This process is both time consuming and resource intensive. Addressing this limitation, our study proposes a novel integration of fully automatic methods for generating ROIs with deep learning algorithms to create predictive maps of tumor infiltration. This approach uses pre-operative multi-parametric MRI (mpMRI) scans, encompassing T1, T1Gd, T2, T2-FLAIR, and ADC sequences, to fully leverage the knowledge from previously drawn ROIs. Subsequently, a patch based Convolutional Neural Network (CNN) model is trained on these automatically generated ROIs to predict areas of potential tumor infiltration. The performance of this model was evaluated using a leave-one-out cross-validation approach. Generated predictive maps binarized for comparison against post-recurrence mpMRI scans. The model demonstrates robust predictive capability, evidenced by the average cross-validated accuracy of 0.87, specificity of 0.88, and sensitivity of 0.90. Notably, the odds ratio of 8.62 indicates that regions identified as high-risk on the predictive map were significantly more likely to exhibit tumor recurrence than low-risk regions. The proposed method demonstrates that a fully automatic mpMRI analysis using deep learning can successfully predict tumor infiltration in peritumoral region for GBM patients while bypassing the intensive requirement for expert-drawn ROIs.

Prediction of Tuberculosis From Lung Tissue Images of Diversity Outbred Mice Using Jump Knowledge Based Cell Graph Neural Network

Tuberculosis (TB), primarily affecting the lungs, is caused by the bacterium Mycobacterium tuberculosis and poses a significant health risk. Detecting acid-fast bacilli (AFB) in stained samples is critical for TB diagnosis. Whole Slide (WS) Imaging allows for digitally examining these stained samples. However, current deep-learning approaches to analyzing large-sized whole slide images (WSIs) often employ patch-wise analysis, potentially missing the complex spatial patterns observed in the granuloma essential for accurate TB classification. To address this limitation, we propose an approach that models cell characteristics and interactions as a graph, capturing both cell-level information and the overall tissue micro-architecture. This method differs from the strategies in related cell graph-based works that rely on edge thresholds based on sparsity/density in cell graph construction, emphasizing a biologically informed threshold determination instead. We introduce a cell graph-based jumping knowledge neural network (CG-JKNN) that operates on the cell graphs where the edge thresholds are selected based on the length of the mycobacteria's cords and the activated macrophage nucleus's size to reflect the actual biological interactions observed in the tissue. The primary process involves training a Convolutional Neural Network (CNN) to segment AFBs and macrophage nuclei, followed by converting large (42831*41159 pixels) lung histology images into cell graphs where an activated macrophage nucleus/AFB represents each node within the graph and their interactions are denoted as edges. To enhance the interpretability of our model, we employ Integrated Gradients and Shapely Additive Explanations (SHAP). Our analysis incorporated a combination of 33 graph metrics and 20 cell morphology features. In terms of traditional machine learning models, Extreme Gradient Boosting (XGBoost) was the best performer, achieving an F1 score of 0.9813 and an Area under the Precision-Recall Curve (AUPRC) of 0.9848 on the test set. Among graph-based models, our CG-JKNN was the top performer, attaining an F1 score of 0.9549 and an AUPRC of 0.9846 on the held-out test set. The integration of graph-based and morphological features proved highly effective, with CG-JKNN and XGBoost showing promising results in classifying instances into AFB and activated macrophage nucleus. The features identified as significant by our models closely align with the criteria used by pathologists in practice, highlighting the clinical applicability of our approach. Future work will explore knowledge distillation techniques and graph-level classification into distinct TB progression categories.

Diagnosing oral and maxillofacial diseases using deep learning

The classification and localization of odontogenic lesions from panoramic radiographs is a challenging task due to the positional biases and class imbalances of the lesions. To address these challenges, a novel neural network, DOLNet, is proposed that uses mutually influencing hierarchical attention across different image scales to jointly learn the global representation of the entire jaw and the local discrepancy between normal tissue and lesions. The proposed approach uses local attention to learn representations within a patch. From the patch-level representations, we generate inter-patch, i.e., global, attention maps to represent the positional prior of lesions in the whole image. Global attention enables the reciprocal calibration of path-level representations by considering non-local information from other patches, thereby improving the generation of whole-image-level representation. To address class imbalances, we propose an effective data augmentation technique that involves merging lesion crops with normal images, thereby synthesizing new abnormal cases for effective model training. Our approach outperforms recent studies, enhancing the classification performance by up to 42.4% and 44.2% in recall and F1 scores, respectively, and ensuring robust lesion localization with respect to lesion size variations and positional biases. Our approach further outperforms human expert clinicians in classification by 10.7 % and 10.8 % in recall and F1 score, respectively.