Artificial Intelligence-Based Mitosis Detection in Breast Cancer Histopathology Images Using Faster R-CNN and Deep CNNs

Mitosis detection and classification for breast cancer diagnosis: What we know and what is next

Breast cancer is the second most deadly malignancy in women, behind lung cancer. Despite significant improvements in medical research, breast cancer is still accurately diagnosed with histological analysis. During this procedure, pathologists examine a physical sample for the presence of mitotic cells, or dividing cells. However, the high resolution of histopathology images and the difficulty of manually detecting tiny mitotic nuclei make it particularly challenging to differentiate mitotic cells from other types of cells. Numerous studies have addressed the detection and classification of mitosis, owing to increasing capacity and developments in automated approaches. The combination of machine learning and deep learning techniques has greatly revolutionized the process of identifying mitotic cells by offering automated, precise, and efficient solutions. In the last ten years, several pioneering methods have been presented, advancing towards practical applications in clinical settings. Unlike other forms of cancer, breast cancer and gliomas are categorized according to the number of mitotic divisions. Numerous papers have been published on techniques for identifying mitosis due to easy access to datasets and open competitions. Convolutional neural networks and other deep learning architectures can precisely identify mitotic cells, significantly decreasing the amount of labor that pathologists must perform. This article examines the techniques used over the past decade to identify and classify mitotic cells in histologically stained breast cancer hematoxylin and eosin images. Furthermore, we examine the benefits of current research techniques and predict forthcoming developments in the investigation of breast cancer mitosis, specifically highlighting machine learning and deep learning.

Rethinking mitosis detection: Towards diverse data and feature representation for better domain generalization

Mitosis detection is one of the fundamental tasks in computational pathology, which is extremely challenging due to the heterogeneity of mitotic cell. Most of the current studies solve the heterogeneity in the technical aspect by increasing the model complexity. However, lacking consideration of the biological knowledge and the complex model design may lead to the overfitting problem while limited the generalizability of the detection model. In this paper, we systematically study the morphological appearances in different mitotic phases as well as the ambiguous non-mitotic cells and identify that balancing the data and feature diversity can achieve better generalizability. Based on this observation, we propose a novel generalizable framework (MitDet) for mitosis detection. The data diversity is considered by the proposed diversity-guided sample balancing (DGSB). And the feature diversity is preserved by inter- and intra- class feature diversity-preserved module (InCDP). Stain enhancement (SE) module is introduced to enhance the domain-relevant diversity of both data and features simultaneously. Extensive experiments have demonstrated that our proposed model outperforms all the state-of-the-art (SOTA) approaches in several popular mitosis detection datasets in both internal and unseen test sets using point annotations only. Comprehensive ablation studies have also proven the effectiveness of the rethinking of data and feature diversity balancing. By analyzing the results quantitatively and qualitatively, we believe that our proposed model not only achieves SOTA performance but also might inspire the future studies in new perspectives. Code is available at https://github.com/linjiatai/MitDet.

The current landscape of artificial intelligence in computational histopathology for cancer diagnosis

Artificial intelligence (AI) marks a frontier in histopathologic analysis shift towards the clinic, becoming a mainstream choice to interpret histological images. Surveying studies assessing AI applications in histopathology from 2013 to 2024, we review key methods (including supervised, unsupervised, weakly supervised and transfer learning) in deep learning-based pattern recognition in computational histopathology for diagnostic and prognostic purposes. Deep learning methods also showed utility in identifying a wide range of genetic mutations and standard pathology biomarkers from routine histology. This survey of 41 primary studies also encompasses key regions of AI applicability in histopathology in a multi-cancer review while marking prospects to introduce AI into the clinical setting with key examples including Swarm Learning and Data Fusion.

Robust ROI Detection in Whole Slide Images Guided by Pathologists' Viewing Patterns

Deep learning techniques offer improvements in computer-aided diagnosis systems. However, acquiring image domain annotations is challenging due to the knowledge and commitment required of expert pathologists. Pathologists often identify regions in whole slide images with diagnostic relevance rather than examining the entire slide, with a positive correlation between the time spent on these critical image regions and diagnostic accuracy. In this paper, a heatmap is generated to represent pathologists' viewing patterns during diagnosis and used to guide a deep learning architecture during training. The proposed system outperforms traditional approaches based on color and texture image characteristics, integrating pathologists' domain expertise to enhance region of interest detection without needing individual case annotations. Evaluating our best model, a U-Net model with a pre-trained ResNet-18 encoder, on a skin biopsy whole slide image dataset for melanoma diagnosis, shows its potential in detecting regions of interest, surpassing conventional methods with an increase of 20%, 11%, 22%, and 12% in precision, recall, F1-score, and Intersection over Union, respectively. In a clinical evaluation, three dermatopathologists agreed on the model's effectiveness in replicating pathologists' diagnostic viewing behavior and accurately identifying critical regions. Finally, our study demonstrates that incorporating heatmaps as supplementary signals can enhance the performance of computer-aided diagnosis systems. Without the availability of eye tracking data, identifying precise focus areas is challenging, but our approach shows promise in assisting pathologists in improving diagnostic accuracy and efficiency, streamlining annotation processes, and aiding the training of new pathologists.

Label credibility correction based on cell morphological differences for cervical cells classification

Cervical cancer is one of the deadliest cancers that pose a significant threat to women's health. Early detection and treatment are commonly used methods to prevent cervical cancer. The use of pathological image analysis techniques for the automatic interpretation of cervical cells in pathological slides is a prominent area of research in the field of digital medicine. According to The Bethesda System, cervical cytology necessitates further classification of precancerous lesions based on positive interpretations. However, clinical definitions among different categories of lesion are complex and often characterized by fuzzy boundaries. In addition, pathologists can deduce different criteria for judgment based on The Bethesda System, leading to potential confusion during data labeling. Noisy labels due to this reason are a great challenge for supervised learning. To address the problem caused by noisy labels, we propose a method based on label credibility correction for cervical cell images classification network. Firstly, a contrastive learning network is used to extract discriminative features from cell images to obtain more similar intra-class sample features. Subsequently, these features are fed into an unsupervised method for clustering, resulting in unsupervised class labels. Then unsupervised labels are corresponded to the true labels to separate confusable and typical samples. Through a similarity comparison between the cluster samples and the statistical feature centers of each class, the label credibility analysis is carried out to group labels. Finally, a cervical cell images multi-class network is trained using synergistic grouping method. In order to enhance the stability of the classification model, momentum is incorporated into the synergistic grouping loss. Experimental validation is conducted on a dataset comprising approximately 60,000 cells from multiple hospitals, showcasing the effectiveness of our proposed approach. The method achieves 2-class task accuracy of 0.9241 and 5-class task accuracy of 0.8598. Our proposed method achieves better performance than existing classification networks on cervical cancer.

Cells Grouping Detection and Confusing Labels Correction on Cervical Pathology Images

Cervical cancer is one of the most prevalent cancers among women, posing a significant threat to their health. Early screening can detect cervical precancerous lesions in a timely manner, thereby enabling the prevention or treatment of the disease. The use of pathological image analysis technology to automatically interpret cells in pathological slices is a hot topic in digital medicine research, as it can reduce the substantial effort required from pathologists to identify cells and can improve diagnostic efficiency and accuracy. Therefore, we propose a cervical cell detection network based on collecting prior knowledge and correcting confusing labels, called PGCC-Net. Specifically, we utilize clinical prior knowledge to break down the detection task into multiple sub-tasks for cell grouping detection, aiming to more effectively learn the specific structure of cells. Subsequently, we merge region proposals from grouping detection to achieve refined detection. In addition, according to the Bethesda system, clinical definitions among various categories of abnormal cervical cells are complex, and their boundaries are ambiguous. Differences in assessment criteria among pathologists result in ambiguously labeled cells, which poses a significant challenge for deep learning networks. To address this issue, we perform a labels correction module with feature similarity by constructing feature centers for typical cells in each category. Then, cells that are easily confused are mapped with these feature centers in order to update cells' annotations. Accurate cell labeling greatly aids the classification head of the detection network. We conducted experimental validation on a public dataset of 7410 images and a private dataset of 13,526 images. The results indicate that our model outperforms the state-of-the-art cervical cell detection methods.

Evaluation of Ex Vivo Shear Wave Elastography of Axillary Sentinel Lymph Nodes in Patients with Early Breast Cancer

Background: The pretherapeutic assessment of axillary lymph node status is crucial in staging early breast cancer patients, significantly influencing their further treatment and prognosis. According to current guidelines, patients with clinically unsuspicious axillary status regularly undergo a biopsy of sentinel lymph nodes (SLNs), whereby metastasis is detected in up to 20% of cases. In recent years, the use of shear wave elastography (SWE) has been studied as an additional ultrasound tool for the non-invasive assessment of tumors in the breast parenchyma and axillary lymph nodes. Previous studies (examining the axilla in patients) have shown that metastases have significantly higher SWE values than benign nodes. Methods: This study aims to evaluate whether SWE can differentiate between tumor-free and metastatic-affected SLN ex vivo, i.e., by examining the pathological specimen. SWE was performed ex vivo on SLN specimens and compared with final histopathological results. Results: A total of 168 SLNs from 105 patients were measured using ex vivo SWE and subjected to standard histopathological processing. In this group, 17 metastases in 17 patients (16.19%) were detected. Tumor-free SLNs had a mean velocity of 1.33 ± 0.23 m/s, while metastatic nodes showed a mean velocity of 1.35 ± 0.29 m/s (p = 0.724). There was no significant difference in ex vivo SWE between benign and malignant SLNs in this population. Conclusions: Contrary to previous studies, this study did not find SWE effective in differentiating lymph node metastases. Further research is needed to clarify SWE's potential role in axillary staging.

A deep learning framework deploying segment anything to detect pan-cancer mitotic figures from haematoxylin and eosin-stained slides

Mitotic activity is an important feature for grading several cancer types. However, counting mitotic figures (cells in division) is a time-consuming and laborious task prone to inter-observer variation. Inaccurate recognition of MFs can lead to incorrect grading and hence potential suboptimal treatment. This study presents an artificial intelligence-based approach to detect mitotic figures in digitised whole-slide images stained with haematoxylin and eosin. Advances in this area are hampered by the small size and variety of datasets available. To address this, we create the largest dataset of mitotic figures (N = 74,620), combining an in-house dataset of soft tissue tumours with five open-source datasets. We then employ a two-stage framework, named the Optimised Mitoses Generator Network (OMG-Net), to identify mitotic figures. This framework first deploys the Segment Anything Model to automatically outline cells, followed by an adapted ResNet18 that distinguishes mitotic figures. OMG-Net achieves an F1 score of 0.84 in detecting pan-cancer mitotic figures, including human breast carcinoma, neuroendocrine tumours, and melanoma. It outperforms previous state-of-the-art models in hold-out test sets. To summarise, our study introduces a generalisable data creation and curation pipeline and a high-performance detection model, which can largely contribute to the field of computer-aided mitotic figure detection.

The 1000 Mitoses Project: A Consensus-Based International Collaborative Study on Mitotic Figures Classification

Introduction. The identification of mitotic figures is essential for the diagnosis, grading, and classification of various different tumors. Despite its importance, there is a paucity of literature reporting the consistency in interpreting mitotic figures among pathologists. This study leverages publicly accessible datasets and social media to recruit an international group of pathologists to score an image database of more than 1000 mitotic figures collectively. Materials and Methods. Pathologists were instructed to randomly select a digital slide from The Cancer Genome Atlas (TCGA) datasets and annotate 10-20 mitotic figures within a 2 mm2 area. The first 1010 submitted mitotic figures were used to create an image dataset, with each figure transformed into an individual tile at 40x magnification. The dataset was redistributed to all pathologists to review and determine whether each tile constituted a mitotic figure. Results. Overall pathologists had a median agreement rate of 80.2% (range 42.0%-95.7%). Individual mitotic figure tiles had a median agreement rate of 87.1% and a fair inter-rater agreement across all tiles (kappa = 0.284). Mitotic figures in prometaphase had lower percentage agreement rates compared to other phases of mitosis. Conclusion. This dataset stands as the largest international consensus study for mitotic figures to date and can be utilized as a training set for future studies. The agreement range reflects a spectrum of criteria that pathologists use to decide what constitutes a mitotic figure, which may have potential implications in tumor diagnostics and clinical management.

[Research progress of breast pathology image diagnosis based on deep learning]

Breast cancer is a malignancy caused by the abnormal proliferation of breast epithelial cells, predominantly affecting female patients, and it is commonly diagnosed using histopathological images. Currently, deep learning techniques have made significant breakthroughs in medical image processing, outperforming traditional detection methods in breast cancer pathology classification tasks. This paper first reviewed the advances in applying deep learning to breast pathology images, focusing on three key areas: multi-scale feature extraction, cellular feature analysis, and classification. Next, it summarized the advantages of multimodal data fusion methods for breast pathology images. Finally, the study discussed the challenges and future prospects of deep learning in breast cancer pathology image diagnosis, providing important guidance for advancing the use of deep learning in breast diagnosis.

ConvNext Mitosis Identification-You Only Look Once (CNMI-YOLO): Domain Adaptive and Robust Mitosis Identification in Digital Pathology

In digital pathology, accurate mitosis detection in histopathological images is critical for cancer diagnosis and prognosis. However, this remains challenging due to the inherent variability in cell morphology and the domain shift problem. This study introduces ConvNext Mitosis Identification-You Only Look Once (CNMI-YOLO), a new 2-stage deep learning method that uses the YOLOv7 architecture for cell detection and the ConvNeXt architecture for cell classification. The goal is to improve the identification of mitosis in different types of cancers. We utilized the Mitosis Domain Generalization Challenge 2022 data set in the experiments to ensure the model's robustness and success across various scanners, species, and cancer types. The CNMI-YOLO model demonstrates superior performance in accurately detecting mitotic cells, significantly outperforming existing models in terms of precision, recall, and F1 score. The CNMI-YOLO model achieved an F1 score of 0.795 on the Mitosis Domain Generalization Challenge 2022 and demonstrated robust generalization with F1 scores of 0.783 and 0.759 on the external melanoma and sarcoma test sets, respectively. Additionally, the study included ablation studies to evaluate various object detection and classification models, such as Faster-RCNN and Swin Transformer. Furthermore, we assessed the model's robustness performance on unseen data, confirming its ability to generalize and its potential for real-world use in digital pathology, using soft tissue sarcoma and melanoma samples not included in the training data set.

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis

Precision and timeliness in breast cancer detection are paramount for improving patient outcomes. Traditional diagnostic methods have predominantly relied on unimodal approaches, but recent advancements in medical data analytics have enabled the integration of diverse data sources beyond conventional imaging techniques. This review critically examines the transformative potential of integrating histopathology images with genomic data, clinical records, and patient histories to enhance diagnostic accuracy and comprehensiveness in multi-modal diagnostic techniques. It explores early, intermediate, and late fusion methods, as well as advanced deep multimodal fusion techniques, including encoder-decoder architectures, attention-based mechanisms, and graph neural networks. An overview of recent advancements in multimodal tasks such as Visual Question Answering (VQA), report generation, semantic segmentation, and cross-modal retrieval is provided, highlighting the utilization of generative AI and visual language models. Additionally, the review delves into the role of Explainable Artificial Intelligence (XAI) in elucidating the decision-making processes of sophisticated diagnostic algorithms, emphasizing the critical need for transparency and interpretability. By showcasing the importance of explainability, we demonstrate how XAI methods, including Grad-CAM, SHAP, LIME, trainable attention, and image captioning, enhance diagnostic precision, strengthen clinician confidence, and foster patient engagement. The review also discusses the latest XAI developments, such as X-VARs, LeGrad, LangXAI, LVLM-Interpret, and ex-ILP, to demonstrate their potential utility in multimodal breast cancer detection, while identifying key research gaps and proposing future directions for advancing the field.

Enhanced gastric cancer classification and quantification interpretable framework using digital histopathology images

Recent developments have highlighted the critical role that computer-aided diagnosis (CAD) systems play in analyzing whole-slide digital histopathology images for detecting gastric cancer (GC). We present a novel framework for gastric histology classification and segmentation (GHCS) that offers modest yet meaningful improvements over existing CAD models for GC classification and segmentation. Our methodology achieves marginal improvements over conventional deep learning (DL) and machine learning (ML) models by adaptively focusing on pertinent characteristics of images. This contributes significantly to our study, highlighting that the proposed model, which performs well on normalized images, is robust in certain respects, particularly in handling variability and generalizing to different datasets. We anticipate that this robustness will lead to better results across various datasets. An expectation-maximizing Naïve Bayes classifier that uses an updated Gaussian Mixture Model is at the heart of the suggested GHCS framework. The effectiveness of our classifier is demonstrated by experimental validation on two publicly available datasets, which produced exceptional classification accuracies of 98.87% and 97.28% on validation sets and 98.47% and 97.31% on test sets. Our framework shows a slight but consistent improvement over previously existing techniques in gastric histopathology image classification tasks, as demonstrated by comparative analysis. This may be attributed to its ability to capture critical features of gastric histopathology images better. Furthermore, using an improved Fuzzy c-means method, our study produces good results in GC histopathology picture segmentation, outperforming state-of-the-art segmentation models with a Dice coefficient of 65.21% and a Jaccard index of 60.24%. The model's interpretability is complemented by Grad-CAM visualizations, which help understand the decision-making process and increase the model's trustworthiness for end-users, especially clinicians.

Artificial intelligence enhances whole-slide interpretation of PD-L1 CPS in triple-negative breast cancer: A multi-institutional ring study

BACKGROUND AND AIMS

Evaluation of the programmed cell death ligand-1 (PD-L1) combined positive score (CPS) is vital to predict the efficacy of the immunotherapy in triple-negative breast cancer (TNBC), but pathologists show substantial variability in the consistency and accuracy of the interpretation. It is of great importance to establish an objective and effective method which is highly repeatable.

METHODS

We proposed a model in a deep learning-based framework, which at the patch level incorporated cell analysis and tissue region analysis, followed by the whole-slide level fusion of patch results. Three rounds of ring studies (RSs) were conducted. Twenty-one pathologists of different levels from four institutions evaluated the PD-L1 CPS in TNBC specimens as continuous scores by visual assessment and our artificial intelligence (AI)-assisted method.

RESULTS

In the visual assessment, the interpretation results of PD-L1 (Dako 22C3) CPS by different levels of pathologists have significant differences and showed weak consistency. Using AI-assisted interpretation, there were no significant differences between all pathologists (P = 0.43), and the intraclass correlation coefficient (ICC) value was increased from 0.618 [95% confidence interval (CI) = 0.524-0.719] to 0.931 (95% CI = 0.902-0.955). The accuracy of interpretation result is further improved to 0.919 (95% CI = 0.886-0.947). Acceptance of AI results by junior pathologists was the highest among all levels, and 80% of the AI results were accepted overall.

CONCLUSION

With the help of the AI-assisted diagnostic method, different levels of pathologists achieved excellent consistency and repeatability in the interpretation of PD-L1 (Dako 22C3) CPS. Our AI-assisted diagnostic approach was proved to strengthen the consistency and repeatability in clinical practice.

Advancing breast ultrasound diagnostics through hybrid deep learning models

Today, doctors rely heavily on medical imaging to identify abnormalities. Proper classification of these abnormalities enables them to take informed actions, leading to early diagnosis and treatment. This paper introduces the "EfficientKNN" model, a novel hybrid deep learning approach that combines the advanced feature extraction capabilities of EfficientNetB3 with the simplicity and effectiveness of the k-Nearest Neighbors (k-NN) algorithm. Initially, EfficientNetB3, pre-trained on ImageNet, is repurposed to serve as a feature extractor. Subsequently, a GlobalAveragePooling2D layer is applied, followed by an optional Principal Component Analysis (PCA) to reduce dimensionality while preserving critical information. PCA is used selectively when deemed necessary. The extracted features are then classified using an optimized k-NN algorithm, fine-tuned through meticulous cross-validation.Our model underwent rigorous training using a curated dataset containing benign, malignant, and normal medical images. Data augmentation techniques, including rotations, shifts, flips, and zooms, were employed to help the model generalize and efficiently handle new, unseen data. To enhance the model's ability to identify the important features necessary for accurate predictions, the dataset was refined using segmentation and overlay techniques. The training utilized an ensemble of optimization algorithms-SGD, Adam, and RMSprop-with hyperparameters set at a learning rate of 0.00045, a batch size of 32, and up to 120 epochs, facilitated by early stopping to prevent overfitting.The results demonstrate that the EfficientKNN model outperforms traditional models such as VGG16, AlexNet, and VGG19 in terms of accuracy, precision, and F1-score. Additionally, the model showed better results compared to EfficientNetB3 alone. Achieving a 100 % accuracy rate on multiple tests, the EfficientKNN model has significant potential for real-world diagnostic applications. This study highlights the model's scalability, efficient use of cloud storage, and real-time prediction capabilities, all while minimizing computational demands.By integrating the strengths of EfficientNetB3's deep learning architecture with the interpretability of k-NN, EfficientKNN presents a significant advancement in medical image classification, promising improved diagnostic accuracy and clinical applicability.

PI-YOLO: dynamic sparse attention and lightweight convolutional based YOLO for vessel detection in pathological images

Vessel density within tumor tissues strongly correlates with tumor proliferation and serves as a critical marker for tumor grading. Recognition of vessel density by pathologists is subject to a strong inter-rater bias, thus limiting its prognostic value. There are many challenges in the task of object detection in pathological images, including complex image backgrounds, dense distribution of small targets, and insignificant differences between the features of the target to be detected and the image background. To address these problems and thus help physicians quantify blood vessels in pathology images, we propose Pathological Images-YOLO (PI-YOLO), an enhanced detection network based on YOLOv7. PI-YOLO incorporates the BiFormer attention mechanism, enhancing global feature extraction and accelerating processing for regions with subtle differences. Additionally, it introduces the CARAFE upsampling module, which optimizes feature utilization and information retention for small targets. Furthermore, the GSConv module improves the ELAN module, reducing model parameters and enhancing inference speed while preserving detection accuracy. Experimental results show that our proposed PI-YOLO network has higher detection accuracy compared to Faster-RCNN, SSD, RetinaNet, YOLOv5 network, and the latest YOLOv7 network, with a mAP value of 87.48%, which is 2.83% higher than the original model. We also validated the performance of this network on the ICPR 2012 mitotic dataset with an F1 value of 0.8678, outperforming other methods, demonstrating the advantages of our network in the task of target detection in complex pathology images.

Impact of Hyperparameter Optimization to Enhance Machine Learning Performance: A Case Study on Breast Cancer Recurrence Prediction

Accurate and early prediction of breast cancer recurrence is crucial to guide medical decisions and treatment success. Machine learning (ML) has shown promise in this domain. However, its effectiveness critically depends on proper hyperparameter setting, a step that is not always performed systematically in the development of ML models. In this study, we aimed to highlight the impact that this process has on the final performance of ML models through a real-world case study by predicting the five-year recurrence of breast cancer patients. We compared the performance of five ML algorithms (Logistic Regression, Decision Tree, Gradient Boosting, eXtreme Gradient Boost, and Deep Neural Network) before and after optimizing their hyperparameters. Simpler algorithms showed better performance using the default hyperparameters. However, after the optimization process, the more complex algorithms demonstrated superior performance. The AUCs obtained before and after adjustment were 0.7 vs. 0.84 for XGB, 0.64 vs. 0.75 for DNN, 0.7 vs. 0.8 for GB, 0.62 vs. 0.7 for DT, and 0.77 vs. 0.72 for LR. The results underscore the critical importance of hyperparameter selection in the development of ML algorithms for the prediction of cancer recurrence. Neglecting this step can undermine the potential of more powerful algorithms and lead to the choice of suboptimal models.

A Deep Learning-Based Assay for Programmed Death Ligand 1 Immunohistochemistry Scoring in Non-Small Cell Lung Carcinoma: Does it Help Pathologists Score?

Several studies have developed various artificial intelligence (AI) models for immunohistochemical analysis of programmed death ligand 1 (PD-L1) in patients with non-small cell lung carcinoma; however, none have focused on specific ways by which AI-assisted systems could help pathologists determine the tumor proportion score (TPS). In this study, we developed an AI model to calculate the TPS of the PD-L1 22C3 assay and evaluated whether and how this AI-assisted system could help pathologists determine the TPS and analyze how AI-assisted systems could affect pathologists' assessment accuracy. We assessed the 4 methods of the AI-assisted system: (1 and 2) pathologists first assessed and then referred to automated AI scoring results (1, positive tumor cell percentage; 2, positive tumor cell percentage and visualized overlay image) for final confirmation, and (3 and 4) pathologists referred to the automated AI scoring results (3, positive tumor cell percentage; 4, positive tumor cell percentage and visualized overlay image) while determining TPS. Mixed-model analysis was used to calculate the odds ratios (ORs) with 95% CI for AI-assisted TPS methods 1 to 4 compared with pathologists' scoring. For all 584 samples of the tissue microarray, the OR for AI-assisted TPS methods 1 to 4 was 0.94 to 1.07 and not statistically significant. Of them, we found 332 discordant cases, on which the pathologists' judgments were inconsistent; the ORs for AI-assisted TPS methods 1, 2, 3, and 4 were 1.28 (1.06-1.54; P = .012), 1.29 (1.06-1.55; P = .010), 1.28 (1.06-1.54; P = .012), and 1.29 (1.06-1.55; P = .010), respectively, which were statistically significant. For discordant cases, the OR for each AI-assisted TPS method compared with the others was 0.99 to 1.01 and not statistically significant. This study emphasized the usefulness of the AI-assisted system for cases in which pathologists had difficulty determining the PD-L1 TPS.

A novel multilevel iterative training strategy for the ResNet50 based mitotic cell classifier

The number of mitotic cells is an important indicator of grading invasive breast cancer. It is very challenging for pathologists to identify and count mitotic cells in pathological sections with naked eyes under the microscope. Therefore, many computational models for the automatic identification of mitotic cells based on machine learning, especially deep learning, have been proposed. However, converging to the local optimal solution is one of the main problems in model training. In this paper, we proposed a novel multilevel iterative training strategy to address the problem. To evaluate the proposed training strategy, we constructed the mitotic cell classification model with ResNet50 and trained the model with different training strategies. The results showed that the models trained with the proposed training strategy performed better than those trained with the conventional strategy in the independent test set, illustrating the effectiveness of the new training strategy. Furthermore, after training with our proposed strategy, the ResNet50 model with Adam optimizer has achieved 89.26% F1 score on the public MITOSI14 dataset, which is higher than that of the state-of-the-art methods reported in the literature.

Early Breast Cancer Risk Assessment: Integrating Histopathology with Artificial Intelligence

Effective risk assessment in early breast cancer is essential for informed clinical decision-making, yet consensus on defining risk categories remains challenging. This paper explores evolving approaches in risk stratification, encompassing histopathological, immunohistochemical, and molecular biomarkers alongside cutting-edge artificial intelligence (AI) techniques. Leveraging machine learning, deep learning, and convolutional neural networks, AI is reshaping predictive algorithms for recurrence risk, thereby revolutionizing diagnostic accuracy and treatment planning. Beyond detection, AI applications extend to histological subtyping, grading, lymph node assessment, and molecular feature identification, fostering personalized therapy decisions. With rising cancer rates, it is crucial to implement AI to accelerate breakthroughs in clinical practice, benefiting both patients and healthcare providers. However, it is important to recognize that while AI offers powerful automation and analysis tools, it lacks the nuanced understanding, clinical context, and ethical considerations inherent to human pathologists in patient care. Hence, the successful integration of AI into clinical practice demands collaborative efforts between medical experts and computational pathologists to optimize patient outcomes.

Whole Slide Imaging Technology and Its Applications: Current and Emerging Perspectives

Background. Whole slide imaging (WSI) represents a paradigm shift in pathology, serving as a necessary first step for a wide array of digital tools to enter the field. It utilizes virtual microscopy wherein glass slides are converted into digital slides and are viewed by pathologists by automated image analysis. Its impact on pathology workflow, reproducibility, dissemination of educational material, expansion of service to underprivileged areas, and institutional collaboration exemplifies a significant innovative movement. The recent US Food and Drug Administration approval to WSI for its use in primary surgical pathology diagnosis has opened opportunities for wider application of this technology in routine practice. Main Text. The ongoing technological advances in digital scanners, image visualization methods, and the integration of artificial intelligence-derived algorithms with these systems provide avenues to exploit its applications. Its benefits are innumerable such as ease of access through the internet, avoidance of physical storage space, and no risk of deterioration of staining quality or breakage of slides to name a few. Although the benefits of WSI to pathology practices are many, the complexities of implementation remain an obstacle to widespread adoption. Some barriers including the high cost, technical glitches, and most importantly professional hesitation to adopt a new technology have hindered its use in routine pathology. Conclusions. In this review, we summarize the technical aspects of WSI, its applications in diagnostic pathology, training, and research along with future perspectives. It also highlights improved understanding of the current challenges to implementation, as well as the benefits and successes of the technology. WSI provides a golden opportunity for pathologists to guide its evolution, standardization, and implementation to better acquaint them with the key aspects of this technology and its judicial use. Also, implementation of routine digital pathology is an extra step requiring resources which (currently) does not usually result increased efficiency or payment.

Mitosis detection, fast and slow: Robust and efficient detection of mitotic figures

Counting of mitotic figures is a fundamental step in grading and prognostication of several cancers. However, manual mitosis counting is tedious and time-consuming. In addition, variation in the appearance of mitotic figures causes a high degree of discordance among pathologists. With advances in deep learning models, several automatic mitosis detection algorithms have been proposed but they are sensitive to domain shift often seen in histology images. We propose a robust and efficient two-stage mitosis detection framework, which comprises mitosis candidate segmentation (Detecting Fast) and candidate refinement (Detecting Slow) stages. The proposed candidate segmentation model, termed EUNet, is fast and accurate due to its architectural design. EUNet can precisely segment candidates at a lower resolution to considerably speed up candidate detection. Candidates are then refined using a deeper classifier network, EfficientNet-B7, in the second stage. We make sure both stages are robust against domain shift by incorporating domain generalization methods. We demonstrate state-of-the-art performance and generalizability of the proposed model on the three largest publicly available mitosis datasets, winning the two mitosis domain generalization challenge contests (MIDOG21 and MIDOG22). Finally, we showcase the utility of the proposed algorithm by processing the TCGA breast cancer cohort (1,124 whole-slide images) to generate and release a repository of more than 620K potential mitotic figures (not exhaustively validated).

Uninformed Teacher-Student for hard-samples distillation in weakly supervised mitosis localization

BACKGROUND AND OBJECTIVE

Mitotic activity is a crucial biomarker for diagnosing and predicting outcomes for different types of cancers, particularly breast cancer. However, manual mitosis counting is challenging and time-consuming for pathologists, with moderate reproducibility due to biopsy slide size, low mitotic cell density, and pattern heterogeneity. In recent years, deep learning methods based on convolutional neural networks (CNNs) have been proposed to address these limitations. Nonetheless, these methods have been hampered by the available data labels, which usually consist only of the centroids of mitosis, and by the incoming noise from annotated hard negatives. As a result, complex algorithms with multiple stages are often required to refine the labels at the pixel level and reduce the number of false positives.

METHODS

This article presents a novel weakly supervised approach for mitosis detection that utilizes only image-level labels on histological hematoxylin and eosin (H&E) images, avoiding the need for complex labeling scenarios. Also, an Uninformed Teacher-Student (UTS) pipeline is introduced to detect and distill hard samples by comparing weakly supervised localizations and the annotated centroids, using strong augmentations to enhance uncertainty. Additionally, an automatic proliferation score is proposed that mimicks the pathologist-annotated mitotic activity index (MAI). The proposed approach is evaluated on three publicly available datasets for mitosis detection on breast histology samples, and two datasets for mitotic activity counting in whole-slide images.

RESULTS

The proposed framework achieves competitive performance with relevant prior literature in all the datasets used for evaluation without explicitly using the mitosis location information during training. This approach challenges previous methods that rely on strong mitosis location information and multiple stages to refine false positives. Furthermore, the proposed pipeline for hard-sample distillation demonstrates promising dataset-specific improvements. Concretely, when the annotation has not been thoroughly refined by multiple pathologists, the UTS model offers improvements of up to ∼4% in mitosis localization, thanks to the detection and distillation of uncertain cases. Concerning the mitosis counting task, the proposed automatic proliferation score shows a moderate positive correlation with the MAI annotated by pathologists at the biopsy level on two external datasets.

CONCLUSIONS

The proposed Uninformed Teacher-Student pipeline leverages strong augmentations to distill uncertain samples and measure dissimilarities between predicted and annotated mitosis. Results demonstrate the feasibility of the weakly supervised approach and highlight its potential as an objective evaluation tool for tumor proliferation.

Efficient mitosis detection: leveraging pre-trained faster R-CNN and cell-level classification

The assessment of mitotic activity is an integral part of the comprehensive evaluation of breast cancer pathology. Understanding the level of tumor dissemination is essential for assessing the severity of the malignancy and guiding appropriate treatment strategies. A pathologist must manually perform the intricate and time-consuming task of counting mitoses by examining biopsy slices stained with Hematoxylin and Eosin (H&E) under a microscope. Mitotic cells can be challenging to distinguish in H&E-stained sections due to limited available datasets and similarities among mitotic and non-mitotic cells. Computer-assisted mitosis detection approaches have simplified the whole procedure by selecting, detecting, and labeling mitotic cells. Traditional detection strategies rely on image processing techniques that apply custom criteria to distinguish between different aspects of an image. Additionally, the automatic feature extraction from histopathology images that exhibit mitosis using neural networks.Additionally, the possibility of automatically extracting features from histopathological images using deep neural networks was investigated. This study examines mitosis detection as an object detection problem using multiple neural networks. From a medical standpoint, mitosis at the tissue level was also investigated utilising pre-trained Faster R-CNN and raw image data. Experiments were done on the MITOS-ATYPIA- 14 dataset and TUPAC16 dataset, and the results were compared to those of other methods described in the literature.

Keeping Pathologists in the Loop and an Adaptive F1-Score Threshold Method for Mitosis Detection in Canine Perivascular Wall Tumours

Performing a mitosis count (MC) is the diagnostic task of histologically grading canine Soft Tissue Sarcoma (cSTS). However, mitosis count is subject to inter- and intra-observer variability. Deep learning models can offer a standardisation in the process of MC used to histologically grade canine Soft Tissue Sarcomas. Subsequently, the focus of this study was mitosis detection in canine Perivascular Wall Tumours (cPWTs). Generating mitosis annotations is a long and arduous process open to inter-observer variability. Therefore, by keeping pathologists in the loop, a two-step annotation process was performed where a pre-trained Faster R-CNN model was trained on initial annotations provided by veterinary pathologists. The pathologists reviewed the output false positive mitosis candidates and determined whether these were overlooked candidates, thus updating the dataset. Faster R-CNN was then trained on this updated dataset. An optimal decision threshold was applied to maximise the F1-score predetermined using the validation set and produced our best F1-score of 0.75, which is competitive with the state of the art in the canine mitosis domain.

Nuclei detection in breast histopathology images with iterative correction

This work presents a deep network architecture to improve nuclei detection performance and achieve the high localization accuracy of nuclei in breast cancer histopathology images. The proposed model consists of two parts, generating nuclear candidate module and refining nuclear localization module. We first design a novel patch learning method to obtain high-quality nuclear candidates, where in addition to categories, location representations are also added to the patch information to implement the multi-task learning process of nuclear classification and localization; meanwhile, the deep supervision mechanism is introduced to obtain the coherent contributions from each scale layer. In order to refine nuclear localization, we propose an iterative correction strategy to make the prediction progressively closer to the ground truth, which significantly improves the accuracy of nuclear localization and facilitates neighbor size selection in the nonmaximum suppression step. Experimental results demonstrate the superior performance of our method for nuclei detection on the H&E stained histopathological image dataset as compared to previous state-of-the-art methods, especially in multiple cluttered nuclei detection, can achieve better results than existing techniques.

A novel dilated contextual attention module for breast cancer mitosis cell detection

Background and object: Mitotic count (MC) is a critical histological parameter for accurately assessing the degree of invasiveness in breast cancer, holding significant clinical value for cancer treatment and prognosis. However, accurately identifying mitotic cells poses a challenge due to their morphological and size diversity. Objective: We propose a novel end-to-end deep-learning method for identifying mitotic cells in breast cancer pathological images, with the aim of enhancing the performance of recognizing mitotic cells. Methods: We introduced the Dilated Cascading Network (DilCasNet) composed of detection and classification stages. To enhance the model's ability to capture distant feature dependencies in mitotic cells, we devised a novel Dilated Contextual Attention Module (DiCoA) that utilizes sparse global attention during the detection. For reclassifying mitotic cell areas localized in the detection stage, we integrate the EfficientNet-B7 and VGG16 pre-trained models (InPreMo) in the classification step. Results: Based on the canine mammary carcinoma (CMC) mitosis dataset, DilCasNet demonstrates superior overall performance compared to the benchmark model. The specific metrics of the model's performance are as follows: F1 score of 82.9%, Precision of 82.6%, and Recall of 83.2%. With the incorporation of the DiCoA attention module, the model exhibited an improvement of over 3.5% in the F1 during the detection stage. Conclusion: The DilCasNet achieved a favorable detection performance of mitotic cells in breast cancer and provides a solution for detecting mitotic cells in pathological images of other cancers.

Enhancing mitosis quantification and detection in meningiomas with computational digital pathology

Mitosis is a critical criterion for meningioma grading. However, pathologists' assessment of mitoses is subject to significant inter-observer variation due to challenges in locating mitosis hotspots and accurately detecting mitotic figures. To address this issue, we leverage digital pathology and propose a computational strategy to enhance pathologists' mitosis assessment. The strategy has two components: (1) A depth-first search algorithm that quantifies the mathematically maximum mitotic count in 10 consecutive high-power fields, which can enhance the preciseness, especially in cases with borderline mitotic count. (2) Implementing a collaborative sphere to group a set of pathologists to detect mitoses under each high-power field, which can mitigate subjective random errors in mitosis detection originating from individual detection errors. By depth-first search algorithm (1) , we analyzed 19 meningioma slides and discovered that the proposed algorithm upgraded two borderline cases verified at consensus conferences. This improvement is attributed to the algorithm's ability to quantify the mitotic count more comprehensively compared to other conventional methods of counting mitoses. In implementing a collaborative sphere (2) , we evaluated the correctness of mitosis detection from grouped pathologists and/or pathology residents, where each member of the group annotated a set of 48 high-power field images for mitotic figures independently. We report that groups with sizes of three can achieve an average precision of 0.897 and sensitivity of 0.699 in mitosis detection, which is higher than an average pathologist in this study (precision: 0.750, sensitivity: 0.667). The proposed computational strategy can be integrated with artificial intelligence workflow, which envisions the future of achieving a rapid and robust mitosis assessment by interactive assisting algorithms that can ultimately benefit patient management.

Multi-resolution deep learning characterizes tertiary lymphoid structures and their prognostic relevance in solid tumors

BACKGROUND

Tertiary lymphoid structures (TLSs) are dense accumulations of lymphocytes in inflamed peripheral tissues, including cancer, and are associated with improved survival and response to immunotherapy in various solid tumors. Histological TLS quantification has been proposed as a novel predictive and prognostic biomarker, but lack of standardized methods of TLS characterization hampers assessment of TLS densities across different patients, diseases, and clinical centers.

METHODS

We introduce an approach based on HookNet-TLS, a multi-resolution deep learning model, for automated and unbiased TLS quantification and identification of germinal centers in routine hematoxylin and eosin stained digital pathology slides. We developed HookNet-TLS using n = 1019 manually annotated TCGA slides from clear cell renal cell carcinoma, muscle-invasive bladder cancer, and lung squamous cell carcinoma.

RESULTS

Here we show that HookNet-TLS automates TLS quantification across multiple cancer types achieving human-level performance and demonstrates prognostic associations similar to visual assessment.

CONCLUSIONS

HookNet-TLS has the potential to be used as a tool for objective quantification of TLS in routine H&E digital pathology slides. We make HookNet-TLS publicly available to promote its use in research.

Refining PD-1/PD-L1 assessment for biomarker-guided immunotherapy: A review

Anti-programmed cell death ligand 1 (anti-PD-L1)  immunotherapy is an increasingly crucial in cancer treatment. To date, the Federal Drug Administration (FDA) has approved four PD-L1 immunohistochemistry (IHC) staining protocols, commercially available in the form of "kits", facilitating testing for PD-L1 expression. These kits comprise four PD-L1 antibodies on two separate IHC platforms, each utilizing distinct, non-interchangeable scoring systems. Several factors, including tumor heterogeneity and the size of the tissue specimens assessed, can lead to PD-L1 status misclassification, potentially hindering the initiation of therapy. Therefore, the development of more accurate predictive biomarkers to distinguish between responders and non-responders prior to anti-PD-1/PD-L1 therapy warrants further research. Achieving this goal necessitates refining sampling criteria, enhancing current methods of PD-L1 detection, and deepening our understanding of the impact of additional biomarkers. In this article, we review potential solutions to improve the predictive accuracy of PD-L1 assessment in order to more precisely anticipate patients' responses to anti-PD-1/PD-L1 therapy, monitor disease progression and predict clinical outcomes.

Detection of Muscularis propria Invasion in Urothelial Carcinoma Using Artificial Intelligence

Background & Objective: Assessment of muscularis propria invasion is a crucial step in the management of urothelial carcinoma since it necessitates aggressive treatment. The diagnosis of muscle invasion is a challenging process for pathologists. Artificial intelligence is developing rapidly and being implemented in various fields of pathology. The purpose of this study was to develop an algorithm for the detection of muscularis propria invasion in urothelial carcinoma. Methods: The Training cohort consisted of 925 images from 50 specimens of urothelial carcinoma. Ninety-seven images from 10 new specimens were used as a validation cohort. Clinical validation used 127 whole specimens with a total of 617 slides. The algorithm determined areas where tumor and muscularis propria events were in nearest proximity, and presented these areas to the pathologist. Results: Analytical evaluation showed a sensitivity of 72% for muscularis propria and 65% for tumor, and a specificity of 46% and 77% for muscularis propria and tumor detection, respectively. The incorporation of the spatial proximity factor between muscularis propria and tumor in the clinical validation significantly improved the detection of muscularis propria invasion, as the algorithm managed to identify all except for one case with muscle invasive bladder cancer in the clinical validation cohort. The case missed by the algorithm was nested urothelial carcinoma, a rare subtype with unusual morphologic features. The pathologist managed to identify muscle invasion based on the images provided by the algorithm in a short time, with an average of approximately 5 s. Conclusion: The algorithm we developed may greatly aid in accurate identification of muscularis propria invasion by imitating the thought process of the pathologist.

Mitotic Nuclei Segmentation and Classification Using Chaotic Butterfly Optimization Algorithm with Deep Learning on Histopathology Images

Histopathological grading of the tumors provides insights about the patient's disease conditions, and it also helps in customizing the treatment plans. Mitotic nuclei classification involves the categorization and identification of nuclei in histopathological images based on whether they are undergoing the cell division (mitosis) process or not. This is an essential procedure in several research and medical contexts, especially in diagnosis and prognosis of cancer. Mitotic nuclei classification is a challenging task since the size of the nuclei is too small to observe, while the mitotic figures possess a different appearance as well. Automated calculation of mitotic nuclei is a stimulating one due to their great similarity to non-mitotic nuclei and their heteromorphic appearance. Both Computer Vision (CV) and Machine Learning (ML) approaches are used in the automated identification and the categorization of mitotic nuclei in histopathological images that endure the procedure of cell division (mitosis). With this background, the current research article introduces the mitotic nuclei segmentation and classification using the chaotic butterfly optimization algorithm with deep learning (MNSC-CBOADL) technique. The main objective of the MNSC-CBOADL technique is to perform automated segmentation and the classification of the mitotic nuclei. In the presented MNSC-CBOADL technique, the U-Net model is initially applied for the purpose of segmentation. Additionally, the MNSC-CBOADL technique applies the Xception model for feature vector generation. For the classification process, the MNSC-CBOADL technique employs the deep belief network (DBN) algorithm. In order to enhance the detection performance of the DBN approach, the CBOA is designed for the hyperparameter tuning model. The proposed MNSC-CBOADL system was validated through simulation using the benchmark database. The extensive results confirmed the superior performance of the proposed MNSC-CBOADL system in the classification of mitotic nuclei.

Artificial Intelligence in Breast Cancer Diagnosis and Personalized Medicine

Breast cancer is a significant cause of cancer-related mortality in women worldwide. Early and precise diagnosis is crucial, and clinical outcomes can be markedly enhanced. The rise of artificial intelligence (AI) has ushered in a new era, notably in image analysis, paving the way for major advancements in breast cancer diagnosis and individualized treatment regimens. In the diagnostic workflow for patients with breast cancer, the role of AI encompasses screening, diagnosis, staging, biomarker evaluation, prognostication, and therapeutic response prediction. Although its potential is immense, its complete integration into clinical practice is challenging. Particularly, these challenges include the imperatives for extensive clinical validation, model generalizability, navigating the "black-box" conundrum, and pragmatic considerations of embedding AI into everyday clinical environments. In this review, we comprehensively explored the diverse applications of AI in breast cancer care, underlining its transformative promise and existing impediments. In radiology, we specifically address AI in mammography, tomosynthesis, risk prediction models, and supplementary imaging methods, including magnetic resonance imaging and ultrasound. In pathology, our focus is on AI applications for pathologic diagnosis, evaluation of biomarkers, and predictions related to genetic alterations, treatment response, and prognosis in the context of breast cancer diagnosis and treatment. Our discussion underscores the transformative potential of AI in breast cancer management and emphasizes the importance of focused research to realize the full spectrum of benefits of AI in patient care.

CervixFormer: A Multi-scale swin transformer-Based cervical pap-Smear WSI classification framework

BACKGROUND AND OBJECTIVES

Cervical cancer affects around 0.5 million women per year, resulting in over 0.3 million fatalities. Therefore, repetitive screening for cervical cancer is of utmost importance. Computer-assisted diagnosis is key for scaling up cervical cancer screening. Current recognition algorithms, however, perform poorly on the whole-slide image (WSI) analysis, fail to generalize for different staining methods and on uneven distribution for subtype imaging, and provide sub-optimal clinical-level interpretations. Herein, we developed CervixFormer-an end-to-end, multi-scale swin transformer-based adversarial ensemble learning framework to assess pre-cancerous and cancer-specific cervical malignant lesions on WSIs.

METHODS

The proposed framework consists of (1) a self-attention generative adversarial network (SAGAN) for generating synthetic images during patch-level training to address the class imbalanced problems; (2) a multi-scale transformer-based ensemble learning method for cell identification at various stages, including atypical squamous cells (ASC) and atypical squamous cells of undetermined significance (ASCUS), which have not been demonstrated in previous studies; and (3) a fusion model for concatenating ensemble-based results and producing final outcomes.

RESULTS

In the evaluation, the proposed method is first evaluated on a private dataset of 717 annotated samples from six classes, obtaining a high recall and precision of 0.940 and 0.934, respectively, in roughly 1.2 minutes. To further examine the generalizability of CervixFormer, we evaluated it on four independent, publicly available datasets, namely, the CRIC cervix, Mendeley LBC, SIPaKMeD Pap Smear, and Cervix93 Extended Depth of Field image datasets. CervixFormer obtained a fairly better performance on two-, three-, four-, and six-class classification of smear- and cell-level datasets. For clinical interpretation, we used GradCAM to visualize a coarse localization map, highlighting important regions in the WSI. Notably, CervixFormer extracts feature mostly from the cell nucleus and partially from the cytoplasm.

CONCLUSIONS

In comparison with the existing state-of-the-art benchmark methods, the CervixFormer outperforms them in terms of recall, accuracy, and computing time.

Clinical applications of graph neural networks in computational histopathology: A review

Pathological examination is the optimal approach for diagnosing cancer, and with the advancement of digital imaging technologies, it has spurred the emergence of computational histopathology. The objective of computational histopathology is to assist in clinical tasks through image processing and analysis techniques. In the early stages, the technique involved analyzing histopathology images by extracting mathematical features, but the performance of these models was unsatisfactory. With the development of artificial intelligence (AI) technologies, traditional machine learning methods were applied in this field. Although the performance of the models improved, there were issues such as poor model generalization and tedious manual feature extraction. Subsequently, the introduction of deep learning techniques effectively addressed these problems. However, models based on traditional convolutional architectures could not adequately capture the contextual information and deep biological features in histopathology images. Due to the special structure of graphs, they are highly suitable for feature extraction in tissue histopathology images and have achieved promising performance in numerous studies. In this article, we review existing graph-based methods in computational histopathology and propose a novel and more comprehensive graph construction approach. Additionally, we categorize the methods and techniques in computational histopathology according to different learning paradigms. We summarize the common clinical applications of graph-based methods in computational histopathology. Furthermore, we discuss the core concepts in this field and highlight the current challenges and future research directions.

Applications of Artificial Intelligence in Breast Pathology

CONTEXT.—

Increasing implementation of whole slide imaging together with digital workflow and advances in computing capacity enable the use of artificial intelligence (AI) in pathology, including breast pathology. Breast pathologists often face a significant workload, with diagnosis complexity, tedious repetitive tasks, and semiquantitative evaluation of biomarkers. Recent advances in developing AI algorithms have provided promising approaches to meet the demand in breast pathology.

OBJECTIVE.—

To provide an updated review of AI in breast pathology. We examined the success and challenges of current and potential AI applications in diagnosing and grading breast carcinomas and other pathologic changes, detecting lymph node metastasis, quantifying breast cancer biomarkers, predicting prognosis and therapy response, and predicting potential molecular changes.

DATA SOURCES.—

We obtained data and information by searching and reviewing literature on AI in breast pathology from PubMed and based our own experience.

CONCLUSIONS.—

With the increasing application in breast pathology, AI not only assists in pathology diagnosis to improve accuracy and reduce pathologists' workload, but also provides new information in predicting prognosis and therapy response.

Machine learning in computational histopathology: Challenges and opportunities

Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.

Comparative Evaluation of Breast Ductal Carcinoma Grading: A Deep-Learning Model and General Pathologists' Assessment Approach

Breast cancer is the most prevalent neoplasia among women, with early and accurate diagnosis critical for effective treatment. In clinical practice, however, the subjective nature of histological grading of infiltrating ductal adenocarcinoma of the breast (DAC-NOS) often leads to inconsistencies among pathologists, posing a significant challenge to achieving optimal patient outcomes. Our study aimed to address this reproducibility problem by leveraging artificial intelligence (AI). We trained a deep-learning model using a convolutional neural network-based algorithm (CNN-bA) on 100 whole slide images (WSIs) of DAC-NOS from the Cancer Genome Atlas Breast Invasive Carcinoma (TCGA-BRCA) dataset. Our model demonstrated high precision, sensitivity, and F1 score across different grading components in about 17.5 h with 19,000 iterations. However, the agreement between the model's grading and that of general pathologists varied, showing the highest agreement for the mitotic count score. These findings suggest that AI has the potential to enhance the accuracy and reproducibility of breast cancer grading, warranting further refinement and validation of this approach.

SB. Multi CNN based automatic detection of mitotic nuclei in breast histopathological images

In breast cancer diagnosis, the number of mitotic cells in a specific area is an important measure. It indicates how far the tumour has spread, which has consequences for forecasting the aggressiveness of cancer. Mitosis counting is a time-consuming and challenging technique that a pathologist does manually by examining Hematoxylin and Eosin (H&E) stained biopsy slices under a microscope. Due to limited datasets and the resemblance between mitotic and non-mitotic cells, detecting mitosis in H&E stained slices is difficult. By assisting in the screening, identifying, and labelling of mitotic cells, computer-aided mitosis detection technologies make the entire procedure much easier. For computer-aided detection approaches of smaller datasets, pre-trained convolutional neural networks are extensively employed. The usefulness of a multi CNN framework with three pre-trained CNNs is investigated in this research for mitosis detection. Features were collected from histopathology data and identified using VGG16, ResNet50, and DenseNet201 pre-trained networks. The proposed framework utilises all training folders of the MITOS dataset provided for the MITOS-ATYPIA contest 2014 and all the 73 folders of the TUPAC16 dataset. Each pre-trained Convolutional Neural Network model, such as VGG16, ResNet50 and DenseNet201, provides an accuracy of 83.22%, 73.67%, and 81.75%, respectively. Different combinations of these pre-trained CNNs constitute a multi CNN framework. Performance measures of multi CNN consisting of 3 pre-trained CNNs with Linear SVM give 93.81% precision and 92.41% F1-score compared to multi CNN combinations with other classifiers such as Adaboost and Random Forest.

Semi-Supervised Learning to Automate Tumor Bud Detection in Cytokeratin-Stained Whole-Slide Images of Colorectal Cancer

Tumor budding is a histopathological biomarker associated with metastases and adverse survival outcomes in colorectal carcinoma (CRC) patients. It is characterized by the presence of single tumor cells or small clusters of cells within the tumor or at the tumor-invasion front. In order to obtain a tumor budding score for a patient, the region with the highest tumor bud density must first be visually identified by a pathologist, after which buds will be counted in the chosen hotspot field. The automation of this process will expectedly increase efficiency and reproducibility. Here, we present a deep learning convolutional neural network model that automates the above procedure. For model training, we used a semi-supervised learning method, to maximize the detection performance despite the limited amount of labeled training data. The model was tested on an independent dataset in which human- and machine-selected hotspots were mapped in relation to each other and manual and machine detected tumor bud numbers in the manually selected fields were compared. We report the results of the proposed method in comparison with visual assessment by pathologists. We show that the automated tumor bud count achieves a prognostic value comparable with visual estimation, while based on an objective and reproducible quantification. We also explore novel metrics to quantify buds such as density and dispersion and report their prognostic value. We have made the model available for research use on the grand-challenge platform.

A Survey of Convolutional Neural Network in Breast Cancer

Problems

For people all over the world, cancer is one of the most feared diseases. Cancer is one of the major obstacles to improving life expectancy in countries around the world and one of the biggest causes of death before the age of 70 in 112 countries. Among all kinds of cancers, breast cancer is the most common cancer for women. The data showed that female breast cancer had become one of the most common cancers.

Aims

A large number of clinical trials have proved that if breast cancer is diagnosed at an early stage, it could give patients more treatment options and improve the treatment effect and survival ability. Based on this situation, there are many diagnostic methods for breast cancer, such as computer-aided diagnosis (CAD).

Methods

We complete a comprehensive review of the diagnosis of breast cancer based on the convolutional neural network (CNN) after reviewing a sea of recent papers. Firstly, we introduce several different imaging modalities. The structure of CNN is given in the second part. After that, we introduce some public breast cancer data sets. Then, we divide the diagnosis of breast cancer into three different tasks: 1. classification; 2. detection; 3. segmentation.

Conclusion

Although this diagnosis with CNN has achieved great success, there are still some limitations. (i) There are too few good data sets. A good public breast cancer dataset needs to involve many aspects, such as professional medical knowledge, privacy issues, financial issues, dataset size, and so on. (ii) When the data set is too large, the CNN-based model needs a sea of computation and time to complete the diagnosis. (iii) It is easy to cause overfitting when using small data sets.

A survey on recent trends in deep learning for nucleus segmentation from histopathology images

Nucleus segmentation is an imperative step in the qualitative study of imaging datasets, considered as an intricate task in histopathology image analysis. Segmenting a nucleus is an important part of diagnosing, staging, and grading cancer, but overlapping regions make it hard to separate and tell apart independent nuclei. Deep Learning is swiftly paving its way in the arena of nucleus segmentation, attracting quite a few researchers with its numerous published research articles indicating its efficacy in the field. This paper presents a systematic survey on nucleus segmentation using deep learning in the last five years (2017-2021), highlighting various segmentation models (U-Net, SCPP-Net, Sharp U-Net, and LiverNet) and exploring their similarities, strengths, datasets utilized, and unfolding research areas.

A population-based study to assess two convolutional neural networks for dental age estimation

BACKGROUND

Dental age (DA) estimation using two convolutional neural networks (CNNs), VGG16 and ResNet101, remains unexplored. In this study, we aimed to investigate the possibility of using artificial intelligence-based methods in an eastern Chinese population.

METHODS

A total of 9586 orthopantomograms (OPGs) (4054 boys and 5532 girls) of the Chinese Han population aged from 6 to 20 years were collected. DAs were automatically calculated using the two CNN model strategies. Accuracy, recall, precision, and F1 score of the models were used to evaluate VGG16 and ResNet101 for age estimation. An age threshold was also employed to evaluate the two CNN models.

RESULTS

The VGG16 network outperformed the ResNet101 network in terms of prediction performance. However, the model effect of VGG16 was less favorable than that in other age ranges in the 15-17 age group. The VGG16 network model prediction results for the younger age groups were acceptable. In the 6-to 8-year-old group, the accuracy of the VGG16 model can reach up to 93.63%, which was higher than the 88.73% accuracy of the ResNet101 network. The age threshold also implies that VGG16 has a smaller age-difference error.

CONCLUSIONS

This study demonstrated that VGG16 performed better when dealing with DA estimation via OPGs than the ResNet101 network on a wholescale. CNNs such as VGG16 hold great promise for future use in clinical practice and forensic sciences.

A generalizable and robust deep learning algorithm for mitosis detection in multicenter breast histopathological images

Mitosis counting of biopsies is an important biomarker for breast cancer patients, which supports disease prognostication and treatment planning. Developing a robust mitotic cell detection model is highly challenging due to its complex growth pattern and high similarities with non-mitotic cells. Most mitosis detection algorithms have poor generalizability across image domains and lack reproducibility and validation in multicenter settings. To overcome these issues, we propose a generalizable and robust mitosis detection algorithm (called FMDet), which is independently tested on multicenter breast histopathological images. To capture more refined morphological features of cells, we convert the object detection task as a semantic segmentation problem. The pixel-level annotations for mitotic nuclei are obtained by taking the intersection of the masks generated from a well-trained nuclear segmentation model and the bounding boxes provided by the MIDOG 2021 challenge. In our segmentation framework, a robust feature extractor is developed to capture the appearance variations of mitotic cells, which is constructed by integrating a channel-wise multi-scale attention mechanism into a fully convolutional network structure. Benefiting from the fact that the changes in the low-level spectrum do not affect the high-level semantic perception, we employ a Fourier-based data augmentation method to reduce domain discrepancies by exchanging the low-frequency spectrum between two domains. Our FMDet algorithm has been tested in the MIDOG 2021 challenge and ranked first place. Further, our algorithm is also externally validated on four independent datasets for mitosis detection, which exhibits state-of-the-art performance in comparison with previously published results. These results demonstrate that our algorithm has the potential to be deployed as an assistant decision support tool in clinical practice. Our code has been released at https://github.com/Xiyue-Wang/1st-in-MICCAI-MIDOG-2021-challenge.

Artificial Hummingbird Algorithm with Transfer-Learning-Based Mitotic Nuclei Classification on Histopathologic Breast Cancer Images

Recently, artificial intelligence (AI) is an extremely revolutionized domain of medical image processing. Specifically, image segmentation is a task that generally aids in such an improvement. This boost performs great developments in the conversion of AI approaches in the research lab to real medical applications, particularly for computer-aided diagnosis (CAD) and image-guided operation. Mitotic nuclei estimates in breast cancer instances have a prognostic impact on diagnosis of cancer aggressiveness and grading methods. The automated analysis of mitotic nuclei is difficult due to its high similarity with nonmitotic nuclei and heteromorphic form. This study designs an artificial hummingbird algorithm with transfer-learning-based mitotic nuclei classification (AHBATL-MNC) on histopathologic breast cancer images. The goal of the AHBATL-MNC technique lies in the identification of mitotic and nonmitotic nuclei on histopathology images (HIs). For HI segmentation process, the PSPNet model is utilized to identify the candidate mitotic patches. Next, the residual network (ResNet) model is employed as feature extractor, and extreme gradient boosting (XGBoost) model is applied as a classifier. To enhance the classification performance, the parameter tuning of the XGBoost model takes place by making use of the AHBA approach. The simulation values of the AHBATL-MNC system are tested on medical imaging datasets and the outcomes are investigated in distinct measures. The simulation values demonstrate the enhanced outcomes of the AHBATL-MNC method compared to other current approaches.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

Deep Learning-Based Artificial Intelligence to Investigate Targeted Nanoparticles' Uptake in TNBC Cells

Triple negative breast cancer (TNBC) is the most aggressive subtype of breast cancer in women. It has the poorest prognosis along with limited therapeutic options. Smart nano-based carriers are emerging as promising approaches in treating TNBC due to their favourable characteristics such as specifically delivering different cargos to cancer cells. However, nanoparticles' tumour cell uptake, and subsequent drug release, are essential factors considered during the drug development process. Contemporary qualitative analyses based on imaging are cumbersome and prone to human biases. Deep learning-based algorithms have been well-established in various healthcare settings with promising scope in drug discovery and development. In this study, the performance of five different convolutional neural network models was evaluated. In this research, we investigated two sequential models from scratch and three pre-trained models, VGG16, ResNet50, and Inception V3. These models were trained using confocal images of nanoparticle-treated cells loaded with a fluorescent anticancer agent. Comparative and cross-validation analyses were further conducted across all models to obtain more meaningful results. Our models showed high accuracy in predicting either high or low drug uptake and release into TNBC cells, indicating great translational potential into practice to aid in determining cellular uptake at the early stages of drug development in any area of research.