Computerized image-based detection and grading of lymphocytic infiltration in HER2+ breast cancer histopathology

Association of graph-based spatial features with overall survival status of glioblastoma patients

Glioblastoma is the most common malignant brain tumor with less than 15 months median survival. To aid prognosis, there is a need for decision tools that leverage diagnostic modalities such as MRI to inform survival. In this study, we examine higher-order spatial proximity characteristics from habitats and propose two graph-based methods (minimum spanning tree and graph run-length matrix) to characterize spatial heterogeneity over tumor MRI-derived intensity habitats and assess their relationships with overall survival as well as the immune signature status of patients with glioblastoma. A data set of 74 patients was studied based on the availability of post-contrast T1-weighted and T2-weighted fluid attenuated inversion recovery (FLAIR) image data in The Cancer Image Archive (TCIA). We assessed the predictive value of MST- and GRLM-derived features from 2D images for prediction of 12-month survival status and immune signature status of patients with glioblastoma via a receiver operating characteristic curve analysis. For 12-month survival prediction using MST-based method, sensitivity and specificity were 0.82 and 0.79 respectively. For GRLM-based method, sensitivity and specificity were 0.73 and 0.77 respectively. For immune status, sensitivity and specificity were 0.91 and 0.69, respectively, for the GRLM-based method with an immune effector. Our results show that the proposed MST- and GRLM-derived features are predictive of 12-month survival status as well as the immune signature status of patients with glioblastoma. To our knowledge, this is the first application of MST- and GRLM-based proximity analyses for the study of radiologically-defined tumor habitats in glioblastoma.

Artificial intelligence in diagnostic pathology

Digital pathology (DP) is being increasingly employed in cancer diagnostics, providing additional tools for faster, higher-quality, accurate diagnosis. The practice of diagnostic pathology has gone through a staggering transformation wherein new tools such as digital imaging, advanced artificial intelligence (AI) algorithms, and computer-aided diagnostic techniques are being used for assisting, augmenting and empowering the computational histopathology and AI-enabled diagnostics. This is paving the way for advancement in precision medicine in cancer. Automated whole slide imaging (WSI) scanners are now rendering diagnostic quality, high-resolution images of entire glass slides and combining these images with innovative digital pathology tools is making it possible to integrate imaging into all aspects of pathology reporting including anatomical, clinical, and molecular pathology. The recent approvals of WSI scanners for primary diagnosis by the FDA as well as the approval of prostate AI algorithm has paved the way for starting to incorporate this exciting technology for use in primary diagnosis. AI tools can provide a unique platform for innovations and advances in anatomical and clinical pathology workflows. In this review, we describe the milestones and landmark trials in the use of AI in clinical pathology with emphasis on future directions.

Applications of Artificial Intelligence in Breast Pathology

CONTEXT.—

Increasing implementation of whole slide imaging together with digital workflow and advances in computing capacity enable the use of artificial intelligence (AI) in pathology, including breast pathology. Breast pathologists often face a significant workload, with diagnosis complexity, tedious repetitive tasks, and semiquantitative evaluation of biomarkers. Recent advances in developing AI algorithms have provided promising approaches to meet the demand in breast pathology.

OBJECTIVE.—

To provide an updated review of AI in breast pathology. We examined the success and challenges of current and potential AI applications in diagnosing and grading breast carcinomas and other pathologic changes, detecting lymph node metastasis, quantifying breast cancer biomarkers, predicting prognosis and therapy response, and predicting potential molecular changes.

DATA SOURCES.—

We obtained data and information by searching and reviewing literature on AI in breast pathology from PubMed and based our own experience.

CONCLUSIONS.—

With the increasing application in breast pathology, AI not only assists in pathology diagnosis to improve accuracy and reduce pathologists' workload, but also provides new information in predicting prognosis and therapy response.

Deep learning model improves tumor-infiltrating lymphocyte evaluation and therapeutic response prediction in breast cancer

Tumor-infiltrating lymphocytes (TILs) have been recognized as key players in the tumor microenvironment of breast cancer, but substantial interobserver variability among pathologists has impeded its utility as a biomarker. We developed a deep learning (DL)-based TIL analyzer to evaluate stromal TILs (sTILs) in breast cancer. Three pathologists evaluated 402 whole slide images of breast cancer and interpreted the sTIL scores. A standalone performance of the DL model was evaluated in the 210 cases (52.2%) exhibiting sTIL score differences of less than 10 percentage points, yielding a concordance correlation coefficient of 0.755 (95% confidence interval [CI], 0.693-0.805) in comparison to the pathologists' scores. For the 226 slides (56.2%) showing a 10 percentage points or greater variance between pathologists and the DL model, revisions were made. The number of discordant cases was reduced to 116 (28.9%) with the DL assistance (p < 0.001). The DL assistance also increased the concordance correlation coefficient of the sTIL score among every two pathologists. In triple-negative and human epidermal growth factor receptor 2 (HER2)-positive breast cancer patients who underwent the neoadjuvant chemotherapy, the DL-assisted revision notably accentuated higher sTIL scores in responders (26.8 ± 19.6 vs. 19.0 ± 16.4, p = 0.003). Furthermore, the DL-assistant revision disclosed the correlation of sTIL-high tumors (sTIL ≥ 50) with the chemotherapeutic response (odd ratio 1.28 [95% confidence interval, 1.01-1.63], p = 0.039). Through enhancing inter-pathologist concordance in sTIL interpretation and predicting neoadjuvant chemotherapy response, here we report the utility of the DL-based tool as a reference for sTIL scoring in breast cancer assessment.

Pitfalls in machine learning-based assessment of tumor-infiltrating lymphocytes in breast cancer: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor-immune interaction in breast cancer is now established, and tumor-infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple-negative (estrogen receptor, progesterone receptor, and HER2-negative) breast cancer and HER2-positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state-of-the-art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false-positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in-depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple-negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Deep Learning-Based Computational Cytopathologic Diagnosis of Metastatic Breast Carcinoma in Pleural Fluid

A Pleural effusion cytology is vital for treating metastatic breast cancer; however, concerns have arisen regarding the low accuracy and inter-observer variability in cytologic diagnosis. Although artificial intelligence-based image analysis has shown promise in cytopathology research, its application in diagnosing breast cancer in pleural fluid remains unexplored. To overcome these limitations, we evaluate the diagnostic accuracy of an artificial intelligence-based model using a large collection of cytopathological slides, to detect the malignant pleural effusion cytology associated with breast cancer. This study includes a total of 569 cytological slides of malignant pleural effusion of metastatic breast cancer from various institutions. We extracted 34,221 augmented image patches from whole-slide images and trained and validated a deep convolutional neural network model (DCNN) (Inception-ResNet-V2) with the images. Using this model, we classified 845 randomly selected patches, which were reviewed by three pathologists to compare their accuracy. The DCNN model outperforms the pathologists by demonstrating higher accuracy, sensitivity, and specificity compared to the pathologists (81.1% vs. 68.7%, 95.0% vs. 72.5%, and 98.6% vs. 88.9%, respectively). The pathologists reviewed the discordant cases of DCNN. After re-examination, the average accuracy, sensitivity, and specificity of the pathologists improved to 87.9, 80.2, and 95.7%, respectively. This study shows that DCNN can accurately diagnose malignant pleural effusion cytology in breast cancer and has the potential to support pathologists.

Imaging in metastatic breast cancer, CT, PET/CT, MRI, WB-DWI, CCA: review and new perspectives

BACKGROUND

Breast cancer is the most frequent cancer in women and remains the second leading cause of death in Western countries. It represents a heterogeneous group of diseases with diverse tumoral behaviour, treatment responsiveness and prognosis. While major progress in diagnosis and treatment has resulted in a decline in breast cancer-related mortality, some patients will relapse and prognosis in this cohort of patients remains poor. Treatment is determined according to tumor subtype; primarily hormone receptor status and HER2 expression. Menopausal status and site of disease relapse are also important considerations in treatment protocols.

MAIN BODY

Staging and repeated evaluation of patients with metastatic breast cancer are central to the accurate assessment of disease extent at diagnosis and during treatment; guiding ongoing clinical management. Advances have been made in the diagnostic and therapeutic fields, particularly with new targeted therapies. In parallel, oncological imaging has evolved exponentially with the development of functional and anatomical imaging techniques. Consistent, reproducible and validated methods of assessing response to therapy is critical in effectively managing patients with metastatic breast cancer.

CONCLUSION

Major progress has been made in oncological imaging over the last few decades. Accurate disease assessment at diagnosis and during treatment is important in the management of metastatic breast cancer. CT (and BS if appropriate) is generally widely available, relatively cheap and sufficient in many cases. However, several additional imaging modalities are emerging and can be used as adjuncts, particularly in pregnancy or other diagnostically challenging cases. Nevertheless, no single imaging technique is without limitation. The authors have evaluated the vast array of imaging techniques - individual, combined parametric and multimodal - that are available or that are emerging in the management of metastatic breast cancer. This includes WB DW-MRI, CCA, novel PET breast cancer-epitope specific radiotracers and radiogenomics.

A pathomic approach for tumor-infiltrating lymphocytes classification on breast cancer digital pathology images

Background and objectives

The detection of tumor-infiltrating lymphocytes (TILs) could aid in the development of objective measures of the infiltration grade and can support decision-making in breast cancer (BC). However, manual quantification of TILs in BC histopathological whole slide images (WSI) is currently based on a visual assessment, thus resulting not standardized, not reproducible, and time-consuming for pathologists. In this work, a novel pathomic approach, aimed to apply high-throughput image feature extraction techniques to analyze the microscopic patterns in WSI, is proposed. In fact, pathomic features provide additional information concerning the underlying biological processes compared to the WSI visual interpretation, thus providing more easily interpretable and explainable results than the most frequently investigated Deep Learning based methods in the literature.

Methods

A dataset containing 1037 regions of interest with tissue compartments and TILs annotated on 195 TNBC and HER2+ BC hematoxylin and eosin (H&E)-stained WSI was used. After segmenting nuclei within tumor-associated stroma using a watershed-based approach, 71 pathomic features were extracted from each nucleus and reduced using a Spearman's correlation filter followed by a nonparametric Wilcoxon rank-sum test and least absolute shrinkage and selection operator. The relevant features were used to classify each candidate nucleus as either TILs or non-TILs using 5 multivariable machine learning classification models trained using 5-fold cross-validation (1) without resampling, (2) with the synthetic minority over-sampling technique and (3) with downsampling. The prediction performance of the models was assessed using ROC curves.

Results

21 features were selected, with most of them related to the well-known TILs properties of having regular shape, clearer margins, high peak intensity, more homogeneous enhancement and different textural pattern than other cells. The best performance was obtained by Random-Forest with ROC AUC of 0.86, regardless of resampling technique.

Conclusions

The presented approach holds promise for the classification of TILs in BC H&E-stained WSI and could provide support to pathologists for a reliable, rapid and interpretable clinical assessment of TILs in BC.

Pathological Detection of Micro and Fuzzy Gastric Cancer Cells Based on Deep Learning

In recent years, with the increasing incidence of cancer, regular physical examination is an important way to find cancer. Nuclear screening is an important method for the diagnosis of gastrointestinal diseases, but it is challenging in the face of small and fuzzy gastrointestinal images. Different from traditional medical objects, pathological slice images are mostly blurry and tiny, which is somewhat difficult to detect and segment. The traditional diagnostic method lacks rapid quantitative analysis and has a certain delay in medical diagnosis, and traditional image processing uses morphological features and pixel distribution to extract features; it is often difficult to achieve the desired effect on small blurry images. This paper proposes a small, microfuzzy pathology detection algorithm based on the attention mechanism; the YOLOv5 is improved under small and micro fuzzy scenarios of the detection of cancer cells in the full field of digital pathology and tests it in the gastric cancer slice dataset. The network structure is improved, and the ability to learn features on small and micro targets is enhanced according to the law of feature distribution. Spatial and channel changes in network attention and attention weight distribution. In the deep blur scenario, the attention mechanism is added to optimize its recognition ability, and the test result shows F1_score is 0.616, and the mAP is 0.611, which can provide the decision support for clinical judgment.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

Region-based feature enhancement using channel-wise attention for classification of breast histopathological images

Breast histopathological image analysis at 400x magnification is essential for the determination of malignant breast tumours. But manual analysis of these images is tedious, subjective, error-prone and requires domain knowledge. To this end, computer-aided tools are gaining much attention in the recent past as it aids pathologists and save time. Furthermore, advances in computational power have leveraged the usage of computer tools. Yet, usage of computer-aided tools to analyse these images is challenging due to various reasons such as heterogeneity of malignant tumours, colour variations and presence of artefacts. Moreover, these images are captured at high resolutions which pose a major challenge to designing deep learning models as it demands high computational requirements. In this context, the present work proposes a new approach to efficiently and effectively extract features from these high-resolution images. In addition, at 400x magnification, the characteristics and structure of nuclei play a prominent role in the decision of malignancy. In this regard, the study introduces a novel CNN architecture called as CWA-Net that uses a colour channel attention module to enhance the features of the potential regions of interest such as nuclei. The developed model is qualitatively and quantitatively evaluated on private and public datasets and achieved an accuracy of 0.95% and 0.96%, respectively. The experimental evaluation demonstrates that the proposed method outperforms state-of-the-art methods on both datasets.

An enhanced Genetic Folding algorithm for prostate and breast cancer detection

Cancer's genomic complexity is gradually increasing as we learn more about it. Genomic classification of various cancers is crucial in providing oncologists with vital information for targeted therapy. Thus, it becomes more pertinent to address issues of patient genomic classification. Prostate cancer is a cancer subtype that exhibits extreme heterogeneity. Prostate cancer contributes to 7.3% of new cancer cases worldwide, with a high prevalence in males. Breast cancer is the most common type of cancer in women and the second most significant cause of death from cancer in women. Breast cancer is caused by abnormal cell growth in the breast tissue, generally referred to as a tumour. Tumours are not synonymous with cancer; they can be benign (noncancerous), pre-malignant (pre-cancerous), or malignant (cancerous). Fine-needle aspiration (FNA) tests are used to biopsy the breast to diagnose breast cancer. Artificial Intelligence (AI) and machine learning (ML) models are used to diagnose with varying accuracy. In light of this, we used the Genetic Folding (GF) algorithm to predict prostate cancer status in a given dataset. An accuracy of 96% was obtained, thus being the current highest accuracy in prostate cancer diagnosis. The model was also used in breast cancer classification with a proposed pipeline that used exploratory data analysis (EDA), label encoding, feature standardization, feature decomposition, log transformation, detect and remove the outliers with Z-score, and the BAGGINGSVM approach attained a 95.96% accuracy. The accuracy of this model was then assessed using the rate of change of PSA, age, BMI, and filtration by race. We discovered that integrating the rate of change of PSA and age in our model raised the model's area under the curve (AUC) by 6.8%, whereas BMI and race had no effect. As for breast cancer classification, no features were removed.

Leveraging artificial intelligence to predict ERG gene fusion status in prostate cancer

Background

TMPRSS2-ERG gene rearrangement, the most common E26 transformation specific (ETS) gene fusion within prostate cancer, is known to contribute to the pathogenesis of this disease and carries diagnostic annotations for prostate cancer patients clinically. The ERG rearrangement status in prostatic adenocarcinoma currently cannot be reliably identified from histologic features on H&E-stained slides alone and hence requires ancillary studies such as immunohistochemistry (IHC), fluorescent in situ hybridization (FISH) or next generation sequencing (NGS) for identification.

Methods

Objective

We accordingly sought to develop a deep learning-based algorithm to identify ERG rearrangement status in prostatic adenocarcinoma based on digitized slides of H&E morphology alone.

Design

Setting, and Participants: Whole slide images from 392 in-house and TCGA cases were employed and annotated using QuPath. Image patches of 224 × 224 pixel were exported at 10 ×, 20 ×, and 40 × for input into a deep learning model based on MobileNetV2 convolutional neural network architecture pre-trained on ImageNet. A separate model was trained for each magnification. Training and test datasets consisted of 261 cases and 131 cases, respectively. The output of the model included a prediction of ERG-positive (ERG rearranged) or ERG-negative (ERG not rearranged) status for each input patch.

Outcome measurements and statistical analysis: Various accuracy measurements including area under the curve (AUC) of the receiver operating characteristic (ROC) curves were used to evaluate the deep learning model.

Results and Limitations

All models showed similar ROC curves with AUC results ranging between 0.82 and 0.85. The sensitivity and specificity of these models were 75.0% and 83.1% (20 × model), respectively.

Conclusions

A deep learning-based model can successfully predict ERG rearrangement status in the majority of prostatic adenocarcinomas utilizing only H&E-stained digital slides. Such an artificial intelligence-based model can eliminate the need for using extra tumor tissue to perform ancillary studies in order to assess for ERG gene rearrangement in prostatic adenocarcinoma.

Global Increase in Breast Cancer Incidence: Risk Factors and Preventive Measures

Breast cancer is a global cause for concern owing to its high incidence around the world. The alarming increase in breast cancer cases emphasizes the management of disease at multiple levels. The management should start from the beginning that includes stringent cancer screening or cancer registry to effective diagnostic and treatment strategies. Breast cancer is highly heterogeneous at morphology as well as molecular levels and needs different therapeutic regimens based on the molecular subtype. Breast cancer patients with respective subtype have different clinical outcome prognoses. Breast cancer heterogeneity emphasizes the advanced molecular testing that will help on-time diagnosis and improved survival. Emerging fields such as liquid biopsy and artificial intelligence would help to under the complexity of breast cancer disease and decide the therapeutic regimen that helps in breast cancer management. In this review, we have discussed various risk factors and advanced technology available for breast cancer diagnosis to combat the worst breast cancer status and areas that need to be focused for the better management of breast cancer.

The future of artificial intelligence in digital pathology - results of a survey across stakeholder groups

AIMS

Artificial intelligence (AI) provides a powerful tool to extract information from digitized histopathology whole slide images. In the last five years, academic and commercial actors have developed new technical solutions for a diverse set of tasks, including tissue segmentation, cell detection, mutation prediction, prognostication and prediction of treatment response. In the light of limited overall resources, it is presently unclear for researchers, practitioners and policymakers which of these topics are stable enough for clinical use in the near future and which topics are still experimental, but worth investing time and effort into.

METHODS

To identify potentially promising applications of AI in pathology, we performed an anonymous online survey of 75 computational pathology domain experts from academia and industry. Participants enrolled in 2021 were queried about their subjective opinion on promising and appealing sub-fields of computational pathology with a focus on solid tumors.

RESULTS

The results of this survey indicate that the prediction of treatment response directly from routine pathology slides is regarded as the most promising future application. This item was ranked highest in the overall analysis and in sub-groups by age and professional background. Furthermore, prediction of genetic alterations, gene expression and survival directly from routine pathology images scored consistently high across subgroups.

CONCLUSIONS

Together, these data demonstrate a possible direction for the development of computational pathology systems in clinical, academic and industrial research in the near future.

MSAL-Net: improve accurate segmentation of nuclei in histopathology images by multiscale attention learning network

Background

The digital pathology images obtain the essential information about the patient’s disease, and the automated nuclei segmentation results can help doctors make better decisions about diagnosing the disease. With the speedy advancement of convolutional neural networks in image processing, deep learning has been shown to play a significant role in the various analysis of medical images, such as nuclei segmentation, mitosis detection and segmentation etc. Recently, several U-net based methods have been developed to solve the automated nuclei segmentation problems. However, these methods fail to deal with the weak features representation from the initial layers and introduce the noise into the decoder path. In this paper, we propose a multiscale attention learning network (MSAL-Net), where the dense dilated convolutions block captures more comprehensive nuclei context information, and a newly modified decoder part is introduced, which integrates with efficient channel attention and boundary refinement modules to effectively learn spatial information for better prediction and further refine the nuclei cell of boundaries.

Results

Both qualitative and quantitative results are obtained on the publicly available MoNuseg dataset. Extensive experiment results verify that our proposed method significantly outperforms state-of-the-art methods as well as the vanilla Unet method in the segmentation task. Furthermore, we visually demonstrate the effect of our modified decoder part.

Conclusion

The MSAL-Net shows superiority with a novel decoder to segment the touching and blurred background nuclei cells obtained from histopathology images with better performance for accurate decoding.

Automatic discrimination of Yamamoto-Kohama classification by machine learning approach for invasive pattern of oral squamous cell carcinoma using digital microscopic images: a retrospective study

OBJECTIVE

The Yamamoto-Kohama criteria are clinically useful for determining the mode of tumor invasion, especially in Japan. However, this evaluation method is based on subjective visual findings and has led to significant differences in determinations between evaluators and facilities. In this retrospective study, we aimed to develop an automatic method of determining the mode of invasion based on the processing of digital medical images.

STUDY DESIGN

Using 101 digitized photographic images of anonymized stained specimen slides, we created a classifier that allowed clinicians to introduce feature values and subjected the cases to machine learning using a random forest approach. We then compared the Yamamoto-Kohama grades (1, 2, 3, 4C, 4D) determined by a human oral and maxillofacial surgeon with those determined using the machine learning approach.

RESULTS

The input of multiple test images into the newly created classifier yielded an overall F-measure value of 87% (grade 1, 93%; grade 2, 67%; grade 3, 89%; grade 4C, 83%; grade 4D, 94%). These results suggest that the output of the classifier was very similar to the judgments of the clinician.

CONCLUSIONS

This system may be valuable for diagnostic support to provide an accurate determination of the mode of invasion.

DRDA-Net: Dense residual dual-shuffle attention network for breast cancer classification using histopathological images

Breast cancer is caused by the uncontrolled growth and division of cells in the breast, whereby a mass of tissue called a tumor is created. Early detection of breast cancer can save many lives. Hence, many researchers worldwide have invested considerable effort in developing robust computer-aided tools for the classification of breast cancer using histopathological images. For this purpose, in this study we designed a dual-shuffle attention-guided deep learning model, called the dense residual dual-shuffle attention network (DRDA-Net). Inspired by the bottleneck unit of the ShuffleNet architecture, in our proposed model we incorporate a channel attention mechanism, which enhances the model's ability to learn the complex patterns of images. Moreover, the model's densely connected blocks address both the overfitting and the vanishing gradient problem, although the model is trained on a substantially small dataset. We have evaluated our proposed model on the publicly available BreaKHis dataset and achieved classification accuracies of 95.72%, 94.41%, 97.43% and 98.1% on four different magnification levels i.e., 40x, 1000x, 200x, 400x respectively which proves the supremacy of the proposed model. The relevant code of the proposed DRDA-Net model can be foundt at: https://github.com/SohamChattopadhyayEE/DRDA-Net.

Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review

Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.

Artificial intelligence applied to breast pathology

The convergence of digital pathology and computer vision is increasingly enabling computers to perform tasks performed by humans. As a result, artificial intelligence (AI) is having an astoundingly positive effect on the field of pathology, including breast pathology. Research using machine learning and the development of algorithms that learn patterns from labeled digital data based on "deep learning" neural networks and feature-engineered approaches to analyze histology images have recently provided promising results. Thus far, image analysis and more complex AI-based tools have demonstrated excellent success performing tasks such as the quantification of breast biomarkers and Ki67, mitosis detection, lymph node metastasis recognition, tissue segmentation for diagnosing breast carcinoma, prognostication, computational assessment of tumor-infiltrating lymphocytes, and prediction of molecular expression as well as treatment response and benefit of therapy from routine H&E images. This review critically examines the literature regarding these applications of AI in the area of breast pathology.

BCHisto-Net: Breast histopathological image classification by global and local feature aggregation

Breast cancer among women is the second most common cancer worldwide. Non-invasive techniques such as mammograms and ultrasound imaging are used to detect the tumor. However, breast histopathological image analysis is inevitable for the detection of malignancy of the tumor. Manual analysis of breast histopathological images is subjective, tedious, laborious and is prone to human errors. Recent developments in computational power and memory have made automation a popular choice for the analysis of these images. One of the key challenges of breast histopathological image classification at 100× magnification is to extract the features of the potential regions of interest to decide on the malignancy of the tumor. The current state-of-the-art CNN based methods for breast histopathological image classification extract features from the entire image (global features) and thus may overlook the features of the potential regions of interest. This can lead to inaccurate diagnosis of breast histopathological images. This research gap has motivated us to propose BCHisto-Net to classify breast histopathological images at 100× magnification. The proposed BCHisto-Net extracts both global and local features required for the accurate classification of breast histopathological images. The global features extract abstract image features while local features focus on potential regions of interest. Furthermore, a feature aggregation branch is proposed to combine these features for the classification of 100× images. The proposed method is quantitatively evaluated on red a private dataset and publicly available BreakHis dataset. An extensive evaluation of the proposed model showed the effectiveness of the local and global features for the classification of these images. The proposed method achieved an accuracy of 95% and 89% on KMC and BreakHis datasets respectively, outperforming state-of-the-art classifiers.

Multi-scale binary pattern encoding network for cancer classification in pathology images

Multi-scale approaches have been widely studied in pathology image analysis. These offer an ability to characterize tissues in an image at various scales, in which the tissues may appear differently. Many of such methods have focused on extracting multi-scale hand-crafted features and applied them to various tasks in pathology image analysis. Even, several deep learning methods explicitly adopt the multi-scale approaches. However, most of these methods simply merge the multi-scale features together or adopt the coarse-to-fine/fine-to-coarse strategy, which uses the features one at a time in a sequential manner. Utilizing the multi-scale features in a cooperative and discriminative fashion, the learning capabilities could be further improved. Herein, we propose a multi-scale approach that can identify and leverage the patterns of the multiple scales within a deep neural network and provide the superior capability of cancer classification. The patterns of the features across multiple scales are encoded as a binary pattern code and further converted to a decimal number, which can be easily embedded in the current framework of the deep neural networks. To evaluate the proposed method, multiple sets of pathology images are employed. Under the various experimental settings, the proposed method is systematically assessed and shows an improved classification performance in comparison to other competing methods.

A Computational Tumor-Infiltrating Lymphocyte Assessment Method Comparable with Visual Reporting Guidelines for Triple-Negative Breast Cancer

Background

Tumor-infiltrating lymphocytes (TILs) are clinically significant in triple-negative breast cancer (TNBC). Although a standardized methodology for visual TILs assessment (VTA) exists, it has several inherent limitations. We established a deep learning-based computational TIL assessment (CTA) method broadly following VTA guideline and compared it with VTA for TNBC to determine the prognostic value of the CTA and a reasonable CTA workflow for clinical practice.

Methods

We trained three deep neural networks for nuclei segmentation, nuclei classification and necrosis classification to establish a CTA workflow. The automatic TIL (aTIL) score generated was compared with manual TIL (mTIL) scores provided by three pathologists in an Asian (n = 184) and a Caucasian (n = 117) TNBC cohort to evaluate scoring concordance and prognostic value.

Findings

The intraclass correlations (ICCs) between aTILs and mTILs varied from 0.40 to 0.70 in two cohorts. Multivariate Cox proportional hazards analysis revealed that the aTIL score was associated with disease free survival (DFS) in both cohorts, as either a continuous [hazard ratio (HR)=0.96, 95% CI 0.94–0.99] or dichotomous variable (HR=0.29, 95% CI 0.12–0.72). A higher C-index was observed in a composite mTIL/aTIL three-tier stratification model than in the dichotomous model, using either mTILs or aTILs alone.

Interpretation

The current study provides a useful tool for stromal TIL assessment and prognosis evaluation for patients with TNBC. A workflow integrating both VTA and CTA may aid pathologists in performing risk management and decision-making tasks.

Computational pathology reveals unique spatial patterns of immune response in H&E images from COVID-19 autopsies: preliminary findings

Purpose: We used computerized image analysis and machine learning approaches to characterize spatial arrangement features of the immune response from digitized autopsied H&E tissue images of the lung in coronavirus disease 2019 (COVID-19) patients. Additionally, we applied our approach to tease out potential morphometric differences from autopsies of patients who succumbed to COVID-19 versus H1N1. Approach: H&E lung whole slide images from autopsy specimens of nine COVID-19 and two H1N1 patients were computationally interrogated. 606 image patches ( ∼ 55 per patient) of 1024 × 882    pixels were extracted from the 11 autopsied patient studies. A watershed-based segmentation approach in conjunction with a machine learning classifier was employed to identify two types of nuclei families: lymphocytes and non-lymphocytes (i.e., other nucleated cells such as pneumocytes, macrophages, and neutrophils). Based off the proximity of the individual nuclei, clusters for each nuclei family were constructed. For each of the resulting clusters, a series of quantitative measurements relating to architecture and density of nuclei clusters were calculated. A receiver operating characteristics-based feature selection method, violin plots, and the t-distributed stochastic neighbor embedding algorithm were employed to study differences in immune patterns. Results: In COVID-19, the immune response consistently showed multiple small-size lymphocyte clusters, suggesting that lymphocyte response is rather modest, possibly due to lymphocytopenia. In H1N1, we found larger lymphocyte clusters that were proximal to large clusters of non-lymphocytes, a possible reflection of increased prevalence of macrophages and other immune cells. Conclusion: Our study shows the potential of computational pathology to uncover immune response features that may not be obvious by routine histopathology visual inspection.

Automated Quantification of sTIL Density with H&E-Based Digital Image Analysis Has Prognostic Potential in Triple-Negative Breast Cancers

Triple-negative breast cancer (TNBC) is an aggressive and difficult-to-treat cancer type that represents approximately 15% of all breast cancers. Recently, stromal tumor-infiltrating lymphocytes (sTIL) resurfaced as a strong prognostic biomarker for overall survival (OS) for TNBC patients. Manual assessment has innate limitations that hinder clinical adoption, and the International Immuno-Oncology Biomarker Working Group (TIL-WG) has therefore envisioned that computational assessment of sTIL could overcome these limitations and recommended that any algorithm should follow the manual guidelines where appropriate. However, no existing studies capture all the concepts of the guideline or have shown the same prognostic evidence as manual assessment. In this study, we present a fully automated digital image analysis pipeline and demonstrate that our hematoxylin and eosin (H&E)-based pipeline can provide a quantitative and interpretable score that correlates with the manual pathologist-derived sTIL status, and importantly, can stratify a retrospective cohort into two significant distinct prognostic groups. We found our score to be prognostic for OS (HR: 0.81 CI: 0.72-0.92 p = 0.001) independent of age, tumor size, nodal status, and tumor type in statistical modeling. While prior studies have followed fragments of the TIL-WG guideline, our approach is the first to follow all complex aspects, where appropriate, supporting the TIL-WG vision of computational assessment of sTIL in the future clinical setting.

Improved bag-of-features using grey relational analysis for classification of histology images

An efficient classification method to categorize histopathological images is a challenging research problem. In this paper, an improved bag-of-features approach is presented as an efficient image classification method. In bag-of-features, a large number of keypoints are extracted from histopathological images that increases the computational cost of the codebook construction step. Therefore, to select the a relevant subset of keypoints, a new keypoints selection method is introduced in the bag-of-features method. To validate the performance of the proposed method, an extensive experimental analysis is conducted on two standard histopathological image datasets, namely ADL and Blue histology datasets. The proposed keypoint selection method reduces the extracted high dimensional features by 95% and 68% from the ADL and Blue histology datasets respectively with less computational time. Moreover, the enhanced bag-of-features method increases classification accuracy by from other considered classification methods.

Clinical Decision Support for Ovarian Carcinoma Subtype Classification: A Pilot Observer Study With Pathology Trainees

CONTEXT.—

Clinical decision support (CDS) systems could assist less experienced pathologists with certain diagnostic tasks for which subspecialty training or extensive experience is typically needed. The effect of decision support on pathologist performance for such diagnostic tasks has not been examined.

OBJECTIVE.—

To examine the impact of a CDS tool for the classification of ovarian carcinoma subtypes by pathology trainees in a pilot observer study using digital pathology.

DESIGN.—

Histologic review on 90 whole slide images from 75 ovarian cancer patients was conducted by 6 pathology residents using: (1) unaided review of whole slide images, and (2) aided review, where in addition to whole slide images observers used a CDS tool that provided information about the presence of 8 histologic features important for subtype classification that were identified previously by an expert in gynecologic pathology. The reference standard of ovarian subtype consisted of majority consensus from a panel of 3 gynecologic pathology experts.

RESULTS.—

Aided review improved pairwise concordance with the reference standard for 5 of 6 observers by 3.3% to 17.8% (for 2 observers, increase was statistically significant) and mean interobserver agreement by 9.2% (not statistically significant). Observers benefited the most when the CDS tool prompted them to look for missed histologic features that were definitive for a certain subtype. Observer performance varied widely across cases with unanimous and nonunanimous reference classification, supporting the need for balancing data sets in terms of case difficulty.

CONCLUSIONS.—

Findings showed the potential of CDS systems to close the knowledge gap between pathologists for complex diagnostic tasks.

An Optimized Approach for Breast Cancer Classification for Histopathological Images Based on Hybrid Feature Set

BACKGROUND

Breast cancer is considered as one of the most perilous sickness among females worldwide and the ratio of new cases is increasing yearly. Many researchers have proposed efficient algorithms to diagnose breast cancer at early stages, which have increased the efficiency and performance by utilizing the learned features of gold standard histopathological images.

OBJECTIVE

Most of these systems have either used traditional handcrafted or deep features, which had a lot of noise and redundancy, and ultimately decrease the performance of the system.

METHODS

A hybrid approach is proposed by fusing and optimizing the properties of handcrafted and deep features to classify the breast cancer images. HOG and LBP features are serially fused with pre-trained models VGG19 and InceptionV3. PCR and ICR are used to evaluate the classification performance of the proposed method.

RESULTS

The method concentrates on histopathological images to classify the breast cancer. The performance is compared with the state-of-the-art techniques, where an overall patient-level accuracy of 97.2% and image-level accuracy of 96.7% is recorded.

CONCLUSION

The proposed hybrid method achieves the best performance as compared to previous methods and it can be used for the intelligent healthcare systems and early breast cancer detection.

Optimized Radial Basis Neural Network for Classification of Breast Cancer Images

BACKGROUND

Breast cancer is a curable disease if diagnosed at an early stage. The chances of having breast cancer are the lowest in married women after the breast-feeding phase because the cancer is formed from the blocked milk ducts.

INTRODUCTION

Nowadays, cancer is considered the leading cause of death globally. Breast cancer is the most common cancer among females. It is possible to develop breast cancer while breast-feeding a baby, but it is rare. Mammography is one of the most effective methods used in hospitals and clinics for early detection of breast cancer. Various researchers are used in artificial intelligence- based mammogram techniques. This process of mammography will reduce the death rate of the patients affected by breast cancer. This process is improved by the image analysing, detection, screening, diagnosing, and other performance measures.

METHODS

The radial basis neural network will be used for classification purposes. The radial basis neural network is designed with the help of the optimization algorithm. The optimization is to tune the classifier to reduce the error rate with the minimum time for the training process. The cuckoo search algorithm will be used for this purpose.

RESULTS

Thus, the proposed optimum RBNN is determined to classify breast cancer images. In this, the three sets of properties were classified by performing the feature extraction and feature reduction. In this breast cancer MRI image, the normal, benign, and malignant is taken to perform the classification. The minimum fitness value is determined to evaluate the optimum value of possible locations. The radial basis function is evaluated with the cuckoo search algorithm to optimize the feature reduction process. The proposed methodology is compared with the traditional radial basis neural network using the evaluation parameter like accuracy, precision, recall and f1-score. The whole system model is done by using Matrix Laboratory (MATLAB) with the adaptation of 2018a since the proposed system is most efficient than most recent related literature.

CONCLUSION

Thus, it concluded with the efficient classification process of RBNN using a cuckoo search algorithm for breast cancer images. The mammogram images are taken into recent research because breast cancer is a major issue for women. This process is carried to classify the various features for three sets of properties. The optimized classifier improves performance and provides a better result. In this proposed research work, the input image is filtered using a wiener filter, and the classifier extracts the feature based on the breast image.

Standardized Method for Defining a 1-mm2 Region of Interest for Calculation of Mitotic Rate on Melanoma Whole Slide Images

CONTEXT.—

Mitotic rate counting is essential in pathologic evaluations in melanoma. The American Joint Committee on Cancer recommends reporting the number of mitotic figures (MFs) in a 1-mm2 area encompassing the "hot spot." There is currently no standard procedure for delineating a 1-mm2 region of interest for MF counting on a digital whole slide image (WSI) of melanoma.

OBJECTIVE.—

To establish a standardized method to enclose a 1-mm2 region of interest for MF counting in melanoma based on WSIs and assess the method's effectiveness.

DESIGN.—

Whole slide images were visualized using the ImageScope viewer (Aperio). Different monitors and viewing magnifications were explored and the annotation tools provided by ImageScope were evaluated. For validation, we compared mitotic rates obtained from WSIs with our method and those from glass slides with traditional microscopy with 30 melanoma cases.

RESULTS.—

Of the monitors we examined, a 32-inch monitor with 3840 × 2160 resolution was optimal for counting MFs within a 1-mm2 region of interest in melanoma. When WSIs were viewed in the ImageScope viewer, ×10 to ×20 magnification during screening could efficiently locate a hot spot and ×20 to ×40 magnification during counting could accurately identify MFs. Fixed-shape annotations with 500 × 500-μm squares or circles can precisely and efficiently enclose a 1-mm2 region of interest. Our method on WSIs was able to produce a higher mitotic rate than with glass slides.

CONCLUSIONS.—

Whole slide images may be used to efficiently count MFs. We recommend fixed-shape annotation with 500 × 500-μm squares or circles for routine practice in counting MFs for melanoma.

Objective Diagnosis for Histopathological Images Based on Machine Learning Techniques: Classical Approaches and New Trends

Histopathology refers to the examination by a pathologist of biopsy samples. Histopathology images are captured by a microscope to locate, examine, and classify many diseases, such as different cancer types. They provide a detailed view of different types of diseases and their tissue status. These images are an essential resource with which to define biological compositions or analyze cell and tissue structures. This imaging modality is very important for diagnostic applications. The analysis of histopathology images is a prolific and relevant research area supporting disease diagnosis. In this paper, the challenges of histopathology image analysis are evaluated. An extensive review of conventional and deep learning techniques which have been applied in histological image analyses is presented. This review summarizes many current datasets and highlights important challenges and constraints with recent deep learning techniques, alongside possible future research avenues. Despite the progress made in this research area so far, it is still a significant area of open research because of the variety of imaging techniques and disease-specific characteristics.

Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning

Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results in a significant improvement in the overall survival rate. In this work, we employ a pipeline based on a cascade of deep neural network classifiers and multi-instance learning to detect the presence of HER2 from Haematoxylin-Eosin slides, which partly mimics the pathologist's behaviour by first recognizing cancer and then evaluating HER2. Our results show that the proposed system presents a good overall effectiveness. Furthermore, the system design is prone to further improvements that can be easily deployed in order to increase the effectiveness score.

Report on computational assessment of Tumor Infiltrating Lymphocytes from the International Immuno-Oncology Biomarker Working Group

Assessment of tumor-infiltrating lymphocytes (TILs) is increasingly recognized as an integral part of the prognostic workflow in triple-negative (TNBC) and HER2-positive breast cancer, as well as many other solid tumors. This recognition has come about thanks to standardized visual reporting guidelines, which helped to reduce inter-reader variability. Now, there are ripe opportunities to employ computational methods that extract spatio-morphologic predictive features, enabling computer-aided diagnostics. We detail the benefits of computational TILs assessment, the readiness of TILs scoring for computational assessment, and outline considerations for overcoming key barriers to clinical translation in this arena. Specifically, we discuss: 1. ensuring computational workflows closely capture visual guidelines and standards; 2. challenges and thoughts standards for assessment of algorithms including training, preanalytical, analytical, and clinical validation; 3. perspectives on how to realize the potential of machine learning models and to overcome the perceptual and practical limits of visual scoring.

Biomarkers for Response to Immune Checkpoint Blockade

Immune checkpoint blockade (ICB) has significant clinical activity in diverse cancer classes and can induce durable remissions in even refractory advanced disease. However, only a minority of cancer patients treated with ICB have long-term benefits, and ICB treatment is associated with significant, potentially life-threatening, autoimmune side effects. There is a great need to develop biomarkers of response to guide patient selection to maximize the chance of benefit and prevent unnecessary toxicity, and current biomarkers do not have optimal positive or negative predictive value. A variety of potential biomarkers are currently being developed, including those based on assessment of checkpoint protein expression, evaluation of tumor-intrinsic features including mutation burden and viral infection, evaluation of features of the tumor immune microenvironment including nature of immune cell infiltration, and features of the host such as composition of the gut microbiome. Better understanding of the underlying fundamental mechanisms of immune response and resistance to ICB, along with the use of complementary assays that interrogate distinct features of the tumor, the tumor microenvironment, and host immune system, will allow more precise use of these therapies to optimize patient outcomes.

Histo-genomics: digital pathology at the forefront of precision medicine

The toughest challenge OMICs face is that they provide extremely high molecular resolution but poor spatial information. Understanding the cellular/histological context of the overwhelming genetic data is critical for a full understanding of the clinical behavior of a malignant tumor. Digital pathology can add an extra layer of information to help visualize in a spatial and microenvironmental context the molecular information of cancer. Thus, histo-genomics provide a unique chance for data integration. In the era of a precision medicine, a four-dimensional (4D) (temporal/spatial) analysis of cancer aided by digital pathology can be a critical step to understand the evolution/progression of different cancers and consequently tailor individual treatment plans. For instance, the integration of molecular biomarkers expression into a three-dimensional (3D) image of a digitally scanned tumor can offer a better understanding of its subtype, behavior, host immune response and prognosis. Using advanced digital image analysis, a larger spectrum of parameters can be analyzed as potential predictors of clinical behavior. Correlation between morphological features and host immune response can be also performed with therapeutic implications. Radio-histomics, or the interface of radiological images and histology is another emerging exciting field which encompasses the integration of radiological imaging with digital pathological images, genomics, and clinical data to portray a more holistic approach to understating and treating disease. These advances in digital slide scanning are not without technical challenges, which will be addressed carefully in this review with quick peek at its future.

Artificial intelligence in digital breast pathology: Techniques and applications

Breast cancer is the most common cancer and second leading cause of cancer-related death worldwide. The mainstay of breast cancer workup is histopathological diagnosis - which guides therapy and prognosis. However, emerging knowledge about the complex nature of cancer and the availability of tailored therapies have exposed opportunities for improvements in diagnostic precision. In parallel, advances in artificial intelligence (AI) along with the growing digitization of pathology slides for the primary diagnosis are a promising approach to meet the demand for more accurate detection, classification and prediction of behaviour of breast tumours. In this article, we cover the current and prospective uses of AI in digital pathology for breast cancer, review the basics of digital pathology and AI, and outline outstanding challenges in the field.

PROSTATE CANCER DETECTION USING HISTOPATHOLOGY IMAGES AND CLASSIFICATION USING IMPROVED RideNN

Medical industry reports prostate cancer as common and high among men and alarms the necessity for detecting prostate cancer for which the required morphology is extracted from the histopathology images. Commonly, the Gleason grading system remains a perfect factor for grading prostate cancer in men, but pathologists suffer from minute inter- and intra-observer variations. Thus, an automatic method for segmenting and classifying prostate cancer is modeled in this paper. The significance of the developed method is that the segmentation and classification are gland-oriented using the Color Space (CS) transformation and Salp Swarm Optimization Algorithm-based Rider Neural Network (SSA-RideNN). The gland region is considered as the morphology for cancer detection from which the maximal significant regions are extracted as features using multiple-kernel scale-invariant feature transform (MK-SIFT). Here, the RideNN classifier is trained optimally using the proposed Salp–Rider Algorithm (SRA), which is the integration of Salp Swarm Optimization Algorithm (SSA) and Rider Optimization Algorithm (ROA). The experimentation is performed using the histopathology images and the analysis based on sensitivity, accuracy, and specificity reveals that the proposed prostate cancer detection method acquired the maximal accuracy, sensitivity, and specificity of 0.8966, 0.8919, and 0.8596, respectively.

Artificial intelligence in digital pathology - new tools for diagnosis and precision oncology

In the past decade, advances in precision oncology have resulted in an increased demand for predictive assays that enable the selection and stratification of patients for treatment. The enormous divergence of signalling and transcriptional networks mediating the crosstalk between cancer, stromal and immune cells complicates the development of functionally relevant biomarkers based on a single gene or protein. However, the result of these complex processes can be uniquely captured in the morphometric features of stained tissue specimens. The possibility of digitizing whole-slide images of tissue has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology, which enable mining of subvisual morphometric phenotypes and might, ultimately, improve patient management. In this Perspective, we critically evaluate various AI-based computational approaches for digital pathology, focusing on deep neural networks and 'hand-crafted' feature-based methodologies. We aim to provide a broad framework for incorporating AI and machine learning tools into clinical oncology, with an emphasis on biomarker development. We discuss some of the challenges relating to the use of AI, including the need for well-curated validation datasets, regulatory approval and fair reimbursement strategies. Finally, we present potential future opportunities for precision oncology.

Society of Toxicologic Pathology Digital Pathology and Image Analysis Special Interest Group Article*: Opinion on the Application of Artificial Intelligence and Machine Learning to Digital Toxicologic Pathology

Toxicologic pathology is transitioning from analog to digital methods. This transition seems inevitable due to a host of ongoing social and medical technological forces. Of these, artificial intelligence (AI) and in particular machine learning (ML) are globally disruptive, rapidly growing sectors of technology whose impact on the long-established field of histopathology is quickly being realized. The development of increasing numbers of algorithms, peering ever deeper into the histopathological space, has demonstrated to the scientific community that AI pathology platforms are now poised to truly impact the future of precision and personalized medicine. However, as with all great technological advances, there are implementation and adoption challenges. This review aims to define common and relevant AI and ML terminology, describe data generation and interpretation, outline current and potential future business cases, discuss validation and regulatory hurdles, and most importantly, propose how overcoming the challenges of this burgeoning technology may shape toxicologic pathology for years to come, enabling pathologists to contribute even more effectively to answering scientific questions and solving global health issues. [Box: see text]

Tumor Malignancy Detection Using Histopathology Imaging

Image segmentation and classification in the biomedical imaging field has high worth in cancer diagnosis and grading. The proposed method classifies the images based on a combination of handcrafted features and shape features using bag of visual words (BoW). The multistage segmentation technique to localize the nuclei in histopathology images includes the stain decomposition and histogram equalization to highlight the nucleus region, followed by the nuclei key point extraction using fast radial symmetry transform, normalized graph cut based on the nuclei region estimation, and nuclei boundary estimation using modified gradient. Subsequently, features from localized regions termed as handcrafted features and the shape features using BoW are extracted for classification. The experiments are performed using both the handcrafted features and BoW to take the advantages of both local nuclei features and globally spatial features. The simulation is performed on the Bisque and BreakHis data sets (with corresponding average accuracies of 93.87% and 96.96%, respectively) and confirms better diagnosis performance using the proposed method.

Analysis-Synthesis Learning With Shared Features: Algorithms for Histology Image Classification

OBJECTIVE

The diversity of tissue structure in histopathological images makes feature extraction for classification a challenging task. Dictionary learning within a sparse representation-based classification (SRC) framework has been shown to be successful for feature discovery. However, there exist stiff practical challenges: 1) computational complexity of SRC can be onerous in the decision stage since it involves solving a sparsity constrained optimization problem and often over a large number of image patches; and 2) images from distinct classes continue to share several geometric features. We propose a novel analysis-synthesis model learning with shared features algorithm (ALSF) for classifying such images more effectively.

METHODS

In the ALSF, a joint analysis and synthesis learning model is introduced to learn the classifier and the feature extractor at the same time. Unlike SRC, no explicit optimization is needed in the inference phase leading to much reduced computation. Crucially, we introduce the learning of a low-rank shared dictionary and a shared analysis operator, which more accurately represents both similarities and differences in histopathological images from distinct classes. We also develop an extension of ALSF with a sparsity constraint, whose presence or absence facilitates a cost-performance tradeoff.

RESULTS

The ALSF is evaluated on three challenging and well-known datasets: 1) spleen tissue images; 2) brain tumor images; and 3) breast cancer tissue dataset, provided by different organizations.

CONCLUSION

Experimental results demonstrate both complexity and performance benefits of the ALSF over state-of-the-art alternatives.

SIGNIFICANCE

Modeling shared features with appropriate quantitative constraints lead to significantly improved classification in histopathology.

Breast cancer outcome prediction with tumour tissue images and machine learning

PURPOSE

Recent advances in machine learning have enabled better understanding of large and complex visual data. Here, we aim to investigate patient outcome prediction with a machine learning method using only an image of tumour sample as an input.

METHODS

Utilising tissue microarray (TMA) samples obtained from the primary tumour of patients (N = 1299) within a nationwide breast cancer series with long-term-follow-up, we train and validate a machine learning method for patient outcome prediction. The prediction is performed by classifying samples into low or high digital risk score (DRS) groups. The outcome classifier is trained using sample images of 868 patients and evaluated and compared with human expert classification in a test set of 431 patients.

RESULTS

In univariate survival analysis, the DRS classification resulted in a hazard ratio of 2.10 (95% CI 1.33-3.32, p = 0.001) for breast cancer-specific survival. The DRS classification remained as an independent predictor of breast cancer-specific survival in a multivariate Cox model with a hazard ratio of 2.04 (95% CI 1.20-3.44, p = 0.007). The accuracy (C-index) of the DRS grouping was 0.60 (95% CI 0.55-0.65), as compared to 0.58 (95% CI 0.53-0.63) for human expert predictions based on the same TMA samples.

CONCLUSIONS

Our findings demonstrate the feasibility of learning prognostic signals in tumour tissue images without domain knowledge. Although further validation is needed, our study suggests that machine learning algorithms can extract prognostically relevant information from tumour histology complementing the currently used prognostic factors in breast cancer.

Unsupervised Domain Adaptation for Classification of Histopathology Whole-Slide Images

Computational image analysis is one means for evaluating digitized histopathology specimens that can increase the reproducibility and reliability with which cancer diagnoses are rendered while simultaneously providing insight as to the underlying mechanisms of disease onset and progression. A major challenge that is confronted when analyzing samples that have been prepared at disparate laboratories and institutions is that the algorithms used to assess the digitized specimens often exhibit heterogeneous staining characteristics because of slight differences in incubation times and the protocols used to prepare the samples. Unfortunately, such variations can render a prediction model learned from one batch of specimens ineffective for characterizing an ensemble originating from another site. In this work, we propose to adopt unsupervised domain adaptation to effectively transfer the discriminative knowledge obtained from any given source domain to the target domain without requiring any additional labeling or annotation of images at the target site. In this paper, our team investigates the use of two approaches for performing the adaptation: (1) color normalization and (2) adversarial training. The adversarial training strategy is implemented through the use of convolutional neural networks to find an invariant feature space and Siamese architecture within the target domain to add a regularization that is appropriate for the entire set of whole-slide images. The adversarial adaptation results in significant classification improvement compared with the baseline models under a wide range of experimental settings.

Unsupervised Feature Extraction via Deep Learning for Histopathological Classification of Colon Tissue Images

Histopathological examination is today's gold standard for cancer diagnosis. However, this task is time consuming and prone to errors as it requires a detailed visual inspection and interpretation of a pathologist. Digital pathology aims at alleviating these problems by providing computerized methods that quantitatively analyze digitized histopathological tissue images. The performance of these methods mainly relies on the features that they use, and thus, their success strictly depends on the ability of these features by successfully quantifying the histopathology domain. With this motivation, this paper presents a new unsupervised feature extractor for effective representation and classification of histopathological tissue images. This feature extractor has three main contributions: First, it proposes to identify salient subregions in an image, based on domain-specific prior knowledge, and to quantify the image by employing only the characteristics of these subregions instead of considering the characteristics of all image locations. Second, it introduces a new deep learning-based technique that quantizes the salient subregions by extracting a set of features directly learned on image data and uses the distribution of these quantizations for image representation and classification. To this end, the proposed deep learning-based technique constructs a deep belief network of the restricted Boltzmann machines (RBMs), defines the activation values of the hidden unit nodes in the final RBM as the features, and learns the quantizations by clustering these features in an unsupervised way. Third, this extractor is the first example for successfully using the restricted Boltzmann machines in the domain of histopathological image analysis. Our experiments on microscopic colon tissue images reveal that the proposed feature extractor is effective to obtain more accurate classification results compared to its counterparts.

Association of Peritumoral Radiomics With Tumor Biology and Pathologic Response to Preoperative Targeted Therapy for HER2 (ERBB2)-Positive Breast Cancer

IMPORTANCE

There has been significant recent interest in understanding the utility of quantitative imaging to delineate breast cancer intrinsic biological factors and therapeutic response. No clinically accepted biomarkers are as yet available for estimation of response to human epidermal growth factor receptor 2 (currently known as ERBB2, but referred to as HER2 in this study)-targeted therapy in breast cancer.

OBJECTIVE

To determine whether imaging signatures on clinical breast magnetic resonance imaging (MRI) could noninvasively characterize HER2-positive tumor biological factors and estimate response to HER2-targeted neoadjuvant therapy.

DESIGN, SETTING, AND PARTICIPANTS

In a retrospective diagnostic study encompassing 209 patients with breast cancer, textural imaging features extracted within the tumor and annular peritumoral tissue regions on MRI were examined as a means to identify increasingly granular breast cancer subgroups relevant to therapeutic approach and response. First, among a cohort of 117 patients who received an MRI prior to neoadjuvant chemotherapy (NAC) at a single institution from April 27, 2012, through September 4, 2015, imaging features that distinguished HER2+ tumors from other receptor subtypes were identified. Next, among a cohort of 42 patients with HER2+ breast cancers with available MRI and RNaseq data accumulated from a multicenter, preoperative clinical trial (BrUOG 211B), a signature of the response-associated HER2-enriched (HER2-E) molecular subtype within HER2+ tumors (n = 42) was identified. The association of this signature with pathologic complete response was explored in 2 patient cohorts from different institutions, where all patients received HER2-targeted NAC (n = 28, n = 50). Finally, the association between significant peritumoral features and lymphocyte distribution was explored in patients within the BrUOG 211B trial who had corresponding biopsy hematoxylin-eosin-stained slide images. Data analysis was conducted from January 15, 2017, to February 14, 2019.

MAIN OUTCOMES AND MEASURES

Evaluation of imaging signatures by the area under the receiver operating characteristic curve (AUC) in identifying HER2+ molecular subtypes and distinguishing pathologic complete response (ypT0/is) to NAC with HER2-targeting.

RESULTS

In the 209 patients included (mean [SD] age, 51.1 [11.7] years), features from the peritumoral regions better discriminated HER2-E tumors (maximum AUC, 0.85; 95% CI, 0.79-0.90; 9-12 mm from the tumor) compared with intratumoral features (AUC, 0.76; 95% CI, 0.69-0.84). A classifier combining peritumoral and intratumoral features identified the HER2-E subtype (AUC, 0.89; 95% CI, 0.84-0.93) and was significantly associated with response to HER2-targeted therapy in both validation cohorts (AUC, 0.80; 95% CI, 0.61-0.98 and AUC, 0.69; 95% CI, 0.53-0.84). Features from the 0- to 3-mm peritumoral region were significantly associated with the density of tumor-infiltrating lymphocytes (R2 = 0.57; 95% CI, 0.39-0.75; P = .002).

CONCLUSIONS AND RELEVANCE

A combination of peritumoral and intratumoral characteristics appears to identify intrinsic molecular subtypes of HER2+ breast cancers from imaging, offering insights into immune response within the peritumoral environment and suggesting potential benefit for treatment guidance.

Spatial Architecture and Arrangement of Tumor-Infiltrating Lymphocytes for Predicting Likelihood of Recurrence in Early-Stage Non-Small Cell Lung Cancer

PURPOSE

The presence of a high degree of tumor-infiltrating lymphocytes (TIL) has been proven to be associated with outcome in patients with non-small cell lung cancer (NSCLC). However, recent evidence indicates that tissue architecture is also prognostic of disease-specific survival and recurrence. We show a set of descriptors (spatial TIL, SpaTIL) that capture density, and spatial colocalization of TILs and tumor cells across digital images that can predict likelihood of recurrence in early-stage NSCLC.

EXPERIMENTAL DESIGN

The association between recurrence in early-stage NSCLC and SpaTIL features was explored on 301 patients across four different cohorts. Cohort D1 (n = 70) was used to identify the most prognostic SpaTIL features and to train a classifier to predict the likelihood of recurrence. The classifier performance was evaluated in cohorts D2 (n = 119), D3 (n = 112), and D4 (n = 112). Two pathologists graded each sample of D1 and D2; intraobserver agreement and association between manual grading and likelihood of recurrence were analyzed.

RESULTS

SpaTIL was associated with likelihood of recurrence in all test sets (log-rank P < 0.02). A multivariate Cox proportional hazards analysis revealed an HR of 3.08 (95% confidence interval, 2.1-4.5, P = 7.3 × 10-5). In contrast, agreement among expert pathologists using tumor grade was moderate (Kappa = 0.5), and the manual TIL grading was only prognostic for one reader in D2 (P = 8.0 × 10-3).

CONCLUSIONS

A set of features related to density and spatial architecture of TILs was found to be associated with a likelihood of recurrence of early-stage NSCLC. This information could potentially be used for helping in treatment planning and management of early-stage NSCLC.See related commentary by Peled et al., p. 1449.

Stromal cell ratio based on automated image analysis as a predictor for platinum-resistant recurrent ovarian cancer

Background

Identifying high-risk patients for platinum resistance is critical for improving clinical management of ovarian cancer. We aimed to use automated image analysis of hematoxylin & eosin (H&E) stained sections to identify the association between microenvironmental composition and platinum-resistant recurrent ovarian cancer.

Methods

Ninety-one patients with ovarian cancer containing the data of automated image analysis for H&E histological sections were initially reviewed.

Results

Seventy-one patients with recurrent disease were finally identified. Among 30 patients with high stromal cell ratio, 60% of the patients had platinum-resistant recurrence, which was significantly higher than the rate in patients with low stromal cell ratio (9.80%, P <  0.001). Multivariate logistic regression analysis revealed elevated CA125 level after 3 cycles of chemotherapy (P <  0.001) and high stromal cell ratio (P = 0.002) were the negative predictors of platinum-resistant relapse. The area under the curve (AUC) of receiver operating characteristic (ROC) curves of the models for predicting platinum-resistant recurrence with stromal cell ratio, normalization of CA125 level, and the combination of two parameters were 0.78, 0.79, and 0.89 respectively.

Conclusions

Our results demonstrated stromal cell ratio based on automated image analysis may be a potential predictor for ovarian cancer patients at high risk of platinum-resistant recurrence, and it could improve the predictive value of model when combined with normalization of CA125 level after 3 cycles of chemotherapy.