Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Machine Learning in Diagnosis and Prognosis of Lung Cancer by PET-CT

As a disease with high morbidity and high mortality, lung cancer has seriously harmed people's health. Therefore, early diagnosis and treatment are more important. PET/CT is usually used to obtain the early diagnosis, staging, and curative effect evaluation of tumors, especially lung cancer, due to the heterogeneity of tumors and the differences in artificial image interpretation and other reasons, it also fails to entirely reflect the real situation of tumors. Artificial intelligence (AI) has been applied to all aspects of life. Machine learning (ML) is one of the important ways to realize AI. With the help of the ML method used by PET/CT imaging technology, there are many studies in the diagnosis and treatment of lung cancer. This article summarizes the application progress of ML based on PET/CT in lung cancer, in order to better serve the clinical. In this study, we searched PubMed using machine learning, lung cancer, and PET/CT as keywords to find relevant articles in the past 5 years or more. We found that PET/CT-based ML approaches have achieved significant results in the detection, delineation, classification of pathology, molecular subtyping, staging, and response assessment with survival and prognosis of lung cancer, which can provide clinicians a powerful tool to support and assist in critical daily clinical decisions. However, ML has some shortcomings such as slightly poor repeatability and reliability.

Progression from ductal carcinoma in situ to invasive breast cancer: molecular features and clinical significance

Ductal carcinoma in situ (DCIS) represents pre-invasive breast carcinoma. In untreated cases, 25-60% DCIS progress to invasive ductal carcinoma (IDC). The challenge lies in distinguishing between non-progressive and progressive DCIS, often resulting in over- or under-treatment in many cases. With increasing screen-detected DCIS in these years, the nature of DCIS has aroused worldwide attention. A deeper understanding of the biological nature of DCIS and the molecular journey of the DCIS-IDC transition is crucial for more effective clinical management. Here, we reviewed the key signaling pathways in breast cancer that may contribute to DCIS initiation and progression. We also explored the molecular features of DCIS and IDC, shedding light on the progression of DCIS through both inherent changes within tumor cells and alterations in the tumor microenvironment. In addition, valuable research tools utilized in studying DCIS including preclinical models and newer advanced technologies such as single-cell sequencing, spatial transcriptomics and artificial intelligence, have been systematically summarized. Further, we thoroughly discussed the clinical advancements in DCIS and IDC, including prognostic biomarkers and clinical managements, with the aim of facilitating more personalized treatment strategies in the future. Research on DCIS has already yielded significant insights into breast carcinogenesis and will continue to pave the way for practical clinical applications.

Mapping cell-to-tissue graphs across human placenta histology whole slide images using deep learning with HAPPY

Accurate placenta pathology assessment is essential for managing maternal and newborn health, but the placenta's heterogeneity and temporal variability pose challenges for histology analysis. To address this issue, we developed the 'Histology Analysis Pipeline.PY' (HAPPY), a deep learning hierarchical method for quantifying the variability of cells and micro-anatomical tissue structures across placenta histology whole slide images. HAPPY differs from patch-based features or segmentation approaches by following an interpretable biological hierarchy, representing cells and cellular communities within tissues at a single-cell resolution across whole slide images. We present a set of quantitative metrics from healthy term placentas as a baseline for future assessments of placenta health and we show how these metrics deviate in placentas with clinically significant placental infarction. HAPPY's cell and tissue predictions closely replicate those from independent clinical experts and placental biology literature.

Deep Learning Promotes Profiling of Multiple miRNAs in Single Extracellular Vesicles for Cancer Diagnosis

Extracellular vesicle microRNAs (EV miRNAs) are critical noninvasive biomarkers for early cancer diagnosis. However, accurate cancer diagnosis based on bulk analysis is hindered by the heterogeneity among EVs. Herein, we report an approach for profiling single-EV multi-miRNA signatures by combining total internal reflection fluorescence (TIRF) imaging with a deep learning (DL) algorithm for the first time. This innovative technique allows for the precise characterization of EV miRNAs at the single-vesicle level, overcoming the challenges posed by EV heterogeneity. TIRF with high resolution and a signal-to-noise ratio can simultaneously detect multi-miRNAs in situ in individual EVs. DL algorithm avoids complicated and inaccurate artificial feature extraction, achieving automated high-resolution image analysis. Using this approach, we reveal that the main variation of EVs from 5 cancer cells and normal plasma is the triple-positive EV subpopulation, and the classification accuracy of single triple-positive EVs from 6 sources can reach above 95%. In the clinical cohort, 20 patients (5 lung cancer, 5 breast cancer, 5 cervical cancer, and 5 colon cancer) and 5 healthy controls are predicted with an overall accuracy of 100%. This single-EV strategy provides new opportunities for exploring more specific EV biomarkers to achieve cancer diagnosis and classification.

Prognostic significance of collagen signatures at breast tumor boundary obtained by combining multiphoton imaging and imaging analysis

PURPOSE

Collagen features in breast tumor microenvironment is closely associated with the prognosis of patients. We aim to explore the prognostic significance of collagen features at breast tumor border by combining multiphoton imaging and imaging analysis.

METHODS

We used multiphoton microscopy (MPM) to label-freely image human breast tumor samples and then constructed an automatic classification model based on deep learning to identify collagen signatures from multiphoton images. We recognized three kinds of collagen signatures at tumor boundary (CSTB I-III) in a small-scale, and furthermore obtained a CSTB score for each patient based on the combined CSTB I-III by using the ridge regression analysis. The prognostic performance of CSTB score is assessed by the area under the receiver operating characteristic curve (AUC), Cox proportional hazard regression analysis, as well as Kaplan-Meier survival analysis.

RESULTS

As an independent prognostic factor, statistical results reveal that the prognostic performance of CSTB score is better than that of the clinical model combining three independent prognostic indicators, molecular subtype, tumor size, and lymph nodal metastasis (AUC, Training dataset: 0.773 vs. 0.749; External validation: 0.753 vs. 0.724; HR, Training dataset: 4.18 vs. 3.92; External validation: 4.98 vs. 4.16), and as an auxiliary indicator, it can greatly improve the accuracy of prognostic prediction. And furthermore, a nomogram combining the CSTB score with the clinical model is established for prognosis prediction and clinical decision making.

CONCLUSION

This standardized and automated imaging prognosticator may convince pathologists to adopt it as a prognostic factor, thereby customizing more effective treatment plans for patients.

Immunohistochemical Stain-Aided Annotation Accelerates Machine Learning and Deep Learning Model Development in the Pathologic Diagnosis of Nasopharyngeal Carcinoma

Nasopharyngeal carcinoma (NPC) is an epithelial cancer originating in the nasopharynx epithelium. Nevertheless, annotating pathology slides remains a bottleneck in the development of AI-driven pathology models and applications. In the present study, we aim to demonstrate the feasibility of using immunohistochemistry (IHC) for annotation by non-pathologists and to develop an efficient model for distinguishing NPC without the time-consuming involvement of pathologists. For this study, we gathered NPC slides from 251 different patients, comprising hematoxylin and eosin (H&E) slides, pan-cytokeratin (Pan-CK) IHC slides, and Epstein-Barr virus-encoded small RNA (EBER) slides. The annotation of NPC regions in the H&E slides was carried out by a non-pathologist trainee who had access to corresponding Pan-CK IHC slides, both with and without EBER slides. The training process utilized ResNeXt, a deep neural network featuring a residual and inception architecture. In the validation set, NPC exhibited an AUC of 0.896, with a sensitivity of 0.919 and a specificity of 0.878. This study represents a significant breakthrough: the successful application of deep convolutional neural networks to identify NPC without the need for expert pathologist annotations. Our results underscore the potential of laboratory techniques to substantially reduce the workload of pathologists.

Revolutionizing chronic lymphocytic leukemia diagnosis: A deep dive into the diverse applications of machine learning

Chronic lymphocytic leukemia (CLL) is a B cell neoplasm characterized by the accumulation of aberrant monoclonal B lymphocytes. CLL is the predominant type of leukemia in Western countries, accounting for 25% of cases. Although many patients remain asymptomatic, a subset may exhibit typical lymphoma symptoms, acquired immunodeficiency disorders, or autoimmune complications. Diagnosis involves blood tests showing increased lymphocytes and further examination using peripheral blood smear and flow cytometry to confirm the disease. With the significant advancements in machine learning (ML) and artificial intelligence (AI) in recent years, numerous models and algorithms have been proposed to support the diagnosis and classification of CLL. In this review, we discuss the benefits and drawbacks of recent applications of ML algorithms in the diagnosis and evaluation of patients diagnosed with CLL.

Deep Learning for Detecting BRCA Mutations in High-Grade Ovarian Cancer Based on an Innovative Tumor Segmentation Method From Whole Slide Images

BRCA1 and BRCA2 genes play a crucial role in repairing DNA double-strand breaks through homologous recombination. Their mutations represent a significant proportion of homologous recombination deficiency and are a reliable effective predictor of sensitivity of high-grade ovarian cancer (HGOC) to poly(ADP-ribose) polymerase inhibitors. However, their testing by next-generation sequencing is costly and time-consuming and can be affected by various preanalytical factors. In this study, we present a deep learning classifier for BRCA mutational status prediction from hematoxylin-eosin-safran-stained whole slide images (WSI) of HGOC. We constituted the OvarIA cohort composed of 867 patients with HGOC with known BRCA somatic mutational status from 2 different pathology departments. We first developed a tumor segmentation model according to dynamic sampling and then trained a visual representation encoder with momentum contrastive learning on the predicted tumor tiles. We finally trained a BRCA classifier on more than a million tumor tiles in multiple instance learning with an attention-based mechanism. The tumor segmentation model trained on 8 WSI obtained a dice score of 0.915 and an intersection-over-union score of 0.847 on a test set of 50 WSI, while the BRCA classifier achieved the state-of-the-art area under the receiver operating characteristic curve of 0.739 in 5-fold cross-validation and 0.681 on the testing set. An additional multiscale approach indicates that the relevant information for predicting BRCA mutations is located more in the tumor context than in the cell morphology. Our results suggest that BRCA somatic mutations have a discernible phenotypic effect that could be detected by deep learning and could be used as a prescreening tool in the future.

Artificial intelligence in diagnostic pathology

Digital pathology (DP) is being increasingly employed in cancer diagnostics, providing additional tools for faster, higher-quality, accurate diagnosis. The practice of diagnostic pathology has gone through a staggering transformation wherein new tools such as digital imaging, advanced artificial intelligence (AI) algorithms, and computer-aided diagnostic techniques are being used for assisting, augmenting and empowering the computational histopathology and AI-enabled diagnostics. This is paving the way for advancement in precision medicine in cancer. Automated whole slide imaging (WSI) scanners are now rendering diagnostic quality, high-resolution images of entire glass slides and combining these images with innovative digital pathology tools is making it possible to integrate imaging into all aspects of pathology reporting including anatomical, clinical, and molecular pathology. The recent approvals of WSI scanners for primary diagnosis by the FDA as well as the approval of prostate AI algorithm has paved the way for starting to incorporate this exciting technology for use in primary diagnosis. AI tools can provide a unique platform for innovations and advances in anatomical and clinical pathology workflows. In this review, we describe the milestones and landmark trials in the use of AI in clinical pathology with emphasis on future directions.

Delineating intra-tumoral heterogeneity and tumor evolution in breast cancer using precision-based approaches

The burden of breast cancer continues to increase worldwide as it remains the most diagnosed tumor in females and the second leading cause of cancer-related deaths. Breast cancer is a heterogeneous disease characterized by different subtypes which are driven by aberrations in key genes such as BRCA1 and BRCA2, and hormone receptors. However, even within each subtype, heterogeneity that is driven by underlying evolutionary mechanisms is suggested to underlie poor response to therapy, variance in disease progression, recurrence, and relapse. Intratumoral heterogeneity highlights that the evolvability of tumor cells depends on interactions with cells of the tumor microenvironment. The complexity of the tumor microenvironment is being unraveled by recent advances in screening technologies such as high throughput sequencing; however, there remain challenges that impede the practical use of these approaches, considering the underlying biology of the tumor microenvironment and the impact of selective pressures on the evolvability of tumor cells. In this review, we will highlight the advances made thus far in defining the molecular heterogeneity in breast cancer and the implications thereof in diagnosis, the design and application of targeted therapies for improved clinical outcomes. We describe the different precision-based approaches to diagnosis and treatment and their prospects. We further propose that effective cancer diagnosis and treatment are dependent on unpacking the tumor microenvironment and its role in driving intratumoral heterogeneity. Underwriting such heterogeneity are Darwinian concepts of natural selection that we suggest need to be taken into account to ensure evolutionarily informed therapeutic decisions.

Prediction of Ovarian Cancer Response to Therapy Based on Deep Learning Analysis of Histopathology Images

BACKGROUND

Ovarian cancer remains the leading gynecological cause of cancer mortality. Predicting the sensitivity of ovarian cancer to chemotherapy at the time of pathological diagnosis is a goal of precision medicine research that we have addressed in this study using a novel deep-learning neural network framework to analyze the histopathological images.

METHODS

We have developed a method based on the Inception V3 deep learning algorithm that complements other methods for predicting response to standard platinum-based therapy of the disease. For the study, we used histopathological H&E images (pre-treatment) of high-grade serous carcinoma from The Cancer Genome Atlas (TCGA) Genomic Data Commons portal to train the Inception V3 convolutional neural network system to predict whether cancers had independently been labeled as sensitive or resistant to subsequent platinum-based chemotherapy. The trained model was then tested using data from patients left out of the training process. We used receiver operating characteristic (ROC) and confusion matrix analyses to evaluate model performance and Kaplan-Meier survival analysis to correlate the predicted probability of resistance with patient outcome. Finally, occlusion sensitivity analysis was piloted as a start toward correlating histopathological features with a response.

RESULTS

The study dataset consisted of 248 patients with stage 2 to 4 serous ovarian cancer. For a held-out test set of forty patients, the trained deep learning network model distinguished sensitive from resistant cancers with an area under the curve (AUC) of 0.846 ± 0.009 (SE). The probability of resistance calculated from the deep-learning network was also significantly correlated with patient survival and progression-free survival. In confusion matrix analysis, the network classifier achieved an overall predictive accuracy of 85% with a sensitivity of 73% and specificity of 90% for this cohort based on the Youden-J cut-off. Stage, grade, and patient age were not statistically significant for this cohort size. Occlusion sensitivity analysis suggested histopathological features learned by the network that may be associated with sensitivity or resistance to the chemotherapy, but multiple marker studies will be necessary to follow up on those preliminary results.

CONCLUSIONS

This type of analysis has the potential, if further developed, to improve the prediction of response to therapy of high-grade serous ovarian cancer and perhaps be useful as a factor in deciding between platinum-based and other therapies. More broadly, it may increase our understanding of the histopathological variables that predict response and may be adaptable to other cancer types and imaging modalities.

DeepBLS: Deep Feature-Based Broad Learning System for Tissue Phenotyping in Colorectal Cancer WSIs

Tissue phenotyping is a fundamental step in computational pathology for the analysis of tumor micro-environment in whole slide images (WSIs). Automatic tissue phenotyping in whole slide images (WSIs) of colorectal cancer (CRC) assists pathologists in better cancer grading and prognostication. In this paper, we propose a novel algorithm for the identification of distinct tissue components in colon cancer histology images by blending a comprehensive learning system with deep features extraction in the current work. Firstly, we extracted the features from the pre-trained VGG19 network which are then transformed into mapped features space for nodes enhancement generation. Utilizing both mapped features and enhancement nodes, the proposed algorithm classifies seven distinct tissue components including stroma, tumor, complex stroma, necrotic, normal benign, lymphocytes, and smooth muscle. To validate our proposed model, the experiments are performed on two publicly available colorectal cancer histology datasets. We showcase that our approach achieves a remarkable performance boost surpassing existing state-of-the-art methods by (1.3% AvTP, 2% F1) and (7% AvTP, 6% F1) on CRCD-1, and CRCD-2, respectively.

Heterogeneity of the tumor immune microenvironment and clinical interventions

The tumor immune microenvironment (TIME) is broadly composed of various immune cells, and its heterogeneity is characterized by both immune cells and stromal cells. During the course of tumor formation and progression and anti-tumor treatment, the composition of the TIME becomes heterogeneous. Such immunological heterogeneity is not only present between populations but also exists on temporal and spatial scales. Owing to the existence of TIME, clinical outcomes can differ when a similar treatment strategy is provided to patients. Therefore, a comprehensive assessment of TIME heterogeneity is essential for developing precise and effective therapies. Facilitated by advanced technologies, it is possible to understand the complexity and diversity of the TIME and its influence on therapy responses. In this review, we discuss the potential reasons for TIME heterogeneity and the current approaches used to explore it. We also summarize clinical intervention strategies based on associated mechanisms or targets to control immunological heterogeneity.

Modelling the Tumour Microenvironment, but What Exactly Do We Mean by "Model"?

The Oxford English Dictionary includes 17 definitions for the word "model" as a noun and another 11 as a verb. Therefore, context is necessary to understand the meaning of the word model. For instance, "model railways" refer to replicas of railways and trains at a smaller scale and a "model student" refers to an exemplary individual. In some cases, a specific context, like cancer research, may not be sufficient to provide one specific meaning for model. Even if the context is narrowed, specifically, to research related to the tumour microenvironment, "model" can be understood in a wide variety of ways, from an animal model to a mathematical expression. This paper presents a review of different "models" of the tumour microenvironment, as grouped by different definitions of the word into four categories: model organisms, in vitro models, mathematical models and computational models. Then, the frequencies of different meanings of the word "model" related to the tumour microenvironment are measured from numbers of entries in the MEDLINE database of the United States National Library of Medicine at the National Institutes of Health. The frequencies of the main components of the microenvironment and the organ-related cancers modelled are also assessed quantitatively with specific keywords. Whilst animal models, particularly xenografts and mouse models, are the most commonly used "models", the number of these entries has been slowly decreasing. Mathematical models, as well as prognostic and risk models, follow in frequency, and these have been growing in use.

Recent advances of pathomics in colorectal cancer diagnosis and prognosis

Colorectal cancer (CRC) is one of the most common malignancies, with the third highest incidence and the second highest mortality in the world. To improve the therapeutic outcome, the risk stratification and prognosis predictions would help guide clinical treatment decisions. Achieving these goals have been facilitated by the fast development of artificial intelligence (AI) -based algorithms using radiological and pathological data, in combination with genomic information. Among them, features extracted from pathological images, termed pathomics, are able to reflect sub-visual characteristics linking to better stratification and prediction of therapeutic responses. In this paper, we review recent advances in pathological image-based algorithms in CRC, focusing on diagnosis of benign and malignant lesions, micro-satellite instability, as well as prediction of neoadjuvant chemoradiotherapy and the prognosis of CRC patients.

Application of Artificial Intelligence in the Diagnosis, Treatment, and Prognostic Evaluation of Mediastinal Malignant Tumors

Artificial intelligence (AI), also known as machine intelligence, is widely utilized in the medical field, promoting medical advances. Malignant tumors are the critical focus of medical research and improvement of clinical diagnosis and treatment. Mediastinal malignancy is an important tumor that attracts increasing attention today due to the difficulties in treatment. Combined with artificial intelligence, challenges from drug discovery to survival improvement are constantly being overcome. This article reviews the progress of the use of AI in the diagnosis, treatment, and prognostic prospects of mediastinal malignant tumors based on current literature findings.

Deep transfer learning for IDC breast cancer detection using fast AI technique and Sqeezenet architecture

One of the most effective approaches for identifying breast cancer is histology, which is the meticulous inspection of tissues under a microscope. The kind of cancer cells, or whether they are cancerous (malignant) or non-cancerous, is typically determined by the type of tissue that is analyzed by the test performed by the technician (benign). The goal of this study was to automate IDC classification within breast cancer histology samples using a transfer learning technique. To improve our outcomes, we combined a Gradient Color Activation Mapping (Grad CAM) and image coloring mechanism with a discriminative fine-tuning methodology employing a one-cycle strategy using FastAI techniques. There have been lots of research studies related to deep transfer learning which use the same mechanism, but this report uses a transfer learning mechanism based on lightweight Squeeze Net architecture, a variant of CNN (Convolution neural network). This strategy demonstrates that fine-tuning on Squeeze Net makes it possible to achieve satisfactory results when transitioning generic features from natural images to medical images.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

A new AI assisted medical molecular image diagnostic model

In recent times, ML algorithms that plays a significant role right from drug discovery to clinical decision making. The recent advances in DL technologies contribute towards improved performance for carrying out computer aided medical image analysis and disease diagnosis. The key benefit of AI in processing of medical big data offers spectacular insights into the hierarchal relationships that exist among data which can be algorithmically explored thus replacing the tedious manual processes to extract and localize specific areas of interests in medical images thus considerably changing the way medicine has been practiced so far. In bio medical related clinical applications, there is a constant demand pertaining the research and development with respect to deploying AI as a mainstream tool to perform several medical imaging activities like analysis, diagnosis, segmentation as well as classification. The increased usage of electronic health records and medical images being its integral component the need for appropriate and efficient AI assisted medical image analysis system that takes care of accurate and automated decision making could be of great help to radiologists and medical practitioners. Molecular image analysis is a dynamic field that makes use of ML and DL algorithms that utilizes labeled and structured information which also proves to be helpful to the patients as they serve as an initial interface before further diagnosis and treatments. Thus our research aims to offer a novel and efficient AI based medical analysis system that can assist clinical practitioners to focus on enhancing the disease diagnosis through DL based medical image analysis and decision making. In addition, we also address specific challenges related to disease diagnosis and propose novel GAN model for improved diagnosis and implementation. Our proposed technique can also be generalized to generate synthetic data for further issues related to molecular image analysis in the field of medicine and help towards building a better disease diagnosis model.

Multicenter automatic detection of invasive carcinoma on breast whole slide images

Breast cancer is one of the most prevalent cancers worldwide and pathologists are closely involved in establishing a diagnosis. Tools to assist in making a diagnosis are required to manage the increasing workload. In this context, artificial intelligence (AI) and deep-learning based tools may be used in daily pathology practice. However, it is challenging to develop fast and reliable algorithms that can be trusted by practitioners, whatever the medical center. We describe a patch-based algorithm that incorporates a convolutional neural network to detect and locate invasive carcinoma on breast whole-slide images. The network was trained on a dataset extracted from a reference acquisition center. We then performed a calibration step based on transfer learning to maintain the performance when translating on a new target acquisition center by using a limited amount of additional training data. Performance was evaluated using classical binary measures (accuracy, recall, precision) for both centers (referred to as "test reference dataset" and "test target dataset") and at two levels: patch and slide level. At patch level, accuracy, recall, and precision of the model on the reference and target test sets were 92.1% and 96.3%, 95% and 87.8%, and 73.9% and 70.6%, respectively. At slide level, accuracy, recall, and precision were 97.6% and 92.0%, 90.9% and 100%, and 100% and 70.8% for test sets 1 and 2, respectively. The high performance of the algorithm at both centers shows that the calibration process is efficient. This is performed using limited training data from the new target acquisition center and requires that the model is trained beforehand on a large database from a reference center. This methodology allows the implementation of AI diagnostic tools to help in routine pathology practice.

Colorectal cancer lymph node metastasis prediction with weakly supervised transformer-based multi-instance learning

Lymph node metastasis examined by the resected lymph nodes is considered one of the most important prognostic factors for colorectal cancer (CRC). However, it requires careful and comprehensive inspection by expert pathologists. To relieve the pathologists' burden and speed up the diagnostic process, in this paper, we develop a deep learning system with the binary positive/negative labels of the lymph nodes to solve the CRC lymph node classification task. The multi-instance learning (MIL) framework is adopted in our method to handle the whole slide images (WSIs) of gigapixels in size at once and get rid of the labor-intensive and time-consuming detailed annotations. First, a transformer-based MIL model, DT-DSMIL, is proposed in this paper based on the deformable transformer backbone and the dual-stream MIL (DSMIL) framework. The local-level image features are extracted and aggregated with the deformable transformer, and the global-level image features are obtained with the DSMIL aggregator. The final classification decision is made based on both the local and the global-level features. After the effectiveness of our proposed DT-DSMIL model is demonstrated by comparing its performance with its predecessors, a diagnostic system is developed to detect, crop, and finally identify the single lymph nodes within the slides based on the DT-DSMIL and the Faster R-CNN model. The developed diagnostic model is trained and tested on a clinically collected CRC lymph node metastasis dataset composed of 843 slides (864 metastasis lymph nodes and 1415 non-metastatic lymph nodes), achieving the accuracy of 95.3% and the area under the receiver operating characteristic curve (AUC) of 0.9762 (95% confidence interval [CI]: 0.9607-0.9891) for the single lymph node classification. As for the lymph nodes with micro-metastasis and macro-metastasis, our diagnostic system achieves the AUC of 0.9816 (95% CI: 0.9659-0.9935) and 0.9902 (95% CI: 0.9787-0.9983), respectively. Moreover, the system shows reliable diagnostic region localizing performance: the model can always identify the most likely metastases, no matter the model's predictions or manual labels, showing great potential in avoiding false negatives and discovering incorrectly labeled slides in actual clinical use.

FabNet: A Features Agglomeration-Based Convolutional Neural Network for Multiscale Breast Cancer Histopathology Images Classification

The definitive diagnosis of histology specimen images is largely based on the radiologist's comprehensive experience; however, due to the fine to the coarse visual appearance of such images, experts often disagree with their assessments. Sophisticated deep learning approaches can help to automate the diagnosis process of the images and reduce the analysis duration. More efficient and accurate automated systems can also increase the diagnostic impartiality by reducing the difference between the operators. We propose a FabNet model that can learn the fine-to-coarse structural and textural features of multi-scale histopathological images by using accretive network architecture that agglomerate hierarchical feature maps to acquire significant classification accuracy. We expand on a contemporary design by incorporating deep and close integration to finely combine features across layers. Our deep layer accretive model structure combines the feature hierarchy in an iterative and hierarchically manner that infers higher accuracy and fewer parameters. The FabNet can identify malignant tumors from images and patches from histopathology images. We assessed the efficiency of our suggested model standard cancer datasets, which included breast cancer as well as colon cancer histopathology images. Our proposed avant garde model significantly outperforms existing state-of-the-art models in respect of the accuracy, F1 score, precision, and sensitivity, with fewer parameters.

Deep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction

Breast cancer is one of the precarious conditions that affect women, and a substantive cure has not yet been discovered for it. With the advent of Artificial intelligence (AI), recently, deep learning techniques have been used effectively in breast cancer detection, facilitating early diagnosis and therefore increasing the chances of patients' survival. Compared to classical machine learning techniques, deep learning requires less human intervention for similar feature extraction. This study presents a systematic literature review on the deep learning-based methods for breast cancer detection that can guide practitioners and researchers in understanding the challenges and new trends in the field. Particularly, different deep learning-based methods for breast cancer detection are investigated, focusing on the genomics and histopathological imaging data. The study specifically adopts the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which offer a detailed analysis and synthesis of the published articles. Several studies were searched and gathered, and after the eligibility screening and quality evaluation, 98 articles were identified. The results of the review indicated that the Convolutional Neural Network (CNN) is the most accurate and extensively used model for breast cancer detection, and the accuracy metrics are the most popular method used for performance evaluation. Moreover, datasets utilized for breast cancer detection and the evaluation metrics are also studied. Finally, the challenges and future research direction in breast cancer detection based on deep learning models are also investigated to help researchers and practitioners acquire in-depth knowledge of and insight into the area.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

Application of Deep Learning in Histopathology Images of Breast Cancer: A Review

With the development of artificial intelligence technology and computer hardware functions, deep learning algorithms have become a powerful auxiliary tool for medical image analysis. This study was an attempt to use statistical methods to analyze studies related to the detection, segmentation, and classification of breast cancer in pathological images. After an analysis of 107 articles on the application of deep learning to pathological images of breast cancer, this study is divided into three directions based on the types of results they report: detection, segmentation, and classification. We introduced and analyzed models that performed well in these three directions and summarized the related work from recent years. Based on the results obtained, the significant ability of deep learning in the application of breast cancer pathological images can be recognized. Furthermore, in the classification and detection of pathological images of breast cancer, the accuracy of deep learning algorithms has surpassed that of pathologists in certain circumstances. Our study provides a comprehensive review of the development of breast cancer pathological imaging-related research and provides reliable recommendations for the structure of deep learning network models in different application scenarios.

Validation and real-world clinical application of an artificial intelligence algorithm for breast cancer detection in biopsies

Breast cancer is the most common malignant disease worldwide, with over 2.26 million new cases in 2020. Its diagnosis is determined by a histological review of breast biopsy specimens, which can be labor-intensive, subjective, and error-prone. Artificial Intelligence (AI)-based tools can support cancer detection and classification in breast biopsies ensuring rapid, accurate, and objective diagnosis. We present here the development, external clinical validation, and deployment in routine use of an AI-based quality control solution for breast biopsy review. The underlying AI algorithm is trained to identify 51 different types of clinical and morphological features, and it achieves very high accuracy in a large, multi-site validation study. Specifically, the area under the receiver operating characteristic curves (AUC) for the detection of invasive carcinoma and of ductal carcinoma in situ (DCIS) are 0.99 (specificity and sensitivity of 93.57 and 95.51%, respectively) and 0.98 (specificity and sensitivity of 93.79 and 93.20% respectively), respectively. The AI algorithm differentiates well between subtypes of invasive and different grades of in situ carcinomas with an AUC of 0.97 for invasive ductal carcinoma (IDC) vs. invasive lobular carcinoma (ILC) and AUC of 0.92 for DCIS high grade vs. low grade/atypical ductal hyperplasia, respectively, as well as accurately identifies stromal tumor-infiltrating lymphocytes (TILs) with an AUC of 0.965. Deployment of this AI solution as a real-time quality control solution in clinical routine leads to the identification of cancers initially missed by the reviewing pathologist, demonstrating both clinical utility and accuracy in real-world clinical application.

Breast Cancer Dataset, Classification and Detection Using Deep Learning

Incorporating scientific research into clinical practice via clinical informatics, which includes genomics, proteomics, bioinformatics, and biostatistics, improves patients' treatment. Computational pathology is a growing subspecialty with the potential to integrate whole slide images, multi-omics data, and health informatics. Pathology and laboratory medicine are critical to diagnosing cancer. This work will review existing computational and digital pathology methods for breast cancer diagnosis with a special focus on deep learning. The paper starts by reviewing public datasets related to breast cancer diagnosis. Additionally, existing deep learning methods for breast cancer diagnosis are reviewed. The publicly available code repositories are introduced as well. The paper is closed by highlighting challenges and future works for deep learning-based diagnosis.

Predicting neoadjuvant chemotherapy benefit using deep learning from stromal histology in breast cancer

Neoadjuvant chemotherapy (NAC) is a standard treatment option for locally advanced breast cancer. However, not all patients benefit from NAC; some even obtain worse outcomes after therapy. Hence, predictors of treatment benefit are crucial for guiding clinical decision-making. Here, we investigated the predictive potential of breast cancer stromal histology via a deep learning (DL)-based approach and proposed the tumor-associated stroma score (TS-score) for predicting pathological complete response (pCR) to NAC with a multicenter dataset. The TS-score was demonstrated to be an independent predictor of pCR, and it not only outperformed the baseline variables and stromal tumor-infiltrating lymphocytes (sTILs) but also significantly improved the prediction performance of the baseline variable-based model. Furthermore, we discovered that unlike lymphocytes, collagen and fibroblasts in the stroma were likely associated with a poor response to NAC. The TS-score has the potential to better stratify breast cancer patients in NAC settings.

Deep Learning Approaches in Histopathology

The revolution of artificial intelligence and its impacts on our daily life has led to tremendous interest in the field and its related subtypes: machine learning and deep learning. Scientists and developers have designed machine learning- and deep learning-based algorithms to perform various tasks related to tumor pathologies, such as tumor detection, classification, grading with variant stages, diagnostic forecasting, recognition of pathological attributes, pathogenesis, and genomic mutations. Pathologists are interested in artificial intelligence to improve the diagnosis precision impartiality and to minimize the workload combined with the time consumed, which affects the accuracy of the decision taken. Regrettably, there are already certain obstacles to overcome connected to artificial intelligence deployments, such as the applicability and validation of algorithms and computational technologies, in addition to the ability to train pathologists and doctors to use these machines and their willingness to accept the results. This review paper provides a survey of how machine learning and deep learning methods could be implemented into health care providers' routine tasks and the obstacles and opportunities for artificial intelligence application in tumor morphology.

Deep learning-based breast cancer grading and survival analysis on whole-slide histopathology images

Breast cancer tumor grade is strongly associated with patient survival. In current clinical practice, pathologists assign tumor grade after visual analysis of tissue specimens. However, different studies show significant inter-observer variation in breast cancer grading. Computer-based breast cancer grading methods have been proposed but only work on specifically selected tissue areas and/or require labor-intensive annotations to be applied to new datasets. In this study, we trained and evaluated a deep learning-based breast cancer grading model that works on whole-slide histopathology images. The model was developed using whole-slide images from 706 young (< 40 years) invasive breast cancer patients with corresponding tumor grade (low/intermediate vs. high), and its constituents nuclear grade, tubule formation and mitotic rate. The performance of the model was evaluated using Cohen's kappa on an independent test set of 686 patients using annotations by expert pathologists as ground truth. The predicted low/intermediate (n = 327) and high (n = 359) grade groups were used to perform survival analysis. The deep learning system distinguished low/intermediate versus high tumor grade with a Cohen's Kappa of 0.59 (80% accuracy) compared to expert pathologists. In subsequent survival analysis the two groups predicted by the system were found to have a significantly different overall survival (OS) and disease/recurrence-free survival (DRFS/RFS) (p < 0.05). Univariate Cox hazard regression analysis showed statistically significant hazard ratios (p < 0.05). After adjusting for clinicopathologic features and stratifying for molecular subtype the hazard ratios showed a trend but lost statistical significance for all endpoints. In conclusion, we developed a deep learning-based model for automated grading of breast cancer on whole-slide images. The model distinguishes between low/intermediate and high grade tumors and finds a trend in the survival of the two predicted groups.

Global research trends and foci of artificial intelligence-based tumor pathology: a scientometric study

Background

With the development of digital pathology and the renewal of deep learning algorithm, artificial intelligence (AI) is widely applied in tumor pathology. Previous researches have demonstrated that AI-based tumor pathology may help to solve the challenges faced by traditional pathology. This technology has attracted the attention of scholars in many fields and a large amount of articles have been published. This study mainly summarizes the knowledge structure of AI-based tumor pathology through bibliometric analysis, and discusses the potential research trends and foci.

Methods

Publications related to AI-based tumor pathology from 1999 to 2021 were selected from Web of Science Core Collection. VOSviewer and Citespace were mainly used to perform and visualize co-authorship, co-citation, and co-occurrence analysis of countries, institutions, authors, references and keywords in this field.

Results

A total of 2753 papers were included. The papers on AI-based tumor pathology research had been continuously increased since 1999. The United States made the largest contribution in this field, in terms of publications (1138, 41.34%), H-index (85) and total citations (35,539 times). We identified the most productive institution and author were Harvard Medical School and Madabhushi Anant, while Jemal Ahmedin was the most co-cited author. Scientific Reports was the most prominent journal and after analysis, Lecture Notes in Computer Science was the journal with highest total link strength. According to the result of references and keywords analysis, “breast cancer histopathology” “convolutional neural network” and “histopathological image” were identified as the major future research foci.

Conclusions

AI-based tumor pathology is in the stage of vigorous development and has a bright prospect. International transboundary cooperation among countries and institutions should be strengthened in the future. It is foreseeable that more research foci will be lied in the interpretability of deep learning-based model and the development of multi-modal fusion model.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12967-022-03615-0.

Deep learning-based fully automated differential diagnosis of eyelid basal cell and sebaceous carcinoma using whole slide images

Background

The differential diagnosis of eyelid basal cell carcinoma (BCC) and sebaceous carcinoma (SC) is highly dependent on pathologist’s experience. Herein, we proposed a fully automated differential diagnostic method, which used deep learning (DL) to accurately classify eyelid BCC and SC based on whole slide images (WSIs).

Methods

We used 116 haematoxylin and eosin (H&E)-stained sections from 116 eyelid BCC patients and 180 H&E-stained sections from 129 eyelid SC patients treated at the Shanghai Ninth People’s Hospital from 2017 to 2019. The method comprises two stages: patch prediction by the DenseNet-161 architecture-based DL model and WSI differentiation by an average-probability strategy-based integration module, and its differential performance was assessed by the carcinoma differentiation accuracy and F1 score. We compared the classification performance of the method with that of three pathologists, two junior and one senior. To validate the auxiliary value of the method, we compared the pathologists’ BCC and SC classification with and without the assistance of our proposed method.

Results

Our proposed method achieved an accuracy of 0.983, significantly higher than that of the three pathologists (0.644 and 0.729 for the two junior pathologists and 0.831 for the senior pathologist). With the method’s assistance, the pathologists’ accuracy increased significantly (P<0.05), by 28.8% and 15.2%, respectively, for the two junior pathologists and by 11.8% for the senior pathologist.

Conclusions

Our proposed method accurately classifies eyelid BCC and SC and effectively improves the diagnostic accuracy of pathologists. It may therefore facilitate the development of appropriate and timely therapeutic plans.

Artificial Intelligence-Based Tissue Phenotyping in Colorectal Cancer Histopathology Using Visual and Semantic Features Aggregation

Tissue phenotyping of the tumor microenvironment has a decisive role in digital profiling of intra-tumor heterogeneity, epigenetics, and progression of cancer. Most of the existing methods for tissue phenotyping often rely on time-consuming and error-prone manual procedures. Recently, with the advent of advanced technologies, these procedures have been automated using artificial intelligence techniques. In this paper, a novel deep histology heterogeneous feature aggregation network (HHFA-Net) is proposed based on visual and semantic information fusion for the detection of tissue phenotypes in colorectal cancer (CRC). We adopted and tested various data augmentation techniques to avoid computationally expensive stain normalization procedures and handle limited and imbalanced data problems. Three publicly available datasets are used in the experiments: CRC tissue phenotyping (CRC-TP), CRC histology (CRCH), and colon cancer histology (CCH). The proposed HHFA-Net achieves higher accuracies than the state-of-the-art methods for tissue phenotyping in CRC histopathology images.

Quantitative analysis of breast cancer tissue composition and associations with tumor subtype

The tumor microenvironment is an important determinant of breast cancer progression, but standard methods for describing the tumor microenvironment are lacking. Measures of microenvironment composition such as stromal area and immune infiltrate are labor-intensive and few large studies have systematically collected this data. However, digital histologic approaches are becoming more widely available, allowing high-throughput, quantitative estimation. We applied such methods to tissue microarrays of tumors from 1687 women (mean 4 cores per case) in the Carolina Breast Cancer Study Phase 3. Tumor composition was quantified as percentage of epithelium, stroma, adipose, and lymphocytic infiltrate (with the latter as presence/absence using a ≥1% cutoff). Composition proportions and presence/absence were evaluated in association with clinical and molecular features of breast cancer (intrinsic subtype and RNA-based risk of recurrence [ROR] scores) using multivariable linear and logistic regression. Lower stromal content was associated with aggressive tumor phenotypes, including triple-negative (31.1% vs. 41.6% in HR+/HER2-; RFD [95% CI]: -10.5%, [-13.1, -7.9]), Basal-like subtypes (29.0% vs. 44.0% in Luminal A; RFD [95% CI]: -14.9%, [-17.8, -12.0]), and high RNA-based PAM50 ROR scores (27.6% vs. 48.1% in ROR low; RFD [95% CI]: -20.5%, [24.3, 16.7]), after adjusting for age and race. HER2+ tumors also had lower stromal content, particularly among RNA-based HER2-enriched (35.2% vs. 44.0% in Luminal A; RFD [95% CI]: -8.8%, [-13.8, -3.8]). Similar associations were observed between immune infiltrate and tumor phenotypes. Quantitative digital image analysis of the breast cancer microenvironment showed significant associations with demographic characteristics and biological indicators of aggressive behavior.

Rapid screening using pathomorphological interpretation to detect BRAFV600E mutation and microsatellite instability in colorectal cancer

PURPOSE

Rapid decision-making is essential in precision medicine for initiating molecular targeted therapy for cancer patients. This study aimed to extract pathomorphological features that enable the accurate prediction of genetic abnormalities in cancer from hematoxylin and eosin (H&E) images using deep learning (DL).

EXPERIMENTAL DESIGN

A total of 1,657 images (one representative image per patient) of thin formalin-fixed, paraffin-embedded (FFPE) tissue sections from either primary or metastatic tumors with next-generation sequencing (NGS)-confirmed genetic abnormalities-including BRAFV600E and KRAS mutations, and microsatellite instability high (MSI-H)-that are directly relevant to therapeutic strategies for advanced colorectal cancer (CRC) were obtained from the nationwide SCRUM-Japan GI-SCREEN project. The images were divided into three groups of 986, 248, and 423 images to create one training and two validation cohorts, respectively. Pathomorphological feature-prediction DL models were first developed based on pathomorphological features. Subsequently, gene-prediction DL models were constructed for all possible combinations of pathomorphological features that enabled the predicting of gene abnormalities based on images filtered by the combination of pathomorphological feature-prediction models.

RESULTS

High accuracies were achieved, with areas under the curve (AUCs) &gt; 0.90 and 0.80 for 12 and 27, respectively, of 33 analyzed pathomorphological features, with high AUCs being yielded for both BRAFV600E (0.851 and 0.859) and MSI-H (0.923 and 0.862).

CONCLUSIONS

These findings show that novel next-generation pathology methods can predict genetic abnormalities without the need for standard-of-care gene tests and this novel next-generation pathology method can be applied for CRC treatment planning in the near future.

The Role of Artificial Intelligence in Early Cancer Diagnosis

Improving the proportion of patients diagnosed with early-stage cancer is a key priority of the World Health Organisation. In many tumour groups, screening programmes have led to improvements in survival, but patient selection and risk stratification are key challenges. In addition, there are concerns about limited diagnostic workforces, particularly in light of the COVID-19 pandemic, placing a strain on pathology and radiology services. In this review, we discuss how artificial intelligence algorithms could assist clinicians in (1) screening asymptomatic patients at risk of cancer, (2) investigating and triaging symptomatic patients, and (3) more effectively diagnosing cancer recurrence. We provide an overview of the main artificial intelligence approaches, including historical models such as logistic regression, as well as deep learning and neural networks, and highlight their early diagnosis applications. Many data types are suitable for computational analysis, including electronic healthcare records, diagnostic images, pathology slides and peripheral blood, and we provide examples of how these data can be utilised to diagnose cancer. We also discuss the potential clinical implications for artificial intelligence algorithms, including an overview of models currently used in clinical practice. Finally, we discuss the potential limitations and pitfalls, including ethical concerns, resource demands, data security and reporting standards.

Molecular Pathology of Lung Cancer

This overview of the molecular pathology of lung cancer includes a review of the most salient molecular alterations of the genome, transcriptome, and the epigenome. The insights provided by the growing use of next-generation sequencing (NGS) in lung cancer will be discussed, and interrelated concepts such as intertumor heterogeneity, intratumor heterogeneity, tumor mutational burden, and the advent of liquid biopsy will be explored. Moreover, this work describes how the evolving field of molecular pathology refines the understanding of different histologic phenotypes of non-small-cell lung cancer (NSCLC) and the underlying biology of small-cell lung cancer. This review will provide an appreciation for how ongoing scientific findings and technologic advances in molecular pathology are crucial for development of biomarkers, therapeutic agents, clinical trials, and ultimately improved patient care.

Deep Learning-Based Fully Automated Diagnosis of Melanocytic Lesions by Using Whole Slide Images

Background Erroneous diagnoses of melanocytic lesions (benign, atypical, and malignant types) result in inappropriate surgical treatment plans.Objective To propose a deep learning (DL)-based fully automated diagnostic method using whole slide images (WSIs) for melanocytic lesions.Methods The method consisted of patch prediction using a DL model and patient diagnosis using an aggregation module. The method was developed with 745 WSIs, and evaluated using internal and external testing sets comprising 182 WSIs and 54 WSIs, respectively. The results were compared with those of the classification by one junior and two senior pathologists. Furthermore, we compared the performance of the three pathologists in the classification of melanocytic lesions with and without the assistance of our method.Results The method achieved an accuracy of 0.963 and 0.930 on the internal and external testing sets, respectively, which was significantly higher than that of the junior pathologist (0.419 and 0.535). With assistance from the method, all three pathologists achieved higher accuracy on the internal and external testing sets; the accuracy of the junior pathologist increased by 39.0% and 30.2%, respectively (p < 0.05).Conclusion This generalizable method can accurately classify melanocytic lesions and effectively improve the diagnostic accuracy of pathologists.

Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review

Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.

Effectiveness of convolutional neural networks in the interpretation of pulmonary cytologic images in endobronchial ultrasound procedures

BACKGROUND

Rapid on-site cytologic evaluation (ROSE) helps to improve the diagnostic accuracy in endobronchial ultrasound (EBUS) procedures. However, cytologists are seldom available to perform ROSE in many institutions. Recent studies have investigated the application of deep learning in cytologic image analysis. As such, the present study analyzed lung cytologic images obtained by EBUS procedures, and employed deep-learning methods to distinguish between benign and malignant cells and to semantically segment malignant cells.

METHODS

Ninety-seven patients who underwent 104 EBUS procedures were enrolled. Four hundred and ninety-nine lung cytologic images obtained via ROSE, including 425 malignant and 74 benign, and most malignant were lung adenocarcinoma (64.3%). All the images were used to train a residual network model with 101 layers (ResNet101), with suitable hyperparameters selected to classify benign and malignant lung cytologic images. An HRNet model was also employed to mark the area of malignant cells. Automatic patch-cropping was adopted to facilitate dataset preparation.

RESULTS

Malignant cells were successfully classified by ResNet101 with 98.8% classification accuracy, 98.8% sensitivity, and 98.8% specificity in patch-based classification; 95.5% classification accuracy in image-based classification; and 92.9% classification accuracy in patient-based classification. Malignant cell area was successfully marked by HRNet with a mean intersection over union of 89.2%. The automatic cropping method enabled the system to complete diagnosis within 1 s.

CONCLUSIONS

This is the first study to combine lung cytologic image deep-learning classification with semantic segmentation. The model was optimized for high accuracy and the automatic cropping facilitates the clinical application of our model. The success in both lung cytologic images classification and semantic segmentation on our dataset shows a promising result for clinical application in the future.

JOINT AND INDIVIDUAL ANALYSIS OF BREAST CANCER HISTOLOGIC IMAGES AND GENOMIC COVARIATES

The two main approaches in the study of breast cancer are histopathology (analyzing visual characteristics of tumors) and genomics. While both histopathology and genomics are fundamental to cancer research, the connections between these fields have been relatively superficial. We bridge this gap by investigating the Carolina Breast Cancer Study through the development of an integrative, exploratory analysis framework. Our analysis gives insights - some known, some novel - that are engaging to both pathologists and geneticists. Our analysis framework is based on Angle-based Joint and Individual Variation Explained (AJIVE) for statistical data integration and exploits Convolutional Neural Networks (CNNs) as a powerful, automatic method for image feature extraction. CNNs raise interpretability issues that we address by developing novel methods to explore visual modes of variation captured by statistical algorithms (e.g. PCA or AJIVE) applied to CNN features.

Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review

BACKGROUND

Over the past decade, the development of molecular high-throughput methods (omics) increased rapidly and provided new insights for cancer research. In parallel, deep learning approaches revealed the enormous potential for medical image analysis, especially in digital pathology. Combining image and omics data with deep learning tools may enable the discovery of new cancer biomarkers and a more precise prediction of patient prognosis. This systematic review addresses different multimodal fusion methods of convolutional neural network-based image analyses with omics data, focussing on the impact of data combination on the classification performance.

METHODS

PubMed was screened for peer-reviewed articles published in English between January 2015 and June 2021 by two independent researchers. Search terms related to deep learning, digital pathology, omics, and multimodal fusion were combined.

RESULTS

We identified a total of 11 studies meeting the inclusion criteria, namely studies that used convolutional neural networks for haematoxylin and eosin image analysis of patients with cancer in combination with integrated omics data. Publications were categorised according to their endpoints: 7 studies focused on survival analysis and 4 studies on prediction of cancer subtypes, malignancy or microsatellite instability with spatial analysis.

CONCLUSIONS

Image-based classifiers already show high performances in prognostic and predictive cancer diagnostics. The integration of omics data led to improved performance in all studies described here. However, these are very early studies that still require external validation to demonstrate their generalisability and robustness. Further and more comprehensive studies with larger sample sizes are needed to evaluate performance and determine clinical benefits.

Mammary collagen architecture and its association with mammographic density and lesion severity among women undergoing image-guided breast biopsy

Background

Elevated mammographic breast density is a strong breast cancer risk factor with poorly understood etiology. Increased deposition of collagen, one of the main fibrous proteins present in breast stroma, has been associated with increased mammographic density. Collagen fiber architecture has been linked to poor outcomes in breast cancer. However, relationships of quantitative collagen fiber features assessed in diagnostic biopsies with mammographic density and lesion severity are not well-established.

Methods

Clinically indicated breast biopsies from 65 in situ or invasive breast cancer cases and 73 frequency matched-controls with a benign biopsy result were used to measure collagen fiber features (length, straightness, width, alignment, orientation and density (fibers/µm²)) using second harmonic generation microscopy in up to three regions of interest (ROIs) per biopsy: normal, benign breast disease, and cancer. Local and global mammographic density volumes were quantified in the ipsilateral breast in pre-biopsy full-field digital mammograms. Associations of fibrillar collagen features with mammographic density and severity of biopsy diagnosis were evaluated using generalized estimating equation models with an independent correlation structure to account for multiple ROIs within each biopsy section.

Results

Collagen fiber density was positively associated with the proportion of stroma on the biopsy slide (p < 0.001) and with local percent mammographic density volume at both the biopsy target (p = 0.035) and within a 2 mm perilesional ring (p = 0.02), but not with global mammographic density measures. As severity of the breast biopsy diagnosis increased at the ROI level, collagen fibers tended to be less dense, shorter, straighter, thinner, and more aligned with one another (p < 0.05).

Conclusions

Collagen fiber density was positively associated with local, but not global, mammographic density, suggesting that collagen microarchitecture may not translate into macroscopic mammographic features. However, collagen fiber features may be markers of cancer risk and/or progression among women referred for biopsy based on abnormal breast imaging.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13058-021-01482-z.

Convolutional neural network for human cancer types prediction by integrating protein interaction networks and omics data

Many studies have proven the power of gene expression profile in cancer identification, however, the explosive growth of genomics data increasing needs of tools for cancer diagnosis and prognosis in high accuracy and short times. Here, we collected 6136 human samples from 11 cancer types, and integrated their gene expression profiles and protein-protein interaction (PPI) network to generate 2D images with spectral clustering method. To predict normal samples and 11 cancer tumor types, the images of these 6136 human cancer network were separated into training and validation dataset to develop convolutional neural network (CNN). Our model showed 97.4% and 95.4% accuracies in identification of normal versus tumors and 11 cancer types, respectively. We also provided the results that tumors located in neighboring tissues or in the same cell types, would induce machine make error classification due to the similar gene expression profiles. Furthermore, we observed some patients may exhibit better prognosis if their tumors often misjudged into normal samples. As far as we know, we are the first to generate thousands of cancer networks to predict and classify multiple cancer types with CNN architecture. We believe that our model not only can be applied to cancer diagnosis and prognosis, but also promote the discovery of multiple cancer biomarkers.

Deep learning in cancer diagnosis, prognosis and treatment selection

Deep learning is a subdiscipline of artificial intelligence that uses a machine learning technique called artificial neural networks to extract patterns and make predictions from large data sets. The increasing adoption of deep learning across healthcare domains together with the availability of highly characterised cancer datasets has accelerated research into the utility of deep learning in the analysis of the complex biology of cancer. While early results are promising, this is a rapidly evolving field with new knowledge emerging in both cancer biology and deep learning. In this review, we provide an overview of emerging deep learning techniques and how they are being applied to oncology. We focus on the deep learning applications for omics data types, including genomic, methylation and transcriptomic data, as well as histopathology-based genomic inference, and provide perspectives on how the different data types can be integrated to develop decision support tools. We provide specific examples of how deep learning may be applied in cancer diagnosis, prognosis and treatment management. We also assess the current limitations and challenges for the application of deep learning in precision oncology, including the lack of phenotypically rich data and the need for more explainable deep learning models. Finally, we conclude with a discussion of how current obstacles can be overcome to enable future clinical utilisation of deep learning.