Prognostic analysis of histopathological images using pre-trained convolutional neural networks: application to hepatocellular carcinoma

Deep learning and digital pathology powers prediction of HCC development in steatotic liver disease

BACKGROUND AND AIMS

Identifying patients with steatotic liver disease who are at a high risk of developing HCC remains challenging. We present a deep learning (DL) model to predict HCC development using hematoxylin and eosin-stained whole-slide images of biopsy-proven steatotic liver disease.

APPROACH AND RESULTS

We included 639 patients who did not develop HCC for ≥7 years after biopsy (non-HCC class) and 46 patients who developed HCC <7 years after biopsy (HCC class). Paired cases of the HCC and non-HCC classes matched by biopsy date and institution were used for training, and the remaining nonpaired cases were used for validation. The DL model was trained using deep convolutional neural networks with 28,000 image tiles cropped from whole-slide images of the paired cases, with an accuracy of 81.0% and an AUC of 0.80 for predicting HCC development. Validation using the nonpaired cases also demonstrated a good accuracy of 82.3% and an AUC of 0.84. These results were comparable to the predictive ability of logistic regression model using fibrosis stage. Notably, the DL model also detected the cases of HCC development in patients with mild fibrosis. The saliency maps generated by the DL model highlighted various pathological features associated with HCC development, including nuclear atypia, hepatocytes with a high nuclear-cytoplasmic ratio, immune cell infiltration, fibrosis, and a lack of large fat droplets.

CONCLUSIONS

The ability of the DL model to capture subtle pathological features beyond fibrosis suggests its potential for identifying early signs of hepatocarcinogenesis in patients with steatotic liver disease.

NecroGlobalGCN: Integrating micronecrosis information in HCC prognosis prediction via graph convolutional neural networks

BACKGROUND AND OBJECTIVE

Hepatocellular carcinoma (HCC) ranks fourth in cancer mortality, underscoring the importance of accurate prognostic predictions to improve postoperative survival rates in patients. Although micronecrosis has been shown to have high prognostic value in HCC, its application in clinical prognosis prediction requires specialized knowledge and complex calculations, which poses challenges for clinicians. It would be of interest to develop a model to help clinicians make full use of micronecrosis to assess patient survival.

METHODS

To address these challenges, we propose a HCC prognosis prediction model that integrates pathological micronecrosis information through Graph Convolutional Neural Networks (GCN). This approach enables GCN to utilize micronecrosis, which has been shown to be highly correlated with prognosis, thereby significantly enhancing prognostic stratification quality. We developed our model using 3622 slides from 752 patients with primary HCC from the FAH-ZJUMS dataset and conducted internal and external validations on the FAH-ZJUMS and TCGA-LIHC datasets, respectively.

RESULTS

Our method outperformed the baseline by 8.18% in internal validation and 9.02% in external validations. Overall, this paper presents a deep learning research paradigm that integrates HCC micronecrosis, enhancing both the accuracy and interpretability of prognostic predictions, with potential applicability to other pathological prognostic markers.

CONCLUSIONS

This study proposes a composite GCN prognostic model that integrates information on HCC micronecrosis, collecting large dataset of HCC histopathological images. This approach could assist clinicians in analyzing HCC patient survival and precisely locating and visualizing necrotic tissues that affect prognosis. Following the research paradigm outlined in this paper, other prognostic biomarker integration models with GCN could be developed, significantly enhancing the predictive performance and interpretability of prognostic model.

The power of deep learning in simplifying feature selection for hepatocellular carcinoma: a review

BACKGROUND

Hepatocellular Carcinoma (HCC) is a highly aggressive, prevalent, and deadly type of liver cancer. With the advent of deep learning techniques, significant advancements have been made in simplifying and optimizing the feature selection process.

OBJECTIVE

Our scoping review presents an overview of the various deep learning models and algorithms utilized to address feature selection for HCC. The paper highlights the strengths and limitations of each approach, along with their potential applications in clinical practice. Additionally, it discusses the benefits of using deep learning to identify relevant features and their impact on the accuracy and efficiency of diagnosis, prognosis, and treatment of HCC.

DESIGN

The review encompasses a comprehensive analysis of the research conducted in the past few years, focusing on the methodologies, datasets, and evaluation metrics adopted by different studies. The paper aims to identify the key trends and advancements in the field, shedding light on the promising areas for future research and development.

RESULTS

The findings of this review indicate that deep learning techniques have shown promising results in simplifying feature selection for HCC. By leveraging large-scale datasets and advanced neural network architectures, these methods have demonstrated improved accuracy and robustness in identifying predictive features.

CONCLUSIONS

We analyze published studies to reveal the state-of-the-art HCC prediction and showcase how deep learning can boost accuracy and decrease false positives. But we also acknowledge the challenges that remain in translating this potential into clinical reality.

The role of artificial intelligence and image processing in the diagnosis, treatment, and prognosis of liver cancer: a narrative-review

Artificial intelligence (AI) and image processing are revolutionising the diagnosis and management of liver cancer. Recent advancements showcase AI's ability to analyse medical imaging data, like computed tomography scans and magnetic resonance imaging, accurately detecting and classifying liver cancer lesions for early intervention. Predictive models aid prognosis estimation and recurrence pattern identification, facilitating personalised treatment planning. Image processing techniques enhance data analysis by precise segmentation of liver structures, fusion of information from multiple modalities, and feature extraction for informed decision-making. Despite progress, challenges persist, including the need for standardised datasets and regulatory considerations.

Deep Learning Classification and Quantification of Pejorative and Nonpejorative Architectures in Resected Hepatocellular Carcinoma from Digital Histopathologic Images

Liver resection is one of the best treatments for small hepatocellular carcinoma (HCC), but post-resection recurrence is frequent. Biotherapies have emerged as an efficient adjuvant treatment, making the identification of patients at high risk of recurrence critical. Microvascular invasion (mVI), poor differentiation, pejorative macrotrabecular architectures, and vessels encapsulating tumor clusters architectures are the most accurate histologic predictors of recurrence, but their evaluation is time-consuming and imperfect. Herein, a supervised deep learning-based approach with ResNet34 on 680 whole slide images (WSIs) from 107 liver resection specimens was used to build an algorithm for the identification and quantification of these pejorative architectures. This model achieved an accuracy of 0.864 at patch level and 0.823 at WSI level. To assess its robustness, it was validated on an external cohort of 29 HCCs from another hospital, with an accuracy of 0.787 at WSI level, affirming its generalization capabilities. Moreover, the largest connected areas of the pejorative architectures extracted from the model were positively correlated to the presence of mVI and the number of tumor emboli. These results suggest that the identification of pejorative architectures could be an efficient surrogate of mVI and have a strong predictive value for the risk of recurrence. This study is the first step in the construction of a composite predictive algorithm for early post-resection recurrence of HCC, including artificial intelligence-based features.

State-of-the-Art of Breast Cancer Diagnosis in Medical Images via Convolutional Neural Networks (CNNs)

Early detection of breast cancer is crucial for a better prognosis. Various studies have been conducted where tumor lesions are detected and localized on images. This is a narrative review where the studies reviewed are related to five different image modalities: histopathological, mammogram, magnetic resonance imaging (MRI), ultrasound, and computed tomography (CT) images, making it different from other review studies where fewer image modalities are reviewed. The goal is to have the necessary information, such as pre-processing techniques and CNN-based diagnosis techniques for the five modalities, readily available in one place for future studies. Each modality has pros and cons, such as mammograms might give a high false positive rate for radiographically dense breasts, while ultrasounds with low soft tissue contrast result in early-stage false detection, and MRI provides a three-dimensional volumetric image, but it is expensive and cannot be used as a routine test. Various studies were manually reviewed using particular inclusion and exclusion criteria; as a result, 91 recent studies that classify and detect tumor lesions on breast cancer images from 2017 to 2022 related to the five image modalities were included. For histopathological images, the maximum accuracy achieved was around 99 % , and the maximum sensitivity achieved was 97.29 % by using DenseNet, ResNet34, and ResNet50 architecture. For mammogram images, the maximum accuracy achieved was 96.52 % using a customized CNN architecture. For MRI, the maximum accuracy achieved was 98.33 % using customized CNN architecture. For ultrasound, the maximum accuracy achieved was around 99 % by using DarkNet-53, ResNet-50, G-CNN, and VGG. For CT, the maximum sensitivity achieved was 96 % by using Xception architecture. Histopathological and ultrasound images achieved higher accuracy of around 99 % by using ResNet34, ResNet50, DarkNet-53, G-CNN, and VGG compared to other modalities for either of the following reasons: use of pre-trained architectures with pre-processing techniques, use of modified architectures with pre-processing techniques, use of two-stage CNN, and higher number of studies available for Artificial Intelligence (AI)/machine learning (ML) researchers to reference. One of the gaps we found is that only a single image modality is used for CNN-based diagnosis; in the future, a multiple image modality approach can be used to design a CNN architecture with higher accuracy.

Explainable domain transfer of distant supervised cancer subtyping model via imaging-based rules extraction

Image texture analysis has for decades represented a promising opportunity for cancer assessment and disease progression evaluation, evolving in a discipline, i.e., radiomics. However, the road to a complete translation into clinical practice is still hampered by intrinsic limitations. As purely supervised classification models fail in devising robust imaging-based biomarkers for prognosis, cancer subtyping approaches would benefit from the employment of distant supervision, for instance exploiting survival/recurrence information. In this work, we assessed, tested, and validated the domain-generality of our previously proposed Distant Supervised Cancer Subtyping model on Hodgkin Lymphoma. We evaluate the model performance on two independent datasets coming from two hospitals, comparing and analyzing the results. Although successful and consistent, the comparison confirmed the instability of radiomics due to an across-center lack of reproducibility, leading to explainable results in one center and poor interpretability in the other. We thus propose a Random Forest-based Explainable Transfer Model for testing the domain-invariance of imaging biomarkers extracted from retrospective cancer subtyping. In doing so, we tested the predictive ability of cancer subtyping in a validation and perspective setting, which led to successful results and supported the domain-generality of the proposed approach. On the other hand, the extraction of decision rules enables to draw of risk factors and robust biomarkers to inform clinical decisions. This work shows the potentialities of the Distant Supervised Cancer Subtyping model to be further evaluated in larger multi-center datasets, to reliably translate radiomics into medical practice. The code is available at this GitHub repository.

Artificial intelligence for the prevention and clinical management of hepatocellular carcinoma

Hepatocellular carcinoma (HCC) currently represents the fifth most common malignancy and the third-leading cause of cancer-related death worldwide, with incidence and mortality rates that are increasing. Recently, artificial intelligence (AI) has emerged as a unique opportunity to improve the full spectrum of HCC clinical care, by improving HCC risk prediction, diagnosis, and prognostication. AI approaches include computational search algorithms, machine learning (ML) and deep learning (DL) models. ML consists of a computer running repeated iterations of models, in order to progressively improve performance of a specific task, such as classifying an outcome. DL models are a subtype of ML, based on neural network structures that are inspired by the neuroanatomy of the human brain. A growing body of recent data now apply DL models to diverse data sources - including electronic health record data, imaging modalities, histopathology and molecular biomarkers - to improve the accuracy of HCC risk prediction, detection and prediction of treatment response. Despite the promise of these early results, future research is still needed to standardise AI data, and to improve both the generalisability and interpretability of results. If such challenges can be overcome, AI has the potential to profoundly change the way in which care is provided to patients with or at risk of HCC.

Deep learning in hepatocellular carcinoma: Current status and future perspectives

Hepatocellular carcinoma (HCC) is among the leading causes of cancer incidence and death. Despite decades of research and development of new treatment options, the overall outcomes of patients with HCC continue to remain poor. There are areas of unmet need in risk prediction, early diagnosis, accurate prognostication, and individualized treatments for patients with HCC. Recent years have seen an explosive growth in the application of artificial intelligence (AI) technology in medical research, with the field of HCC being no exception. Among the various AI-based machine learning algorithms, deep learning algorithms are considered state-of-the-art techniques for handling and processing complex multimodal data ranging from routine clinical variables to high-resolution medical images. This article will provide a comprehensive review of the recently published studies that have applied deep learning for risk prediction, diagnosis, prognostication, and treatment planning for patients with HCC.

Artificial Intelligence for solid tumor diagnosis in digital pathology

Tumor diagnosis relies on the visual examination of histological slides by pathologists through a microscope eyepiece. Digital pathology, the digitalization of histological slides at high magnification with slides scanners, has raised the opportunity to extract quantitative information thanks to image analysis. In the last decade, medical image analysis has made exceptional progress due to the development of artificial intelligence (AI) algorithms. AI has been successfully used in the field of medical imaging and more recently in digital pathology. The feasibility and usefulness of AI assisted pathology tasks have been demonstrated in the very last years and we can expect those developments to be applied on routine histopathology in the future. In this review, we will describe and illustrate this technique and present the most recent applications in the field of tumor histopathology.

A generalized deep learning framework for whole-slide image segmentation and analysis

Histopathology tissue analysis is considered the gold standard in cancer diagnosis and prognosis. Whole-slide imaging (WSI), i.e., the scanning and digitization of entire histology slides, are now being adopted across the world in pathology labs. Trained histopathologists can provide an accurate diagnosis of biopsy specimens based on WSI data. Given the dimensionality of WSIs and the increase in the number of potential cancer cases, analyzing these images is a time-consuming process. Automated segmentation of tumorous tissue helps in elevating the precision, speed, and reproducibility of research. In the recent past, deep learning-based techniques have provided state-of-the-art results in a wide variety of image analysis tasks, including the analysis of digitized slides. However, deep learning-based solutions pose many technical challenges, including the large size of WSI data, heterogeneity in images, and complexity of features. In this study, we propose a generalized deep learning-based framework for histopathology tissue analysis to address these challenges. Our framework is, in essence, a sequence of individual techniques in the preprocessing-training-inference pipeline which, in conjunction, improve the efficiency and the generalizability of the analysis. The combination of techniques we have introduced includes an ensemble segmentation model, division of the WSI into smaller overlapping patches while addressing class imbalances, efficient techniques for inference, and an efficient, patch-based uncertainty estimation framework. Our ensemble consists of DenseNet-121, Inception-ResNet-V2, and DeeplabV3Plus, where all the networks were trained end to end for every task. We demonstrate the efficacy and improved generalizability of our framework by evaluating it on a variety of histopathology tasks including breast cancer metastases (CAMELYON), colon cancer (DigestPath), and liver cancer (PAIP). Our proposed framework has state-of-the-art performance across all these tasks and is ranked within the top 5 currently for the challenges based on these datasets. The entire framework along with the trained models and the related documentation are made freely available at GitHub and PyPi. Our framework is expected to aid histopathologists in accurate and efficient initial diagnosis. Moreover, the estimated uncertainty maps will help clinicians to make informed decisions and further treatment planning or analysis.

imPlatelet classifier: image-converted RNA biomarker profiles enable blood-based cancer diagnostics

Liquid biopsies offer a minimally invasive sample collection, outperforming traditional biopsies employed for cancer evaluation. The widely used material is blood, which is the source of tumor‐educated platelets. Here, we developed the imPlatelet classifier, which converts RNA‐sequenced platelet data into images in which each pixel corresponds to the expression level of a certain gene. Biological knowledge from the Kyoto Encyclopedia of Genes and Genomes was also implemented to improve accuracy. Images obtained from samples can then be compared against standard images for specific cancers to determine a diagnosis. We tested imPlatelet on a cohort of 401 non‐small cell lung cancer patients, 62 sarcoma patients, and 28 ovarian cancer patients. imPlatelet provided excellent discrimination between lung cancer cases and healthy controls, with accuracy equal to 1 in the independent dataset. When discriminating between noncancer cases and sarcoma or ovarian cancer patients, accuracy equaled 0.91 or 0.95, respectively, in the independent datasets. According to our knowledge, this is the first study implementing an image‐based deep‐learning approach combined with biological knowledge to classify human samples. The performance of imPlatelet considerably exceeds previously published methods and our own alternative attempts of sample discrimination. We show that the deep‐learning image‐based classifier accurately identifies cancer, even when a limited number of samples are available.

AI based colorectal disease detection using real-time screening colonoscopy

Colonoscopy is an effective tool for early screening of colorectal diseases. However, the application of colonoscopy in distinguishing different intestinal diseases still faces great challenges of efficiency and accuracy. Here we constructed and evaluated a deep convolution neural network (CNN) model based on 117 055 images from 16 004 individuals, which achieved a high accuracy of 0.933 in the validation dataset in identifying patients with polyp, colitis, colorectal cancer (CRC) from normal. The proposed approach was further validated on multi-center real-time colonoscopy videos and images, which achieved accurate diagnostic performance on detecting colorectal diseases with high accuracy and precision to generalize across external validation datasets. The diagnostic performance of the model was further compared to the skilled endoscopists and the novices. In addition, our model has potential in diagnosis of adenomatous polyp and hyperplastic polyp with an area under the receiver operating characteristic curve of 0.975. Our proposed CNN models have potential in assisting clinicians in making clinical decisions with efficiency during application.

Update on Hepatocellular Carcinoma: a Brief Review from Pathologist Standpoint

Hepatocellular carcinoma is one of the most common cancers and an important health problem all over the world. Its prognosis is poor. For better patient care, early diagnosis is essential. Although new imaging techniques have a big impact on hepatocellular carcinoma diagnosis, histopathological examination is still the gold standard for precise diagnosis. Histopathological evaluation gives exact diagnosis in the meaning of tumor size, histological subtypes, grading, and differential diagnosis from metastasis and other tumors. Immunohistochemistry as a part of diagnostic histopathological technique plays an important role in routine practice. Immunohistochemistry is useful for confirming of hepatocytic origin, supporting hepatocellular malignancy, and differential diagnosis. It also gives prognostic information. There are growing attempts to classify tumors by their molecular genetic signatures. This is also actual for hepatocellular carcinoma. This mini review focuses on the histopathology of hepatocellular carcinoma including subtypes; differential diagnosis and immunohistochemistry as an ancillary diagnostic tool, updated or added entities, i.e., combined hepatocellular-cholangiocarcinoma; small hepatocellular carcinoma; correlation with molecular studies; and future perspectives.

Toward automated severe pharyngitis detection with smartphone camera using deep learning networks

PURPOSE

Severe pharyngitis is frequently associated with inflammations caused by streptococcal pharyngitis, which can cause immune-mediated and post-infectious complications. The recent global pandemic of coronavirus disease (COVID-19) encourages the use of telemedicine for patients with respiratory symptoms. This study, therefore, purposes automated detection of severe pharyngitis using a deep learning framework with self-taken throat images.

METHODS

A dataset composed of two classes of 131 throat images with pharyngitis and 208 normal throat images was collected. Before the training classifier, we constructed a cycle consistency generative adversarial network (CycleGAN) to augment the training dataset. The ResNet50, Inception-v3, and MobileNet-v2 architectures were trained with transfer learning and validated using a randomly selected test dataset. The performance of the models was evaluated based on the accuracy and area under the receiver operating characteristic curve (ROC-AUC).

RESULTS

The CycleGAN-based synthetic images reflected the pragmatic characteristic features of pharyngitis. Using the synthetic throat images, the deep learning model demonstrated a significant improvement in the accuracy of the pharyngitis diagnosis. ResNet50 with GAN-based augmentation showed the best ROC-AUC of 0.988 for pharyngitis detection in the test dataset. In the 4-fold cross-validation using the ResNet50, the highest detection accuracy and ROC-AUC achieved were 95.3% and 0.992, respectively.

CONCLUSION

The deep learning model for smartphone-based pharyngitis screening allows fast identification of severe pharyngitis with a potential of the timely diagnosis of pharyngitis. In the recent pandemic of COVID-19, this framework will help patients with upper respiratory symptoms to improve convenience in diagnosis and reduce transmission.

A CNN-based active learning framework to identify mycobacteria in digitized Ziehl-Neelsen stained human tissues

Tuberculosis is the most common mycobacterial disease that affects humans worldwide. Rapid and reliable diagnosis of mycobacteria is crucial to identify infected individuals, to initiate and monitor treatment and to minimize or prevent transmission. Microscopic identification of acid-fast mycobacteria (AFB) in tissue sections is usually accomplished by examining Ziehl-Neelsen (ZN) stained slides in which AFB appear bright red against the blue background. Because the ZN-stained slides require time consuming and meticulous screening by an experienced pathologist, our team developed a machine learning pipeline to classify digitized ZN-stained slides as AFB-positive or AFB-negative. The pipeline includes two convolutional neural network (CNN) models to recognize tiles containing AFB, and a logistic regression (LR) model to classify slides based on features from AFB-probability maps assembled from the CNN tile-based classification results. The first CNN was trained using tiles from 6 AFB-positive and 8 AFB-negative slides, and the second CNN was trained using the initial tile set expanded by additional tiles from 19 AFB-negative slides selected within an active learning framework. When evaluated on a separate set of tiles, the two CNNs yielded F1 scores of 99.03% and 98.75%, respectively, and were used to classify tiles in a separate set of 134 slides (46 AFB-positive and 88 AFB-negative). The classification yielded two AFB-probability maps, one for each CNN. The LR model was then 10-fold cross-validated using the average of feature vectors extracted from the AFB-probability maps generated by each CNN. The feature vector consisted of seven features of the AFB-probability map histogram and the positive tile rate (PTR). The sensitivity (87.13%), specificity (87.62%) and F1 (80.18%) achieved by this model were superior to the baseline performance of PTR-based separation of slides that yielded F1 scores of 73.13% and 66.67% in the AFB-probability maps outputted by the CNN trained within the active learning framework and the CNN trained only on the initial set of slides, respectively. Our CNNs outperformed several recently published models for AFB detection. Active learning induced robust learning of features by the CNN and led to improved LR classification performance of slides. In the 52 AFB-positive slides used in the pipeline development, the AFB were infrequent, predominantly single and only rarely found in small clusters. Our pipeline can classify slides and visualize suspected AFB-positive areas in each slide, and thus potentially facilitate evaluation of ZN-stained tissue sections for AFB.