1
|
Yang X, Zhang R, Yang Y, Zhang Y, Chen K. PathEX: Make good choice for whole slide image extraction. PLoS One 2024; 19:e0304702. [PMID: 39208135 PMCID: PMC11361590 DOI: 10.1371/journal.pone.0304702] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Accepted: 05/17/2024] [Indexed: 09/04/2024] Open
Abstract
BACKGROUND The tile-based approach has been widely used for slide-level predictions in whole slide image (WSI) analysis. However, the irregular shapes and variable dimensions of tumor regions pose challenges for the process. To address this issue, we proposed PathEX, a framework that integrates intersection over tile (IoT) and background over tile (BoT) algorithms to extract tile images around boundaries of annotated regions while excluding the blank tile images within these regions. METHODS We developed PathEX, which incorporated IoT and BoT into tile extraction, for training a classification model in CAM (239 WSIs) and PAIP (40 WSIs) datasets. By adjusting the IoT and BoT parameters, we generated eight training sets and corresponding models for each dataset. The performance of PathEX was assessed on the testing set comprising 13,076 tile images from 48 WSIs of CAM dataset and 6,391 tile images from 10 WSIs of PAIP dataset. RESULTS PathEX could extract tile images around boundaries of annotated region differently by adjusting the IoT parameter, while exclusion of blank tile images within annotated regions achieved by setting the BoT parameter. As adjusting IoT from 0.1 to 1.0, and 1-BoT from 0.0 to 0.5, we got 8 train sets. Experimentation revealed that set C demonstrates potential as the most optimal candidate. Nevertheless, a combination of IoT values ranging from 0.2 to 0.5 and 1-BoT values ranging from 0.2 to 0.5 also yielded favorable outcomes. CONCLUSIONS In this study, we proposed PathEX, a framework that integrates IoT and BoT algorithms for tile image extraction at the boundaries of annotated regions while excluding blank tiles within these regions. Researchers can conveniently set the thresholds for IoT and BoT to facilitate tile image extraction in their own studies. The insights gained from this research provide valuable guidance for tile image extraction in digital pathology applications.
Collapse
Affiliation(s)
- Xinda Yang
- Renmin University of China School of Information, Beijing, P.R. China
| | - Ranze Zhang
- Breast Tumor Center, Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong, China
- Breast Tumor Center, Sun Yat-sen Breast Tumor Hospital, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong, China
| | - Yuan Yang
- Department of Research and Development, Health Data (Beijing) Technology Co., Ltd, Guangzhou, Guangdong, P.R. China
| | - Yu Zhang
- Renmin University of China School of Information, Beijing, P.R. China
| | - Kai Chen
- Breast Tumor Center, Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong, China
- Breast Tumor Center, Sun Yat-sen Breast Tumor Hospital, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong, China
- Artificial Intelligence Lab, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, Guangdong, China
| |
Collapse
|
2
|
Yang X, Li X, Xu H, Du S, Wang C, He H. Predicting CTLA4 expression and prognosis in clear cell renal cell carcinoma using a pathomics signature of histopathological images and machine learning. Heliyon 2024; 10:e34877. [PMID: 39145002 PMCID: PMC11320204 DOI: 10.1016/j.heliyon.2024.e34877] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 07/15/2024] [Accepted: 07/17/2024] [Indexed: 08/16/2024] Open
Abstract
Background CTLA4, an immune checkpoint, plays an important role in tumor immunotherapy. The purpose of this study was to develop a pathomics signature to evaluate CTLA4 expression and predict clinical outcomes in clear cell renal cell carcinoma (ccRCC) patients. Methods A total of 354 patients from the TCGA-KIRC dataset were enrolled in this study. The patients were stratified into two groups based on the level of CTLA4 expression, and overall survival rates were analyzed between groups. Pathological features were identified using machine learning algorithms, and a gradient boosting machine (GBM) was employed to construct the pathomics signatures for predicting prognosis and CTLA4 expression. The predictive performance of the model was subsequently assessed. Enrichment analysis was performed on diferentially expressed genes related to the pathomics score (PS). Additionally, correlations between PS and TMB, as well as immune infiltration profiles associated with different PS values, were explored. In vitro experiments, CTLA4 knockdown was performed to investigate its impact on cell proliferation, migration, invasion, TGF-β signaling pathway, and macrophage polarization. Results High expression of CTLA4 was associated with an unfavorable prognosis in ccRCC patients. The pathomics signature displayed good performance in the validation set (AUC = 0.737; P < 0.001 in the log-rank test). The PS was positively correlated with CTLA4 expression. We next explored the underlying mechanism and found the associations between the pathomics signature and TGF-β signaling pathways, TMB, and Tregs. Further in vitro experiments demonstrated that CTLA4 knockdown inhibited cell proliferation, migration, invasion, TGF-β expression, and macrophage M2 polarization. Conclusion High expression of CTLA4 was found to correlate with poor prognosis in ccRCC patients. The pathomics signature established by our group using machine learning effectively predicted both patient prognosis and CTLA4 expression levels in ccRCC cases.
Collapse
Affiliation(s)
- Xiaoqun Yang
- Department of Pathology, Shanghai Ruijin Hospital, Shanghai Jiaotong University School of Medicine, Shanghai, China
| | - Xiangyun Li
- Department of Pathology, Shanghai Ruijin Hospital, Shanghai Jiaotong University School of Medicine, Shanghai, China
| | - Haimin Xu
- Department of Pathology, Shanghai Ruijin Hospital, Shanghai Jiaotong University School of Medicine, Shanghai, China
| | - Silin Du
- University Hospital, Shanghai Jiaotong University, Shanghai, China
| | - Chaofu Wang
- Department of Pathology, Shanghai Ruijin Hospital, Shanghai Jiaotong University School of Medicine, Shanghai, China
| | - Hongchao He
- Department of Urology, Shanghai Ruijin Hospital, Shanghai Jiaotong University School of Medicine, Shanghai, China
- Department of Genitourinary Medical Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, 77030, USA
| |
Collapse
|
3
|
Zhang X, Liu C, Zhu H, Wang T, Du Z, Ding W. A universal multiple instance learning framework for whole slide image analysis. Comput Biol Med 2024; 178:108714. [PMID: 38889627 DOI: 10.1016/j.compbiomed.2024.108714] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Revised: 06/04/2024] [Accepted: 06/04/2024] [Indexed: 06/20/2024]
Abstract
BACKGROUND The emergence of digital whole slide image (WSI) has driven the development of computational pathology. However, obtaining patch-level annotations is challenging and time-consuming due to the high resolution of WSI, which limits the applicability of fully supervised methods. We aim to address the challenges related to patch-level annotations. METHODS We propose a universal framework for weakly supervised WSI analysis based on Multiple Instance Learning (MIL). To achieve effective aggregation of instance features, we design a feature aggregation module from multiple dimensions by considering feature distribution, instances correlation and instance-level evaluation. First, we implement instance-level standardization layer and deep projection unit to improve the separation of instances in the feature space. Then, a self-attention mechanism is employed to explore dependencies between instances. Additionally, an instance-level pseudo-label evaluation method is introduced to enhance the available information during the weak supervision process. Finally, a bag-level classifier is used to obtain preliminary WSI classification results. To achieve even more accurate WSI label predictions, we have designed a key instance selection module that strengthens the learning of local features for instances. Combining the results from both modules leads to an improvement in WSI prediction accuracy. RESULTS Experiments conducted on Camelyon16, TCGA-NSCLC, SICAPv2, PANDA and classical MIL benchmark datasets demonstrate that our proposed method achieves a competitive performance compared to some recent methods, with maximum improvement of 14.6 % in terms of classification accuracy. CONCLUSION Our method can improve the classification accuracy of whole slide images in a weakly supervised way, and more accurately detect lesion areas.
Collapse
Affiliation(s)
- Xueqin Zhang
- College of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China; Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai 201112, China
| | - Chang Liu
- College of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China.
| | - Huitong Zhu
- College of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China
| | - Tianqi Wang
- College of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China
| | - Zunguo Du
- Department of Pathology, Huashan Hospital Affiliated to Fudan University, Shanghai, 200040, China
| | - Weihong Ding
- Department of Urology, Huashan Hospital Affiliated to Fudan University, Shanghai, 200040, China.
| |
Collapse
|
4
|
Bai C, Sun Y, Zhang X, Zuo Z. Assessment of AURKA expression and prognosis prediction in lung adenocarcinoma using machine learning-based pathomics signature. Heliyon 2024; 10:e33107. [PMID: 39022022 PMCID: PMC11253280 DOI: 10.1016/j.heliyon.2024.e33107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2024] [Revised: 06/07/2024] [Accepted: 06/14/2024] [Indexed: 07/20/2024] Open
Abstract
Objective This study aimed to develop quantitative feature-based models from histopathological images to assess aurora kinase A (AURKA) expression and predict the prognosis of patients with lung adenocarcinoma (LUAD). Methods A dataset of patients with LUAD was derived from the cancer genome atlas (TCGA) with information on clinical characteristics, RNA sequencing and histopathological images. The TCGA-LUAD cohort was randomly divided into training (n = 229) and testing (n = 98) sets. We extracted quantitative image features from histopathological slides of patients with LUAD using computational approaches, constructed a predictive model for AURKA expression in the training set, and estimated their predictive performance in the test set. A Cox proportional hazards model was used to assess whether the pathomic scores (PS) generated by the model independently predicted LUAD survival. Results High AURKA expression was an independent risk factor for overall survival (OS) in patients with LUAD (hazard ratio = 1.816, 95 % confidence intervals = 1.257-2.623, P = 0.001). The model based on histopathological image features had significant predictive value for AURKA expression: the area under the curve of the receiver operating characteristic curve in the training set and validation set was 0.809 and 0.739, respectively. Decision curve analysis showed that the model had clinical utility. Patients with high PS and low PS had different survival rates (P = 0.019). Multivariate analysis suggested that PS was an independent prognostic factor for LUAD (hazard ratio = 1.615, 95 % confidence intervals = 1.071-2.438, P = 0.022). Conclusion Pathomics models based on machine learning can accurately predict AURKA expression and the PS generated by the model can predict LUAD prognosis.
Collapse
Affiliation(s)
- Cuiqing Bai
- Department of Respiratory Disease, Affiliated Hospital of Jiangnan University, Wuxi, China
| | - Yan Sun
- Department of Respiratory Disease, Affiliated Hospital of Jiangnan University, Wuxi, China
| | - Xiuqin Zhang
- Department of Respiratory Disease, Affiliated Hospital of Jiangnan University, Wuxi, China
| | - Zhitong Zuo
- Department of Respiratory Disease, Affiliated Hospital of Jiangnan University, Wuxi, China
| |
Collapse
|
5
|
Wang X, Yuan W. Nuclei-level prior knowledge constrained multiple instance learning for breast histopathology whole slide image classification. iScience 2024; 27:109826. [PMID: 38832012 PMCID: PMC11145340 DOI: 10.1016/j.isci.2024.109826] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Revised: 04/17/2024] [Accepted: 04/24/2024] [Indexed: 06/05/2024] Open
Abstract
New breast cancer cases have surpassed lung cancer, becoming the world's most prevalent cancer. Despite advancing medical image analysis, deep learning's lack of interpretability limits its adoption among pathologists. Hence, a nuclei-level prior knowledge constrained multiple instance learning (MIL) (NPKC-MIL) for breast whole slide image (WSI) classification is proposed. NPKC-MIL primarily involves the following steps: Initially, employing the transfer learning to extract patch-level features and aggregate them into slide-level features through attention pooling. Subsequently, abstract the extracted nuclei as nodes, establish nucleus topology using the K-NN (K-Nearest Neighbors, K-NN) algorithm, and create handcrafted features for nodes. Finally, combine patch-level deep learning features with nuclei-level handcrafted features to fine-tune classification results generated by slide-level deep learning features. The experimental results demonstrate that NPKC-MIL outperforms current comparable deep learning models. NPKC-MIL expands the analytical dimension of WSI classification tasks and integrates prior knowledge into deep learning models to improve interpretability.
Collapse
Affiliation(s)
- Xunping Wang
- School of Remote Sensing and Information Engineering, Wuhan University, 129 Luoyu Road, Wuhan 430079, China
| | - Wei Yuan
- Co-Creation Center for Disaster Resilience, International Research Institute of Disaster Science, Tohoku University, Aoba 468-1, Aramaki, Aoba-ku, Sendai 980-8572, Japan
| |
Collapse
|
6
|
Yan Z, Li X, Li Z, Liu S, Chang H. Prognostic significance of TNFRSF4 expression and development of a pathomics model to predict expression in hepatocellular carcinoma. Heliyon 2024; 10:e31882. [PMID: 38841483 PMCID: PMC11152671 DOI: 10.1016/j.heliyon.2024.e31882] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 05/16/2024] [Accepted: 05/23/2024] [Indexed: 06/07/2024] Open
Abstract
Background TNFRSF4 plays a significant role in cancer progression, especially in hepatocellular carcinoma (HCC). This study aims to investigate the prognostic value of TNFRSF4 expression in patients with HCC and to develop a predictive pathomics model for its expression. Methods A cohort of patients with HCC retrieved from the TCGA database was analyzed using RNA-seq analysis to determine TNFRSF4 expression and its impact on overall survival (OS). Additionally, hematoxylin-eosin staining analysis was performed to construct a pathomics model for predicting TNFRSF4 expression. Then, pathway enrichment analysis was conducted, immune checkpoint markers were investigated, and immune cell infiltration was examined to explore the underlying biological mechanism of the pathomics score. Results TNFRSF4 expression was significantly higher in tumor tissues than in normal tissues. TNFRSF4 expression also exhibited significant correlations with various clinical variables, including pathologic stage III/IV and R1/R2/RX residual tumor. Furthermore, elevated TNFRSF4 expression was associated with unfavorable OS. Interestingly, in the subgroup analysis, elevated TNFRSF4 expression was identified as a significant risk factor for OS in male patients. The newly developed pathomics model successfully predicted TNFRSF4 expression with good performance and revealed a significant association between high pathomics scores and worse OS. In male patients, high pathomics scores were also associated with a higher risk of mortality. Moreover, pathomics scores were also involved in specific hallmarks, immune-related characteristics, and apoptosis-related genes in HCC, such as epithelial-mesenchymal transition, Tregs, and BAX expression. Conclusions Our findings suggest that TNFRSF4 expression and the newly devised pathomics scores hold potential as prognostic markers for OS in patients with HCC. Additionally, gender influenced the association between these markers and patient outcomes.
Collapse
Affiliation(s)
- Zhaoyong Yan
- Department of Interventional Radiology, Shaanxi Provincial People's Hospital, Xi'an, 710068, China
| | - Xiang Li
- Department of Immunology, School of Basic Medicine, Tongji Medical College, Huazhong University of Science & Technology, Wuhan, 430000, China
| | - Zeyu Li
- Department of General Surgery, Shaanxi Provincial People's Hospital, Xi'an, 710068, China
| | - Sinan Liu
- Department of SICU, The First Affiliated Hospital of Xi'an Jiaotong University, Xi'an, 710061, China
| | - Hulin Chang
- Department of Hepatobiliary Surgery, Shaanxi Provincial People's Hospital, Xi'an, 710068, China
| |
Collapse
|
7
|
Chen X, Lin J, Wang Y, Zhang W, Xie W, Zheng Z, Wong KC. HE2Gene: image-to-RNA translation via multi-task learning for spatial transcriptomics data. Bioinformatics 2024; 40:btae343. [PMID: 38837395 PMCID: PMC11164830 DOI: 10.1093/bioinformatics/btae343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Revised: 05/06/2024] [Accepted: 05/25/2024] [Indexed: 06/07/2024] Open
Abstract
MOTIVATION Tissue context and molecular profiling are commonly used measures in understanding normal development and disease pathology. In recent years, the development of spatial molecular profiling technologies (e.g. spatial resolved transcriptomics) has enabled the exploration of quantitative links between tissue morphology and gene expression. However, these technologies remain expensive and time-consuming, with subsequent analyses necessitating high-throughput pathological annotations. On the other hand, existing computational tools are limited to predicting only a few dozen to several hundred genes, and the majority of the methods are designed for bulk RNA-seq. RESULTS In this context, we propose HE2Gene, the first multi-task learning-based method capable of predicting tens of thousands of spot-level gene expressions along with pathological annotations from H&E-stained images. Experimental results demonstrate that HE2Gene is comparable to state-of-the-art methods and generalizes well on an external dataset without the need for re-training. Moreover, HE2Gene preserves the annotated spatial domains and has the potential to identify biomarkers. This capability facilitates cancer diagnosis and broadens its applicability to investigate gene-disease associations. AVAILABILITY AND IMPLEMENTATION The source code and data information has been deposited at https://github.com/Microbiods/HE2Gene.
Collapse
Affiliation(s)
- Xingjian Chen
- Cutaneous Biology Research Center, Massachusetts General Hospital, Harvard Medical School, Boston, MA 02129, USA
- Department of Computer Science, City University of Hong Kong, Kowloog Tong 999077, Hong Kong SAR
| | - Jiecong Lin
- Molecular Pathology Unit, Center for Cancer Research, Massachusetts General Hospital, Department of Pathology, Harvard Medical School, Boston, MA 02129, USA
- Department of Computer Science, The University of Hong Kong, Pokfulam 999077, Hong Kong SAR
| | - Yuchen Wang
- Department of Computer Science, City University of Hong Kong, Kowloog Tong 999077, Hong Kong SAR
| | - Weitong Zhang
- Department of Computer Science, City University of Hong Kong, Kowloog Tong 999077, Hong Kong SAR
| | - Weidun Xie
- Department of Computer Science, City University of Hong Kong, Kowloog Tong 999077, Hong Kong SAR
| | - Zetian Zheng
- Department of Computer Science, City University of Hong Kong, Kowloog Tong 999077, Hong Kong SAR
| | - Ka-Chun Wong
- Department of Computer Science, City University of Hong Kong, Kowloog Tong 999077, Hong Kong SAR
- Shenzhen Research Institute, City University of Hong Kong, Shenzhen 518057, China
| |
Collapse
|
8
|
Pathan S, Ali T, P G S, P VK, Rao D. An optimized convolutional neural network architecture for lung cancer detection. APL Bioeng 2024; 8:026121. [PMID: 38868458 PMCID: PMC11168751 DOI: 10.1063/5.0208520] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Accepted: 05/31/2024] [Indexed: 06/14/2024] Open
Abstract
Lung cancer, the treacherous malignancy affecting the respiratory system of a human body, has a devastating impact on the health and well-being of an individual. Due to the lack of automated and noninvasive diagnostic tools, healthcare professionals look forward toward biopsy as a gold standard for diagnosis. However, biopsy could be traumatizing and expensive process. Additionally, the limited availability of dataset and inaccuracy in diagnosis is a major drawback experienced by researchers. The objective of the proposed research is to develop an automated diagnostic tool for screening of lung cancer using optimized hyperparameters such that convolutional neural network (CNN) model generalizes well for universally obtained computerized tomography (CT) slices of lung pathologies. The aforementioned objective is achieved in the following ways: (i) Initially, a preprocessing methodology specific to lung CT scans is formulated to avoid the loss of information due to random image smoothing, and (ii) a sine cosine algorithm optimization algorithm (SCA) is integrated in the CNN model, to optimally select the tuning parameters of CNN. The error rate is used as an objective function, and the SCA algorithm tries to minimize. The proposed method successfully achieved an average classification accuracy of 99% in classification of lung scans in normal, benign, and malignant classes. Further, the generalization ability of the proposed model is tested on unseen dataset, thereby achieving promising results. The quantitative results prove the efficacy of the system to be used by radiologists in a clinical scenario.
Collapse
Affiliation(s)
- Sameena Pathan
- Department of Information and Communication Technology, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, India
| | - Tanweer Ali
- Department of Electronics and Communication Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, India
| | - Sudheesh P G
- Department of Electronics and Communication Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, India
| | - Vasanth Kumar P
- Department of Electronics and Communication Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, India
| | - Divya Rao
- Department of Information and Communication Technology, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal 576104, India
| |
Collapse
|
9
|
McGenity C, Clarke EL, Jennings C, Matthews G, Cartlidge C, Freduah-Agyemang H, Stocken DD, Treanor D. Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy. NPJ Digit Med 2024; 7:114. [PMID: 38704465 PMCID: PMC11069583 DOI: 10.1038/s41746-024-01106-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2023] [Accepted: 04/12/2024] [Indexed: 05/06/2024] Open
Abstract
Ensuring diagnostic performance of artificial intelligence (AI) before introduction into clinical practice is essential. Growing numbers of studies using AI for digital pathology have been reported over recent years. The aim of this work is to examine the diagnostic accuracy of AI in digital pathology images for any disease. This systematic review and meta-analysis included diagnostic accuracy studies using any type of AI applied to whole slide images (WSIs) for any disease. The reference standard was diagnosis by histopathological assessment and/or immunohistochemistry. Searches were conducted in PubMed, EMBASE and CENTRAL in June 2022. Risk of bias and concerns of applicability were assessed using the QUADAS-2 tool. Data extraction was conducted by two investigators and meta-analysis was performed using a bivariate random effects model, with additional subgroup analyses also performed. Of 2976 identified studies, 100 were included in the review and 48 in the meta-analysis. Studies were from a range of countries, including over 152,000 whole slide images (WSIs), representing many diseases. These studies reported a mean sensitivity of 96.3% (CI 94.1-97.7) and mean specificity of 93.3% (CI 90.5-95.4). There was heterogeneity in study design and 99% of studies identified for inclusion had at least one area at high or unclear risk of bias or applicability concerns. Details on selection of cases, division of model development and validation data and raw performance data were frequently ambiguous or missing. AI is reported as having high diagnostic accuracy in the reported areas but requires more rigorous evaluation of its performance.
Collapse
Affiliation(s)
- Clare McGenity
- University of Leeds, Leeds, UK.
- Leeds Teaching Hospitals NHS Trust, Leeds, UK.
| | - Emily L Clarke
- University of Leeds, Leeds, UK
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
| | - Charlotte Jennings
- University of Leeds, Leeds, UK
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
| | | | | | | | | | - Darren Treanor
- University of Leeds, Leeds, UK
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
- Department of Clinical Pathology and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
- Centre for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| |
Collapse
|
10
|
Li D, Zhang J, Guo W, Ma K, Qin Z, Zhang J, Chen L, Xiong L, Huang J, Wan C, Huang P. A diagnostic strategy for pulmonary fat embolism based on routine H&E staining using computational pathology. Int J Legal Med 2024; 138:849-858. [PMID: 37999766 DOI: 10.1007/s00414-023-03136-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Accepted: 11/11/2023] [Indexed: 11/25/2023]
Abstract
Pulmonary fat embolism (PFE) as a cause of death often occurs in trauma cases such as fractures and soft tissue contusions. Traditional PFE diagnosis relies on subjective methods and special stains like oil red O. This study utilizes computational pathology, combining digital pathology and deep learning algorithms, to precisely quantify fat emboli in whole slide images using conventional hematoxylin-eosin (H&E) staining. The results demonstrate deep learning's ability to identify fat droplet morphology in lung microvessels, achieving an area under the receiver operating characteristic (ROC) curve (AUC) of 0.98. The AI-quantified fat globules generally matched the Falzi scoring system with oil red O staining. The relative quantity of fat emboli against lung area was calculated by the algorithm, determining a diagnostic threshold of 8.275% for fatal PFE. A diagnostic strategy based on this threshold achieved a high AUC of 0.984, similar to manual identification with special stains but surpassing H&E staining. This demonstrates computational pathology's potential as an affordable, rapid, and precise method for fatal PFE diagnosis in forensic practice.
Collapse
Affiliation(s)
- Dechan Li
- Department of Forensic Medicine, Guizhou Medical University, Guiyang, China
- Shanghai Key Laboratory of Forensic Medicine, Shanghai Forensic Service Platform, Academy of Forensic Science, Ministry of Justice, Shanghai, China
| | - Ji Zhang
- Shanghai Key Laboratory of Forensic Medicine, Shanghai Forensic Service Platform, Academy of Forensic Science, Ministry of Justice, Shanghai, China
| | - Wenqing Guo
- Shanghai Key Laboratory of Forensic Medicine, Shanghai Forensic Service Platform, Academy of Forensic Science, Ministry of Justice, Shanghai, China
- Department of Forensic Pathology, Shanxi Medical University, Taiyuan, China
| | - Kaijun Ma
- Shanghai Key Laboratory of Crime Scene Evidence, Institute of Criminal Science and Technology, Shanghai Municipal Public Security Bureau, Shanghai, China
| | - Zhiqiang Qin
- Shanghai Key Laboratory of Forensic Medicine, Shanghai Forensic Service Platform, Academy of Forensic Science, Ministry of Justice, Shanghai, China
| | - Jianhua Zhang
- Shanghai Key Laboratory of Forensic Medicine, Shanghai Forensic Service Platform, Academy of Forensic Science, Ministry of Justice, Shanghai, China
| | - Liqin Chen
- Department of Forensic Medicine, Inner Mongolia Medical University, Hohhot, China
| | - Ling Xiong
- Department of Forensic Medicine, Guizhou Medical University, Guiyang, China
| | - Jiang Huang
- Department of Forensic Medicine, Guizhou Medical University, Guiyang, China.
| | - Changwu Wan
- Department of Forensic Medicine, Guizhou Medical University, Guiyang, China.
| | - Ping Huang
- Department of Forensic Medicine, Guizhou Medical University, Guiyang, China.
- Shanghai Key Laboratory of Forensic Medicine, Shanghai Forensic Service Platform, Academy of Forensic Science, Ministry of Justice, Shanghai, China.
| |
Collapse
|
11
|
Hetz MJ, Bucher TC, Brinker TJ. Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images. Med Image Anal 2024; 94:103149. [PMID: 38574542 DOI: 10.1016/j.media.2024.103149] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2022] [Revised: 12/11/2023] [Accepted: 03/20/2024] [Indexed: 04/06/2024]
Abstract
The variation in histologic staining between different medical centers is one of the most profound challenges in the field of computer-aided diagnosis. The appearance disparity of pathological whole slide images causes algorithms to become less reliable, which in turn impedes the wide-spread applicability of downstream tasks like cancer diagnosis. Furthermore, different stainings lead to biases in the training which in case of domain shifts negatively affect the test performance. Therefore, in this paper we propose MultiStain-CycleGAN, a multi-domain approach to stain normalization based on CycleGAN. Our modifications to CycleGAN allow us to normalize images of different origins without retraining or using different models. We perform an extensive evaluation of our method using various metrics and compare it to commonly used methods that are multi-domain capable. First, we evaluate how well our method fools a domain classifier that tries to assign a medical center to an image. Then, we test our normalization on the tumor classification performance of a downstream classifier. Furthermore, we evaluate the image quality of the normalized images using the Structural similarity index and the ability to reduce the domain shift using the Fréchet inception distance. We show that our method proves to be multi-domain capable, provides a very high image quality among the compared methods, and can most reliably fool the domain classifier while keeping the tumor classifier performance high. By reducing the domain influence, biases in the data can be removed on the one hand and the origin of the whole slide image can be disguised on the other, thus enhancing patient data privacy.
Collapse
Affiliation(s)
- Martin J Hetz
- Division of Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tabea-Clara Bucher
- Division of Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Division of Digital Biomarkers for Oncology, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| |
Collapse
|
12
|
Kazemi A, Rasouli-Saravani A, Gharib M, Albuquerque T, Eslami S, Schüffler PJ. A systematic review of machine learning-based tumor-infiltrating lymphocytes analysis in colorectal cancer: Overview of techniques, performance metrics, and clinical outcomes. Comput Biol Med 2024; 173:108306. [PMID: 38554659 DOI: 10.1016/j.compbiomed.2024.108306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2023] [Revised: 03/04/2024] [Accepted: 03/12/2024] [Indexed: 04/02/2024]
Abstract
The incidence of colorectal cancer (CRC), one of the deadliest cancers around the world, is increasing. Tissue microenvironment (TME) features such as tumor-infiltrating lymphocytes (TILs) can have a crucial impact on diagnosis or decision-making for treating patients with CRC. While clinical studies showed that TILs improve the host immune response, leading to a better prognosis, inter-observer agreement for quantifying TILs is not perfect. Incorporating machine learning (ML) based applications in clinical routine may promote diagnosis reliability. Recently, ML has shown potential for making progress in routine clinical procedures. We aim to systematically review the TILs analysis based on ML in CRC histological images. Deep learning (DL) and non-DL techniques can aid pathologists in identifying TILs, and automated TILs are associated with patient outcomes. However, a large multi-institutional CRC dataset with a diverse and multi-ethnic population is necessary to generalize ML methods.
Collapse
Affiliation(s)
- Azar Kazemi
- Department of Medical Informatics, School of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Institute of General and Surgical Pathology, Technical University of Munich, Munich, Germany.
| | - Ashkan Rasouli-Saravani
- Student Research Committee, Department of Immunology, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran.
| | - Masoumeh Gharib
- Department of Pathology, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran.
| | | | - Saeid Eslami
- Department of Medical Informatics, School of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Pharmaceutical Sciences Research Center, Institute of Pharmaceutical Technology, Mashhad University of Medical Sciences, Mashhad, Iran; Department of Medical Informatics, University of Amsterdam, Amsterdam, the Netherlands.
| | - Peter J Schüffler
- Institute of General and Surgical Pathology, Technical University of Munich, Munich, Germany; TUM School of Computation, Information and Technology, Technical University of Munich, Munich, Germany; Munich Center for Machine Learning, Munich, Germany; Munich Data Science Institute, Munich, Germany.
| |
Collapse
|
13
|
Liu Q, Zhang X, Jiang X, Zhang C, Li J, Zhang X, Yang J, Yu N, Zhu Y, Liu J, Xie F, Li Y, Hao Y, Feng Y, Wang Q, Gao Q, Zhang W, Zhang T, Dong T, Cui B. A Histopathologic Image Analysis for the Classification of Endocervical Adenocarcinoma Silva Patterns Depend on Weakly Supervised Deep Learning. THE AMERICAN JOURNAL OF PATHOLOGY 2024; 194:735-746. [PMID: 38382842 DOI: 10.1016/j.ajpath.2024.01.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Revised: 12/25/2023] [Accepted: 01/18/2024] [Indexed: 02/23/2024]
Abstract
Twenty-five percent of cervical cancers are classified as endocervical adenocarcinomas (EACs), which comprise a highly heterogeneous group of tumors. A histopathologic risk stratification system known as the Silva pattern system was developed based on morphology. However, accurately classifying such patterns can be challenging. The study objective was to develop a deep learning pipeline (Silva3-AI) that automatically analyzes whole slide image-based histopathologic images and identifies Silva patterns with high accuracy. Initially, a total of 202 patients with EACs and histopathologic slides were obtained from Qilu Hospital of Shandong University for developing and internally testing the Silva3-AI model. Subsequently, an additional 161 patients and slides were collected from seven other medical centers for independent testing. The Silva3-AI model was developed using a vision transformer and recurrent neural network architecture, utilizing multi-magnification patches, and its performance was evaluated based on a class-specific area under the receiver-operating characteristic curve. Silva3-AI achieved a class-specific area under the receiver-operating characteristic curve of 0.947 for Silva A, 0.908 for Silva B, and 0.947 for Silva C on the independent test set. Notably, the performance of Silva3-AI was consistent with that of professional pathologists with 10 years' diagnostic experience. Furthermore, the visualization of prediction heatmaps facilitated the identification of tumor microenvironment heterogeneity, which is known to contribute to variations in Silva patterns.
Collapse
Affiliation(s)
- Qingqing Liu
- Cheeloo College of Medicine, Shandong University, Jinan City, China
| | - Xiaofang Zhang
- Department of Pathology, School of Basic Medical Sciences and Qilu Hospital, Shandong University, Jinan City, China
| | - Xuji Jiang
- Cheeloo College of Medicine, Shandong University, Jinan City, China
| | - Chunyan Zhang
- Department of Pathology, Affiliated Hospital of Jining Medical University of Shandong, Jining City, China
| | - Jiamei Li
- Department of Pathology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, Jinan City, China
| | - Xuedong Zhang
- Department of Pathology, Liaocheng People's Hospital, Liaocheng City, China
| | - Jingyan Yang
- Department of Pathology, The Second Hospital of Shandong University, Jinan City, China
| | - Ning Yu
- Department of Pathology, Binzhou Medical University Hospital, Binzhou City, China
| | - Yongcun Zhu
- Department of Pathology, Weihai Municipal Hospital of Shandong University, Weihai City, China
| | - Jing Liu
- Department of Pathology, Jining No. 1 People's Hospital, Jining City, China
| | - Fengxiang Xie
- Department of Pathology, KingMed Diagnostics, Jinan City, China
| | - Yawen Li
- Department of Pathology, School of Basic Medical Sciences and Qilu Hospital, Shandong University, Jinan City, China
| | - Yiping Hao
- Cheeloo College of Medicine, Shandong University, Jinan City, China
| | - Yuan Feng
- Cheeloo College of Medicine, Shandong University, Jinan City, China
| | - Qi Wang
- Department of Obstetrics and Gynecology, Shandong Provincial Qianfoshan Hospital, Shandong University, Jinan City, China
| | - Qun Gao
- Department of Obstetrics and Gynecology, The Affiliated Hospital of Qingdao University, Qingdao City, China
| | - Wenjing Zhang
- Department of Obstetrics and Gynecology, Qilu Hospital of Shandong University, Jinan City, China
| | - Teng Zhang
- Department of Obstetrics and Gynecology, Qilu Hospital of Shandong University, Jinan City, China
| | - Taotao Dong
- Department of Obstetrics and Gynecology, Qilu Hospital of Shandong University, Jinan City, China.
| | - Baoxia Cui
- Department of Obstetrics and Gynecology, Qilu Hospital of Shandong University, Jinan City, China.
| |
Collapse
|
14
|
Zhao J, Jiang T, Lin Y, Chan LC, Chan PK, Wen C, Chen H. Adaptive Fusion of Deep Learning With Statistical Anatomical Knowledge for Robust Patella Segmentation From CT Images. IEEE J Biomed Health Inform 2024; 28:2842-2853. [PMID: 38446653 DOI: 10.1109/jbhi.2024.3372576] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/08/2024]
Abstract
Kneeosteoarthritis (KOA), as a leading joint disease, can be decided by examining the shapes of patella to spot potential abnormal variations. To assist doctors in the diagnosis of KOA, a robust automatic patella segmentation method is highly demanded in clinical practice. Deep learning methods, especially convolutional neural networks (CNNs) have been widely applied to medical image segmentation in recent years. Nevertheless, poor image quality and limited data still impose challenges to segmentation via CNNs. On the other hand, statistical shape models (SSMs) can generate shape priors which give anatomically reliable segmentation to varying instances. Thus, in this work, we propose an adaptive fusion framework, explicitly combining deep neural networks and anatomical knowledge from SSM for robust patella segmentation. Our adaptive fusion framework will accordingly adjust the weight of segmentation candidates in fusion based on their segmentation performance. We also propose a voxel-wise refinement strategy to make the segmentation of CNNs more anatomically correct. Extensive experiments and thorough assessment have been conducted on various mainstream CNN backbones for patella segmentation in low-data regimes, which demonstrate that our framework can be flexibly attached to a CNN model, significantly improving its performance when labeled training data are limited and input image data are of poor quality.
Collapse
|
15
|
Wang Y, Zhang W, Chen L, Xie J, Zheng X, Jin Y, Zheng Q, Xue Q, Li B, He C, Chen H, Li Y. Development of an Interpretable Deep Learning Model for Pathological Tumor Response Assessment After Neoadjuvant Therapy. Biol Proced Online 2024; 26:10. [PMID: 38632527 PMCID: PMC11022344 DOI: 10.1186/s12575-024-00234-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2024] [Accepted: 03/22/2024] [Indexed: 04/19/2024] Open
Abstract
BACKGROUND Neoadjuvant therapy followed by surgery has become the standard of care for locally advanced esophageal squamous cell carcinoma (ESCC) and accurate pathological response assessment is critical to assess the therapeutic efficacy. However, it can be laborious and inconsistency between different observers may occur. Hence, we aim to develop an interpretable deep-learning model for efficient pathological response assessment following neoadjuvant therapy in ESCC. METHODS This retrospective study analyzed 337 ESCC resection specimens from 2020-2021 at the Pudong-Branch (Cohort 1) and 114 from 2021-2022 at the Puxi-Branch (External Cohort 2) of Fudan University Shanghai Cancer Center. Whole slide images (WSIs) from these two cohorts were generated using different scanning machines to test the ability of the model in handling color variations. Four pathologists independently assessed the pathological response. The senior pathologists annotated tumor beds and residual tumor percentages on WSIs to determine consensus labels. Furthermore, 1850 image patches were randomly extracted from Cohort 1 WSIs and binarily classified for tumor viability. A deep-learning model employing knowledge distillation was developed to automatically classify positive patches for each WSI and estimate the viable residual tumor percentages. Spatial heatmaps were output for model explanations and visualizations. RESULTS The approach achieved high concordance with pathologist consensus, with an R^2 of 0.8437, a RAcc_0.1 of 0.7586, a RAcc_0.3 of 0.9885, which were comparable to two senior pathologists (R^2 of 0.9202/0.9619, RAcc_0.1 of 8506/0.9425, RAcc_0.3 of 1.000/1.000) and surpassing two junior pathologists (R^2 of 0.5592/0.5474, RAcc_0.1 of 0.5287/0.5287, RAcc_0.3 of 0.9080/0.9310). Visualizations enabled the localization of residual viable tumor to augment microscopic assessment. CONCLUSION This work illustrates deep learning's potential for assisting pathological response assessment. Spatial heatmaps and patch examples provide intuitive explanations of model predictions, engendering clinical trust and adoption (Code and data will be available at https://github.com/WinnieLaugh/ESCC_Percentage once the paper has been conditionally accepted). Integrating interpretable computational pathology could help enhance the efficiency and consistency of tumor response assessment and empower precise oncology treatment decisions.
Collapse
Affiliation(s)
- Yichen Wang
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China, 200032
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
| | - Wenhua Zhang
- Shanghai Aitrox Technology Corporation Limited, Shanghai, China
- Department of Future Technology, Shanghai University, Shanghai, China
| | - Lijun Chen
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China, 200032
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
| | - Jun Xie
- Shanghai Aitrox Technology Corporation Limited, Shanghai, China
| | - Xuebin Zheng
- Shanghai Aitrox Technology Corporation Limited, Shanghai, China
| | - Yan Jin
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China, 200032
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
| | - Qiang Zheng
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China, 200032
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
| | - Qianqian Xue
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China, 200032
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
| | - Bin Li
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
- Department of Thoracic Surgery, Fudan University Shanghai Cancer Center, Shanghai, China
| | - Chuan He
- Shanghai Aitrox Technology Corporation Limited, Shanghai, China
| | - Haiquan Chen
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032
- Department of Thoracic Surgery, Fudan University Shanghai Cancer Center, Shanghai, China
| | - Yuan Li
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China, 200032.
- Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, China, 200032.
| |
Collapse
|
16
|
Omar M, Xu Z, Rand SB, Alexanderani MK, Salles DC, Valencia I, Schaeffer EM, Robinson BD, Lotan TL, Loda M, Marchionni L. Semi-Supervised, Attention-Based Deep Learning for Predicting TMPRSS2:ERG Fusion Status in Prostate Cancer Using Whole Slide Images. Mol Cancer Res 2024; 22:347-359. [PMID: 38284821 PMCID: PMC10985477 DOI: 10.1158/1541-7786.mcr-23-0639] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Revised: 12/26/2023] [Accepted: 01/22/2024] [Indexed: 01/30/2024]
Abstract
IMPLICATIONS Our study illuminates the potential of deep learning in effectively inferring key prostate cancer genetic alterations from the tissue morphology depicted in routinely available histology slides, offering a cost-effective method that could revolutionize diagnostic strategies in oncology.
Collapse
Affiliation(s)
- Mohamed Omar
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
- Dana-Farber Cancer Institute, Boston, Massachusetts
| | - Zhuoran Xu
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
- Dana-Farber Cancer Institute, Boston, Massachusetts
| | - Sophie B. Rand
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
- Dana-Farber Cancer Institute, Boston, Massachusetts
| | | | - Daniela C. Salles
- Department of Pathology, Johns Hopkins University, Baltimore, Maryland
| | - Itzel Valencia
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
| | | | - Brian D. Robinson
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
| | - Tamara L. Lotan
- Department of Pathology, Johns Hopkins University, Baltimore, Maryland
| | - Massimo Loda
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
| | - Luigi Marchionni
- Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, New York
| |
Collapse
|
17
|
Liang M, Jiang X, Cao J, Zhang S, Liu H, Li B, Wang L, Zhang C, Jia X. HSG-MGAF Net: Heterogeneous subgraph-guided multiscale graph attention fusion network for interpretable prediction of whole-slide image. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 247:108099. [PMID: 38442623 DOI: 10.1016/j.cmpb.2024.108099] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/17/2023] [Revised: 02/12/2024] [Accepted: 02/22/2024] [Indexed: 03/07/2024]
Abstract
BACKGROUND AND OBJECTIVE Pathological whole slide image (WSI) prediction and region of interest (ROI) localization are important issues in computer-aided diagnosis and postoperative analysis in clinical applications. Existing computer-aided methods for predicting WSI are mainly based on multiple instance learning (MIL) and its variants. However, most of the methods are based on instance independence and identical distribution assumption and performed at a single scale, which not fully exploit the hierarchical multiscale heterogeneous information contained in WSI. METHODS Heterogeneous Subgraph-Guided Multiscale Graph Attention Fusion Network (HSG-MGAF Net) is proposed to build the topology of critical image patches at two scales for adaptive WSI prediction and lesion localization. The HSG-MGAF Net simulates the hierarchical heterogeneous information of WSI through graph and hypergraph at two scales, respectively. This framework not only fully exploits the low-order and potential high-order correlations of image patches at each scale, but also leverages the heterogeneous information of the two scales for adaptive WSI prediction. RESULTS We validate the superiority of the proposed method on the CAMELYON16 and the TCGA- NSCLC, and the results show that HSG-MGAF Net outperforms the state-of-the-art method on both datasets. The average ACC, AUC and F1 score of HSG-MGAF Net can reach 92.7 %/0.951/0.892 and 92.2 %/0.957/0.919, respectively. The obtained heatmaps can also localize the positive regions more accurately, which have great consistency with the pixel-level labels. CONCLUSIONS The results demonstrate that HSG-MGAF Net outperforms existing weakly supervised learning methods by introducing critical heterogeneous information between the two scales. This approach paves the way for further research on light weighted heterogeneous graph-based WSI prediction and ROI localization.
Collapse
Affiliation(s)
- Meiyan Liang
- School of Physics and Electronic Engineering, Shanxi University, Taiyuan 030006, China.
| | - Xing Jiang
- School of Physics and Electronic Engineering, Shanxi University, Taiyuan 030006, China
| | - Jie Cao
- School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China.
| | - Shupeng Zhang
- School of Physics and Electronic Engineering, Shanxi University, Taiyuan 030006, China
| | - Haishun Liu
- Department of Automation, Tsinghua University, Beijing 100084, China
| | - Bo Li
- Department of Rehabilitation Treatment, Shanxi Rongjun Hospital, Taiyuan 030000, China
| | - Lin Wang
- Department of Pathology, Shanxi Bethune Hospital, Shanxi Academy of Medical Sciences, Tongji Shanxi Hospital, Third Hospital of Shanxi Medical University, Taiyuan 030032, China
| | - Cunlin Zhang
- Department of physics, Capital Normal University, Beijing 100048, China
| | - Xiaojun Jia
- School of Physics and Electronic Engineering, Shanxi University, Taiyuan 030006, China
| |
Collapse
|
18
|
Bao LX, Luo ZM, Zhu XL, Xu YY. Automated identification of protein expression intensity and classification of protein cellular locations in mouse brain regions from immunofluorescence images. Med Biol Eng Comput 2024; 62:1105-1119. [PMID: 38150111 DOI: 10.1007/s11517-023-02985-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Accepted: 11/28/2023] [Indexed: 12/28/2023]
Abstract
Knowledge of protein expression in mammalian brains at regional and cellular levels can facilitate understanding of protein functions and associated diseases. As the mouse brain is a typical mammalian brain considering cell type and structure, several studies have been conducted to analyze protein expression in mouse brains. However, labeling protein expression using biotechnology is costly and time-consuming. Therefore, automated models that can accurately recognize protein expression are needed. Here, we constructed machine learning models to automatically annotate the protein expression intensity and cellular location in different mouse brain regions from immunofluorescence images. The brain regions and sub-regions were segmented through learning image features using an autoencoder and then performing K-means clustering and registration to align with the anatomical references. The protein expression intensities for those segmented structures were computed on the basis of the statistics of the image pixels, and patch-based weakly supervised methods and multi-instance learning were used to classify the cellular locations. Results demonstrated that the models achieved high accuracy in the expression intensity estimation, and the F1 score of the cellular location prediction was 74.5%. This work established an automated pipeline for analyzing mouse brain images and provided a foundation for further study of protein expression and functions.
Collapse
Affiliation(s)
- Lin-Xia Bao
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China
- Guangdong Provincial Key Laboratory of Medical Imaging Processing, Southern Medical University, Guangzhou, 510515, China
- Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510623, China
| | - Zhuo-Ming Luo
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China
- Guangdong Provincial Key Laboratory of Medical Imaging Processing, Southern Medical University, Guangzhou, 510515, China
- Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510623, China
| | - Xi-Liang Zhu
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China
- Guangdong Provincial Key Laboratory of Medical Imaging Processing, Southern Medical University, Guangzhou, 510515, China
- Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510623, China
| | - Ying-Ying Xu
- School of Biomedical Engineering, Southern Medical University, Guangzhou, 510515, China.
- Guangdong Provincial Key Laboratory of Medical Imaging Processing, Southern Medical University, Guangzhou, 510515, China.
- Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou, 510623, China.
| |
Collapse
|
19
|
Yao J, Han L, Guo G, Zheng Z, Cong R, Huang X, Ding J, Yang K, Zhang D, Han J. Position-based anchor optimization for point supervised dense nuclei detection. Neural Netw 2024; 171:159-170. [PMID: 38091760 DOI: 10.1016/j.neunet.2023.12.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 10/10/2023] [Accepted: 12/04/2023] [Indexed: 01/29/2024]
Abstract
Nuclei detection is one of the most fundamental and challenging problems in histopathological image analysis, which can localize nuclei to provide effective computer-aided cancer diagnosis, treatment decision, and prognosis. The fully-supervised nuclei detector requires a large number of nuclei annotations on high-resolution digital images, which is time-consuming and needs human annotators with professional knowledge. In recent years, weakly-supervised learning has attracted significant attention in reducing the labeling burden. However, detecting dense nuclei of complex crowded distribution and diverse appearances remains a challenge. To solve this problem, we propose a novel point-supervised dense nuclei detection framework that introduces position-based anchor optimization to complete morphology-based pseudo-label supervision. Specifically, we first generate cellular-level pseudo labels (CPL) for the detection head via a morphology-based mechanism, which can help to build a baseline point-supervised detection network. Then, considering the crowded distribution of the dense nuclei, we propose a mechanism called Position-based Anchor-quality Estimation (PAE), which utilizes the positional deviation between an anchor and its corresponding point label to suppress low-quality detections far from each nucleus. Finally, to better handle the diverse appearances of nuclei, an Adaptive Anchor Selector (AAS) operation is proposed to automatically select positive and negative anchors according to morphological and positional statistical characteristics of nuclei. We conduct comprehensive experiments on two widely used benchmarks, MO and Lizard, using ResNet50 and PVTv2 as backbones. The results demonstrate that the proposed approach has superior capacity compared with other state-of-the-art methods. In particularly, in dense nuclei scenarios, our method can achieve 95.1% performance of the fully-supervised approach. The code is available at https://github.com/NucleiDet/DenseNucleiDet.
Collapse
Affiliation(s)
- Jieru Yao
- Brain and Artificial Intelligence Lab, School of Automation, Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China
| | - Longfei Han
- School of Computer Science, Beijing Technology and Business University, Beijing, 100048, China; Hefei Comprehensive National Science Center, Hefei, Anhui, 230088, China
| | - Guangyu Guo
- Brain and Artificial Intelligence Lab, School of Automation, Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China
| | - Zhaohui Zheng
- Department of Clinical Immunology, Xijing Hospital, The Fourth Military Medical University, Xi'an, Shaanxi, 710032, China.
| | - Runmin Cong
- School of Control Science and Engineering, Shandong University, Jinan, Shandong, 250100, China
| | - Xiankai Huang
- Beijing Technology and Business University, Beijing, 100048, China
| | - Jin Ding
- Department of Clinical Immunology, Xijing Hospital, The Fourth Military Medical University, Xi'an, Shaanxi, 710032, China
| | - Kaihui Yang
- School of software, Nanchang University, Nanchang, Jiangxi, 330031, China
| | - Dingwen Zhang
- Brain and Artificial Intelligence Lab, School of Automation, Northwestern Polytechnical University, Xi'an, Shaanxi, 710072, China; Hefei Comprehensive National Science Center, Hefei, Anhui, 230088, China; Department of Clinical Immunology, Xijing Hospital, The Fourth Military Medical University, Xi'an, Shaanxi, 710032, China.
| | - Junwei Han
- Hefei Comprehensive National Science Center, Hefei, Anhui, 230088, China
| |
Collapse
|
20
|
Feng X, Shu W, Li M, Li J, Xu J, He M. Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview. J Transl Med 2024; 22:131. [PMID: 38310237 PMCID: PMC10837897 DOI: 10.1186/s12967-024-04915-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Accepted: 01/20/2024] [Indexed: 02/05/2024] Open
Abstract
The capability to gather heterogeneous data, alongside the increasing power of artificial intelligence to examine it, leading a revolution in harnessing multimodal data in the life sciences. However, most approaches are limited to unimodal data, leaving integrated approaches across modalities relatively underdeveloped in computational pathology. Pathogenomics, as an invasive method to integrate advanced molecular diagnostics from genomic data, morphological information from histopathological imaging, and codified clinical data enable the discovery of new multimodal cancer biomarkers to propel the field of precision oncology in the coming decade. In this perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods in pathogenomics. It includes correlation between the pathological and genomic profile of cancer, fusion of histology, and genomics profile of cancer. We also present challenges, opportunities, and avenues for future work.
Collapse
Affiliation(s)
- Xiaobing Feng
- College of Electrical and Information Engineering, Hunan University, Changsha, China
- Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, Zhejiang, China
| | - Wen Shu
- College of Electrical and Information Engineering, Hunan University, Changsha, China
- Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, Zhejiang, China
| | - Mingya Li
- College of Electrical and Information Engineering, Hunan University, Changsha, China
- Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, Zhejiang, China
| | - Junyu Li
- College of Electrical and Information Engineering, Hunan University, Changsha, China
- Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, Zhejiang, China
| | - Junyao Xu
- Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, Zhejiang, China
| | - Min He
- College of Electrical and Information Engineering, Hunan University, Changsha, China.
- Zhejiang Cancer Hospital, Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, 310022, Zhejiang, China.
| |
Collapse
|
21
|
Luo X, Qu L, Guo Q, Song Z, Wang M. Negative Instance Guided Self-Distillation Framework for Whole Slide Image Analysis. IEEE J Biomed Health Inform 2024; 28:964-975. [PMID: 37494153 DOI: 10.1109/jbhi.2023.3298798] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/28/2023]
Abstract
Histopathology image classification is an important clinical task, and current deep learning-based whole-slide image (WSI) classification methods typically cut WSIs into small patches and cast the problem as multi-instance learning. The mainstream approach is to train a bag-level classifier, but their performance on both slide classification and positive patch localization is limited because the instance-level information is not fully explored. In this article, we propose a negative instance-guided, self-distillation framework to directly train an instance-level classifier end-to-end. Instead of depending only on the self-supervised training of the teacher and the student classifiers in a typical self-distillation framework, we input the true negative instances into the student classifier to guide the classifier to better distinguish positive and negative instances. In addition, we propose a prediction bank to constrain the distribution of pseudo instance labels generated by the teacher classifier to prevent the self-distillation from falling into the degeneration of classifying all instances as negative. We conduct extensive experiments and analysis on three publicly available pathological datasets: CAMELYON16, PANDA, and TCGA, as well as an in-house pathological dataset for cervical cancer lymph node metastasis prediction. The results show that our method outperforms existing methods by a large margin. Code will be publicly available.
Collapse
|
22
|
Wang W, Ruan S, Xie Y, Fang S, Yang J, Li X, Zhang Y. Development and Validation of a Pathomics Model Using Machine Learning to Predict CXCL8 Expression and Prognosis in Head and Neck Cancer. Clin Exp Otorhinolaryngol 2024; 17:85-97. [PMID: 38246983 PMCID: PMC10933807 DOI: 10.21053/ceo.2023.00026] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Revised: 01/06/2024] [Accepted: 01/19/2024] [Indexed: 01/23/2024] Open
Abstract
OBJECTIVES The necessity to develop a method for prognostication and to identify novel biomarkers for personalized medicine in patients with head and neck squamous cell carcinoma (HNSCC) cannot be overstated. Recently, pathomics, which relies on quantitative analysis of medical imaging, has come to the forefront. CXCL8, an essential inflammatory cytokine, has been shown to correlate with overall survival (OS). This study examined the relationship between CXCL8 mRNA expression and pathomics features and aimed to explore the biological underpinnings of CXCL8. METHODS Clinical information and transcripts per million mRNA sequencing data were obtained from The Cancer Genome Atlas (TCGA)-HNSCC dataset. We identified correlations between CXCL8 mRNA expression and patient survival rates using a Kaplan-Meier survival curve. A retrospective analysis of 313 samples diagnosed with HNSCC in the TCGA database was conducted. Pathomics features were extracted from hematoxylin and eosin-stained images, and then the minimum redundancy maximum relevance, with recursive feature elimination (mRMR-RFE) method was applied, followed by screening with the logistic regression algorithm. RESULTS Kaplan-Meier curves indicated that high expression of CXCL8 was significantly associated with decreased OS. The logistic regression pathomics model incorporated 16 radiomics features identified by the mRMR-RFE method in the training set and demonstrated strong performance in the testing set. Calibration plots showed that the probability of high gene expression predicted by the pathomics model was in good agreement with actual observations, suggesting the model's high clinical applicability. CONCLUSION The pathomics model of CXCL8 mRNA expression serves as an effective tool for predicting prognosis in patients with HNSCC and can aid in clinical decision-making. Elevated levels of CXCL8 expression may lead to reduced DNA damage and are associated with a pro-inflammatory tumor microenvironment, offering a potential therapeutic target.
Collapse
Affiliation(s)
- Weihua Wang
- Department of Otolaryngology-Head and Neck Surgery, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Suyu Ruan
- Department of Otolaryngology-Head and Neck Surgery, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Yuhang Xie
- Department of Otolaryngology-Head and Neck Surgery, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Shengjian Fang
- Department of Otolaryngology-Head and Neck Surgery, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Junxian Yang
- Department of Otolaryngology-Head and Neck Surgery, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Xueyan Li
- Department of Nursing, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| | - Yu Zhang
- Department of Otolaryngology-Head and Neck Surgery, Shanghai East Hospital, School of Medicine, Tongji University, Shanghai, China
| |
Collapse
|
23
|
Li J, Wang D, Zhang C. Establishment of a pathomic-based machine learning model to predict CD276 (B7-H3) expression in colon cancer. Front Oncol 2024; 13:1232192. [PMID: 38260829 PMCID: PMC10802857 DOI: 10.3389/fonc.2023.1232192] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 11/29/2023] [Indexed: 01/24/2024] Open
Abstract
CD276 is a promising prognostic indicator and an attractive therapeutic target in various malignancies. However, current methods for CD276 detection are time-consuming and expensive, limiting extensive studies and applications of CD276. We aimed to develop a pathomic model for CD276 prediction from H&E-stained pathological images, and explore the underlying mechanism of the pathomic features by associating the pathomic model with transcription profiles. A dataset of colon adenocarcinoma (COAD) patients was retrieved from the Cancer Genome Atlas (TCGA) database. The dataset was divided into the training and validation sets according to the ratio of 8:2 by a stratified sampling method. Using the gradient boosting machine (GBM) algorithm, we established a pathomic model to predict CD276 expression in COAD. Univariate and multivariate Cox regression analyses were conducted to assess the predictive performance of the pathomic model for overall survival in COAD. Gene Set Enrichment Analysis (GESA) was performed to explore the underlying biological mechanisms of the pathomic model. The pathomic model formed by three pathomic features for CD276 prediction showed an area under the curve (AUC) of 0.833 (95%CI: 0.784-0.882) in the training set and 0.758 (95%CI: 0.637-0.878) in the validation set, respectively. The calibration curves and Hosmer-Lemeshow goodness of fit test showed that the prediction probability of high/low expression of CD276 was in favorable agreement with the real situation in both the training and validation sets (P=0.176 and 0.255, respectively). The DCA curves suggested that the pathomic model acquired high clinical benefit. All the subjects were categorized into high pathomic score (PS) (PS-H) and low PS (PS-L) groups according to the cutoff value of PS. Univariate and multivariate Cox regression analysis indicated that PS was a risk factor for overall survival in COAD. Furthermore, through GESA analysis, we found several immune and inflammatory-related pathways and genes were associated with the pathomic model. We constructed a pathomics-based machine learning model for CD276 prediction directly from H&E-stained images in COAD. Through integrated analysis of the pathomic model and transcriptomics, the interpretability of the pathomic model provide a theoretical basis for further hypothesis and experimental research.
Collapse
Affiliation(s)
- Jia Li
- Department of Gastroenterology, The 983rd Hospital of Joint Logistic Support Force of PLA, Tianjin, China
| | - Dongxu Wang
- Department of Gastroenterology, The 983rd Hospital of Joint Logistic Support Force of PLA, Tianjin, China
| | - Chenxin Zhang
- Department of General Surgery, The 983rd Hospital of Joint Logistic Support Force of PLA, Tianjin, China
| |
Collapse
|
24
|
Schreiber BA, Denholm J, Jaeckle F, Arends MJ, Branson KM, Schönlieb CB, Soilleux EJ. Rapid artefact removal and H&E-stained tissue segmentation. Sci Rep 2024; 14:309. [PMID: 38172562 PMCID: PMC10764721 DOI: 10.1038/s41598-023-50183-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2023] [Accepted: 12/16/2023] [Indexed: 01/05/2024] Open
Abstract
We present an innovative method for rapidly segmenting haematoxylin and eosin (H&E)-stained tissue in whole-slide images (WSIs) that eliminates a wide range of undesirable artefacts such as pen marks and scanning artefacts. Our method involves taking a single-channel representation of a low-magnification RGB overview of the WSI in which the pixel values are bimodally distributed such that H&E-stained tissue is easily distinguished from both background and a wide variety of artefacts. We demonstrate our method on 30 WSIs prepared from a wide range of institutions and WSI digital scanners, each containing substantial artefacts, and compare it to segmentations provided by Otsu thresholding and Histolab tissue segmentation and pen filtering tools. We found that our method segmented the tissue and fully removed all artefacts in 29 out of 30 WSIs, whereas Otsu thresholding failed to remove any artefacts, and the Histolab pen filtering tools only partially removed the pen marks. The beauty of our approach lies in its simplicity: manipulating RGB colour space and using Otsu thresholding allows for the segmentation of H&E-stained tissue and the rapid removal of artefacts without the need for machine learning or parameter tuning.
Collapse
Affiliation(s)
- B A Schreiber
- Department of Pathology, University of Cambridge, Tennis Court Road, Cambridge, CB2 1QP, Cambridgeshire, UK.
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, Cambridge, CB3 0WA, Cambridgeshire, UK.
| | - J Denholm
- Department of Pathology, University of Cambridge, Tennis Court Road, Cambridge, CB2 1QP, Cambridgeshire, UK
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, Cambridge, CB3 0WA, Cambridgeshire, UK
- Lyzeum Ltd., Cambridge, CB1 2LA, Cambridgeshire, UK
| | - F Jaeckle
- Department of Pathology, University of Cambridge, Tennis Court Road, Cambridge, CB2 1QP, Cambridgeshire, UK
- Lyzeum Ltd., Cambridge, CB1 2LA, Cambridgeshire, UK
| | - M J Arends
- Edinburgh Pathology, Institute of Genetics and Cancer, University of Edinburgh, Crewe Road, Edinburgh, EH4 2XR, UK
| | - K M Branson
- Artificial Intelligence and Machine Learning, GSK plc., Great West Road, Brentford, TW8 9GS, Middlesex, UK
| | - C-B Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, Cambridge, CB3 0WA, Cambridgeshire, UK
- Lyzeum Ltd., Cambridge, CB1 2LA, Cambridgeshire, UK
| | - E J Soilleux
- Department of Pathology, University of Cambridge, Tennis Court Road, Cambridge, CB2 1QP, Cambridgeshire, UK.
- Lyzeum Ltd., Cambridge, CB1 2LA, Cambridgeshire, UK.
| |
Collapse
|
25
|
Kim M, Quiñones Robles WR, Ko YS, Wong B, Lee S, Yi MY. A predicted-loss based active learning approach for robust cancer pathology image analysis in the workplace. BMC Med Imaging 2024; 24:5. [PMID: 38166690 PMCID: PMC10763414 DOI: 10.1186/s12880-023-01170-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 12/01/2023] [Indexed: 01/05/2024] Open
Abstract
BACKGROUND Convolutional neural network-based image processing research is actively being conducted for pathology image analysis. As a convolutional neural network model requires a large amount of image data for training, active learning (AL) has been developed to produce efficient learning with a small amount of training data. However, existing studies have not specifically considered the characteristics of pathological data collected from the workplace. For various reasons, noisy patches can be selected instead of clean patches during AL, thereby reducing its efficiency. This study proposes an effective AL method for cancer pathology that works robustly on noisy datasets. METHODS Our proposed method to develop a robust AL approach for noisy histopathology datasets consists of the following three steps: 1) training a loss prediction module, 2) collecting predicted loss values, and 3) sampling data for labeling. This proposed method calculates the amount of information in unlabeled data as predicted loss values and removes noisy data based on predicted loss values to reduce the rate at which noisy data are selected from the unlabeled dataset. We identified a suitable threshold for optimizing the efficiency of AL through sensitivity analysis. RESULTS We compared the results obtained with the identified threshold with those of existing representative AL methods. In the final iteration, the proposed method achieved a performance of 91.7% on the noisy dataset and 92.4% on the clean dataset, resulting in a performance reduction of less than 1%. Concomitantly, the noise selection ratio averaged only 2.93% on each iteration. CONCLUSIONS The proposed AL method showed robust performance on datasets containing noisy data by avoiding data selection in predictive loss intervals where noisy data are likely to be distributed. The proposed method contributes to medical image analysis by screening data and producing a robust and effective classification model tailored for cancer pathology image processing in the workplace.
Collapse
Grants
- G01180115 Seegene Medical Foundation, South Korea, under the project "Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning"
- G01180115 Seegene Medical Foundation, South Korea, under the project "Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning"
- G01180115 Seegene Medical Foundation, South Korea, under the project "Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning"
- G01180115 Seegene Medical Foundation, South Korea, under the project "Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning"
- G01180115 Seegene Medical Foundation, South Korea, under the project "Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning"
- G01180115 Seegene Medical Foundation, South Korea, under the project "Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning"
- Seegene Medical Foundation, South Korea, under the project “Research on Developing a Next Generation Medical Diagnosis System Using Deep Learning”
Collapse
Affiliation(s)
- Mujin Kim
- Graduate School of Data Science, Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Willmer Rafell Quiñones Robles
- Graduate School of Data Science, Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Young Sin Ko
- Pathology Center, Seegene Medical Foundation, Seoul, South Korea
| | - Bryan Wong
- Graduate School of Data Science, Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Sol Lee
- Graduate School of Data Science, Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Mun Yong Yi
- Graduate School of Data Science, Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea.
| |
Collapse
|
26
|
Sharma P, Nayak DR, Balabantaray BK, Tanveer M, Nayak R. A survey on cancer detection via convolutional neural networks: Current challenges and future directions. Neural Netw 2024; 169:637-659. [PMID: 37972509 DOI: 10.1016/j.neunet.2023.11.006] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Revised: 10/21/2023] [Accepted: 11/04/2023] [Indexed: 11/19/2023]
Abstract
Cancer is a condition in which abnormal cells uncontrollably split and damage the body tissues. Hence, detecting cancer at an early stage is highly essential. Currently, medical images play an indispensable role in detecting various cancers; however, manual interpretation of these images by radiologists is observer-dependent, time-consuming, and tedious. An automatic decision-making process is thus an essential need for cancer detection and diagnosis. This paper presents a comprehensive survey on automated cancer detection in various human body organs, namely, the breast, lung, liver, prostate, brain, skin, and colon, using convolutional neural networks (CNN) and medical imaging techniques. It also includes a brief discussion about deep learning based on state-of-the-art cancer detection methods, their outcomes, and the possible medical imaging data used. Eventually, the description of the dataset used for cancer detection, the limitations of the existing solutions, future trends, and challenges in this domain are discussed. The utmost goal of this paper is to provide a piece of comprehensive and insightful information to researchers who have a keen interest in developing CNN-based models for cancer detection.
Collapse
Affiliation(s)
- Pallabi Sharma
- School of Computer Science, UPES, Dehradun, 248007, Uttarakhand, India.
| | - Deepak Ranjan Nayak
- Department of Computer Science and Engineering, Malaviya National Institute of Technology, Jaipur, 302017, Rajasthan, India.
| | - Bunil Kumar Balabantaray
- Computer Science and Engineering, National Institute of Technology Meghalaya, Shillong, 793003, Meghalaya, India.
| | - M Tanveer
- Department of Mathematics, Indian Institute of Technology Indore, Simrol, 453552, Indore, India.
| | - Rajashree Nayak
- School of Applied Sciences, Birla Global University, Bhubaneswar, 751029, Odisha, India.
| |
Collapse
|
27
|
Mukashyaka P, Sheridan TB, Foroughi Pour A, Chuang JH. SAMPLER: unsupervised representations for rapid analysis of whole slide tissue images. EBioMedicine 2024; 99:104908. [PMID: 38101298 PMCID: PMC10733087 DOI: 10.1016/j.ebiom.2023.104908] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 11/27/2023] [Accepted: 11/27/2023] [Indexed: 12/17/2023] Open
Abstract
BACKGROUND Deep learning has revolutionized digital pathology, allowing automatic analysis of hematoxylin and eosin (H&E) stained whole slide images (WSIs) for diverse tasks. WSIs are broken into smaller images called tiles, and a neural network encodes each tile. Many recent works use supervised attention-based models to aggregate tile-level features into a slide-level representation, which is then used for downstream analysis. Training supervised attention-based models is computationally intensive, architecture optimization of the attention module is non-trivial, and labeled data are not always available. Therefore, we developed an unsupervised and fast approach called SAMPLER to generate slide-level representations. METHODS Slide-level representations of SAMPLER are generated by encoding the cumulative distribution functions of multiscale tile-level features. To assess effectiveness of SAMPLER, slide-level representations of breast carcinoma (BRCA), non-small cell lung carcinoma (NSCLC), and renal cell carcinoma (RCC) WSIs of The Cancer Genome Atlas (TCGA) were used to train separate classifiers distinguishing tumor subtypes in FFPE and frozen WSIs. In addition, BRCA and NSCLC classifiers were externally validated on frozen WSIs. Moreover, SAMPLER's attention maps identify regions of interest, which were evaluated by a pathologist. To determine time efficiency of SAMPLER, we compared runtime of SAMPLER with two attention-based models. SAMPLER concepts were used to improve the design of a context-aware multi-head attention model (context-MHA). FINDINGS SAMPLER-based classifiers were comparable to state-of-the-art attention deep learning models to distinguish subtypes of BRCA (AUC = 0.911 ± 0.029), NSCLC (AUC = 0.940 ± 0.018), and RCC (AUC = 0.987 ± 0.006) on FFPE WSIs (internal test sets). However, training SAMLER-based classifiers was >100 times faster. SAMPLER models successfully distinguished tumor subtypes on both internal and external test sets of frozen WSIs. Histopathological review confirmed that SAMPLER-identified high attention tiles contained subtype-specific morphological features. The improved context-MHA distinguished subtypes of BRCA and RCC (BRCA-AUC = 0.921 ± 0.027, RCC-AUC = 0.988 ± 0.010) with increased accuracy on internal test FFPE WSIs. INTERPRETATION Our unsupervised statistical approach is fast and effective for analyzing WSIs, with greatly improved scalability over attention-based deep learning methods. The high accuracy of SAMPLER-based classifiers and interpretable attention maps suggest that SAMPLER successfully encodes the distinct morphologies within WSIs and will be applicable to general histology image analysis problems. FUNDING This study was supported by the National Cancer Institute (Grant No. R01CA230031 and P30CA034196).
Collapse
Affiliation(s)
- Patience Mukashyaka
- The Jackson Laboratory for Genomic Medicine, Farmington, CT, USA; Department of Genetics and Genome Sciences, University of Connecticut Health Center, Farmington, CT, USA
| | - Todd B Sheridan
- The Jackson Laboratory for Genomic Medicine, Farmington, CT, USA; Department of Pathology, Hartford Hospital, Hartford, CT, USA
| | | | - Jeffrey H Chuang
- The Jackson Laboratory for Genomic Medicine, Farmington, CT, USA; Department of Genetics and Genome Sciences, University of Connecticut Health Center, Farmington, CT, USA.
| |
Collapse
|
28
|
Schacherer DP, Herrmann MD, Clunie DA, Höfener H, Clifford W, Longabaugh WJR, Pieper S, Kikinis R, Fedorov A, Homeyer A. The NCI Imaging Data Commons as a platform for reproducible research in computational pathology. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 242:107839. [PMID: 37832430 PMCID: PMC10841477 DOI: 10.1016/j.cmpb.2023.107839] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/15/2023] [Revised: 09/20/2023] [Accepted: 10/01/2023] [Indexed: 10/15/2023]
Abstract
BACKGROUND AND OBJECTIVES Reproducibility is a major challenge in developing machine learning (ML)-based solutions in computational pathology (CompPath). The NCI Imaging Data Commons (IDC) provides >120 cancer image collections according to the FAIR principles and is designed to be used with cloud ML services. Here, we explore its potential to facilitate reproducibility in CompPath research. METHODS Using the IDC, we implemented two experiments in which a representative ML-based method for classifying lung tumor tissue was trained and/or evaluated on different datasets. To assess reproducibility, the experiments were run multiple times with separate but identically configured instances of common ML services. RESULTS The results of different runs of the same experiment were reproducible to a large extent. However, we observed occasional, small variations in AUC values, indicating a practical limit to reproducibility. CONCLUSIONS We conclude that the IDC facilitates approaching the reproducibility limit of CompPath research (i) by enabling researchers to reuse exactly the same datasets and (ii) by integrating with cloud ML services so that experiments can be run in identically configured computing environments.
Collapse
Affiliation(s)
- Daniela P Schacherer
- Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany
| | - Markus D Herrmann
- Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | | | - Henning Höfener
- Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany
| | | | | | | | - Ron Kikinis
- Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA
| | - Andrey Fedorov
- Department of Radiology, Brigham and Women's Hospital and Harvard Medical School, Boston, MA, USA
| | - André Homeyer
- Fraunhofer Institute for Digital Medicine MEVIS, Max-von-Laue-Straße 2, 28359 Bremen, Germany.
| |
Collapse
|
29
|
Tavolara TE, Su Z, Gurcan MN, Niazi MKK. One label is all you need: Interpretable AI-enhanced histopathology for oncology. Semin Cancer Biol 2023; 97:70-85. [PMID: 37832751 DOI: 10.1016/j.semcancer.2023.09.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2022] [Revised: 09/06/2023] [Accepted: 09/25/2023] [Indexed: 10/15/2023]
Abstract
Artificial Intelligence (AI)-enhanced histopathology presents unprecedented opportunities to benefit oncology through interpretable methods that require only one overall label per hematoxylin and eosin (H&E) slide with no tissue-level annotations. We present a structured review of these methods organized by their degree of verifiability and by commonly recurring application areas in oncological characterization. First, we discuss morphological markers (tumor presence/absence, metastases, subtypes, grades) in which AI-identified regions of interest (ROIs) within whole slide images (WSIs) verifiably overlap with pathologist-identified ROIs. Second, we discuss molecular markers (gene expression, molecular subtyping) that are not verified via H&E but rather based on overlap with positive regions on adjacent tissue. Third, we discuss genetic markers (mutations, mutational burden, microsatellite instability, chromosomal instability) that current technologies cannot verify if AI methods spatially resolve specific genetic alterations. Fourth, we discuss the direct prediction of survival to which AI-identified histopathological features quantitatively correlate but are nonetheless not mechanistically verifiable. Finally, we discuss in detail several opportunities and challenges for these one-label-per-slide methods within oncology. Opportunities include reducing the cost of research and clinical care, reducing the workload of clinicians, personalized medicine, and unlocking the full potential of histopathology through new imaging-based biomarkers. Current challenges include explainability and interpretability, validation via adjacent tissue sections, reproducibility, data availability, computational needs, data requirements, domain adaptability, external validation, dataset imbalances, and finally commercialization and clinical potential. Ultimately, the relative ease and minimum upfront cost with which relevant data can be collected in addition to the plethora of available AI methods for outcome-driven analysis will surmount these current limitations and achieve the innumerable opportunities associated with AI-driven histopathology for the benefit of oncology.
Collapse
Affiliation(s)
- Thomas E Tavolara
- Center for Artificial Intelligence Research, Wake Forest University School of Medicine, Winston-Salem, NC, USA
| | - Ziyu Su
- Center for Artificial Intelligence Research, Wake Forest University School of Medicine, Winston-Salem, NC, USA
| | - Metin N Gurcan
- Center for Artificial Intelligence Research, Wake Forest University School of Medicine, Winston-Salem, NC, USA
| | - M Khalid Khan Niazi
- Center for Artificial Intelligence Research, Wake Forest University School of Medicine, Winston-Salem, NC, USA.
| |
Collapse
|
30
|
Lei J, Liu Q. Difference of Convex Functions Programming With Machine-Learning Prior for the Imaging Problem in Electrical Capacitance Tomography. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:7535-7547. [PMID: 35604983 DOI: 10.1109/tcyb.2022.3173336] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The electrical capacitance tomography technology has potential benefits for the process industry by providing visualization of material distributions. One of the main technical gaps and impediments that must be overcome is the low-quality tomogram. To address this problem, this study introduces the data-guided prior and combines it with the electrical measurement mechanism and the sparsity prior to produce a new difference of convex functions programming problem that turns the image reconstruction problem into an optimization problem. The data-guided prior is learned from a provided dataset and captures the details of imaging targets since it is a specific image. A new numerical scheme that allows a complex optimization problem to be split into a few less difficult subproblems is developed to solve the challenging difference of convex functions programming problem. A new dimensionality reduction method is developed and combined with the relevance vector machine to generate a new learning engine for the forecast of the data-guided prior. The new imaging method fuses multisource information and unifies data-guided and measurement physics modeling paradigms. Performance evaluation results have validated that the new method successfully works on a series of test tasks with higher reconstruction quality and lower noise sensitivity than the popular imaging methods.
Collapse
|
31
|
Lin YJ, Chen CC, Lee CH, Yeh CY, Jeng YM. Two-tiered deep-learning-based model for histologic diagnosis of Helicobacter gastritis. Histopathology 2023; 83:771-781. [PMID: 37522271 DOI: 10.1111/his.15018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Revised: 07/11/2023] [Accepted: 07/15/2023] [Indexed: 08/01/2023]
Abstract
AIMS Helicobacter pylori (HP) infection is the most common cause of chronic gastritis worldwide. Due to the small size of HP and limited resolution, diagnosing HP infections is more difficult when using digital slides. METHODS AND RESULTS We developed a two-tier deep-learning-based model for diagnosing HP gastritis. A whole-slide model was trained on 885 whole-slide images (WSIs) with only slide-level labels (positive or negative slides). An auxiliary model was trained on 824 areas with HP in nine positive WSIs and 446 negative WSIs for localizing HP. The whole-slide model performed well, with an area under the receiver operating characteristic curve (AUC) of 0.9739 (95% confidence interval [CI], 0.9545-0.9932). The calculated sensitivity and specificity were 93.3% and 90.1%, respectively, whereas those of pathologists were 93.3% and 84.2%, respectively. Using the auxiliary model, the highlighted areas of the localization maps had an average precision of 0.5796. CONCLUSIONS HP gastritis can be diagnosed on haematoxylin-and-eosin-stained WSIs with human-level accuracy using a deep-learning-based model trained on slide-level labels and an auxiliary model for localizing HP and confirming the diagnosis. This two-tiered model can shorten the diagnostic process and reduce the need for special staining.
Collapse
Affiliation(s)
- Yi-Jyun Lin
- Department of Pathology, National Taiwan University Hospital, Taipei, Taiwan
| | | | - Chia-Hsiang Lee
- Department of Pathology, National Taiwan University Hospital, Taipei, Taiwan
| | | | - Yung-Ming Jeng
- Department of Pathology, National Taiwan University Hospital, Taipei, Taiwan
- Graduate Institute of Pathology, College of Medicine, National Taiwan University, Taipei, Taiwan
| |
Collapse
|
32
|
Pang S, Du A, Orgun MA, Wang Y, Sheng QZ, Wang S, Huang X, Yu Z. Beyond CNNs: Exploiting Further Inherent Symmetries in Medical Image Segmentation. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:6776-6787. [PMID: 36044511 DOI: 10.1109/tcyb.2022.3195447] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Automatic tumor or lesion segmentation is a crucial step in medical image analysis for computer-aided diagnosis. Although the existing methods based on convolutional neural networks (CNNs) have achieved the state-of-the-art performance, many challenges still remain in medical tumor segmentation. This is because, although the human visual system can detect symmetries in 2-D images effectively, regular CNNs can only exploit translation invariance, overlooking further inherent symmetries existing in medical images, such as rotations and reflections. To solve this problem, we propose a novel group equivariant segmentation framework by encoding those inherent symmetries for learning more precise representations. First, kernel-based equivariant operations are devised on each orientation, which allows it to effectively address the gaps of learning symmetries in existing approaches. Then, to keep segmentation networks globally equivariant, we design distinctive group layers with layer-wise symmetry constraints. Finally, based on our novel framework, extensive experiments conducted on real-world clinical data demonstrate that a group equivariant Res-UNet (called GER-UNet) outperforms its regular CNN-based counterpart and the state-of-the-art segmentation methods in the tasks of hepatic tumor segmentation, COVID-19 lung infection segmentation, and retinal vessel detection. More importantly, the newly built GER-UNet also shows potential in reducing the sample complexity and the redundancy of filters, upgrading current segmentation CNNs, and delineating organs on other medical imaging modalities.
Collapse
|
33
|
Aftab R, Qiang Y, Zhao J, Urrehman Z, Zhao Z. Graph Neural Network for representation learning of lung cancer. BMC Cancer 2023; 23:1037. [PMID: 37884929 PMCID: PMC10601264 DOI: 10.1186/s12885-023-11516-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Accepted: 10/11/2023] [Indexed: 10/28/2023] Open
Abstract
The emergence of image-based systems to improve diagnostic pathology precision, involving the intent to label sets or bags of instances, greatly hinges on Multiple Instance Learning for Whole Slide Images(WSIs). Contemporary works have shown excellent performance for a neural network in MIL settings. Here, we examine a graph-based model to facilitate end-to-end learning and sample suitable patches using a tile-based approach. We propose MIL-GNN to employ a graph-based Variational Auto-encoder with a Gaussian mixture model to discover relations between sample patches for the purposes to aggregate patch details into an individual vector representation. Using the classical MIL dataset MUSK and distinguishing two lung cancer sub-types, lung cancer called adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC), we exhibit the efficacy of our technique. We achieved a 97.42% accuracy on the MUSK dataset and a 94.3% AUC on the classification of lung cancer sub-types utilizing features.
Collapse
Affiliation(s)
- Rukhma Aftab
- College of Information and Computer, Taiyuan University of Technology, No. 79 Yingze West Street, Taiyuan, 030024 China
| | - Yan Qiang
- College of Information and Computer, Taiyuan University of Technology, No. 79 Yingze West Street, Taiyuan, 030024 China
| | - Juanjuan Zhao
- College of Information and Computer, Taiyuan University of Technology, No. 79 Yingze West Street, Taiyuan, 030024 China
| | - Zia Urrehman
- College of Information and Computer, Taiyuan University of Technology, No. 79 Yingze West Street, Taiyuan, 030024 China
| | - Zijuan Zhao
- College of Information and Computer, Taiyuan University of Technology, No. 79 Yingze West Street, Taiyuan, 030024 China
| |
Collapse
|
34
|
Tian C, Zhu H, Meng X, Ma Z, Yuan S, Li W. Research for accurate auxiliary diagnosis of lung cancer based on intracellular fluorescent fingerprint information. JOURNAL OF BIOPHOTONICS 2023; 16:e202300174. [PMID: 37350031 DOI: 10.1002/jbio.202300174] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 06/09/2023] [Accepted: 06/20/2023] [Indexed: 06/24/2023]
Abstract
The distinctions in pathological types and genetic subtypes of lung cancer have a direct impact on the choice of treatment choices and clinical prognosis in clinical practice. This study used pathological histological sections of surgically removed or biopsied tumor tissue from 36 patients. Based on a small sample size, millions of spectral data points were extracted to investigate the feasibility of employing intracellular fluorescent fingerprint information to diagnose the pathological types and mutational status of lung cancer. The intracellular fluorescent fingerprint information revealed the EGFR gene mutation characteristics in lung cancer, and the area under the curve (AUC) value for the optimal model was 0.98. For the classification of lung cancer pathological types, the macro average AUC value for the ensemble-learning model was 0.97. Our research contributes new idea for pathological diagnosis of lung cancer and offers a quick, easy, and accurate auxiliary diagnostic approach.
Collapse
Affiliation(s)
- Chongxuan Tian
- Department of Biomedical Engineering Institute, School of Control Science and Engineering, Shandong University, Jinan, Shandong, China
| | - He Zhu
- Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Shandong Cancer Hospital Affiliated to Shandong First Medical University, Jinan, Shandong, China
| | - Xiangwei Meng
- Department of Biomedical Engineering Institute, School of Control Science and Engineering, Shandong University, Jinan, Shandong, China
| | - Zhixiang Ma
- Department of Biomedical Engineering Institute, School of Control Science and Engineering, Shandong University, Jinan, Shandong, China
| | - Shuanghu Yuan
- Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Shandong First Medical University and Shandong Academy of Medical Sciences, Shandong Cancer Hospital Affiliated to Shandong First Medical University, Jinan, Shandong, China
- Department of Radiation Oncology, Shandong Cancer Hospital Affiliated to Shandong University, Jinan, Shandong, China
- Department of Radiation Oncology, The Affiliated Cancer Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Wei Li
- Department of Biomedical Engineering Institute, School of Control Science and Engineering, Shandong University, Jinan, Shandong, China
| |
Collapse
|
35
|
Hoque MZ, Keskinarkaus A, Nyberg P, Xu H, Seppänen T. Invasion depth estimation of carcinoma cells using adaptive stain normalization to improve epidermis segmentation accuracy. Comput Med Imaging Graph 2023; 108:102276. [PMID: 37611486 DOI: 10.1016/j.compmedimag.2023.102276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 07/25/2023] [Accepted: 07/26/2023] [Indexed: 08/25/2023]
Abstract
Submucosal invasion depth is a significant prognostic factor when assessing lymph node metastasis and cancer itself to plan proper treatment for the patient. Conventionally, oncologists measure the invasion depth by hand which is a laborious, subjective, and time-consuming process. The manual pathological examination by measuring accurate carcinoma cell invasion with considerable inter-observer and intra-observer variations is still challenging. The increasing use of medical imaging and artificial intelligence reveals a significant role in clinical medicine and pathology. In this paper, we propose an approach to study invasive behavior and measure the invasion depth of carcinoma from stained histopathology images. Specifically, our model includes adaptive stain normalization, color decomposition, and morphological reconstruction with adaptive thresholding to separate the epithelium with blue ratio image. Our method splits the image into multiple non-overlapping meaningful segments and successfully finds the homogeneous segments to measure accurate invasion depth. The invasion depths are measured from the inner epithelium edge to outermost pixels of the deepest part of particles in image. We conduct our experiments on skin melanoma tissue samples as well as on organotypic invasion model utilizing myoma tissue and oral squamous cell carcinoma. The performance is experimentally compared to three closely related reference methods and our method provides a superior result in measuring invasion depth. This computational technique will be beneficial for the segmentation of epithelium and other particles for the development of novel computer-aided diagnostic tools in biobank applications.
Collapse
Affiliation(s)
- Md Ziaul Hoque
- Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland; Division of Nephrology and Intelligent Critical Care, Department of Medicine, University of Florida, Gainesville, USA.
| | - Anja Keskinarkaus
- Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
| | - Pia Nyberg
- Biobank Borealis of Northern Finland, Oulu University Hospital, Finland; Translational Medicine Research Unit, Medical Research Center Oulu, Faculty of Medicine, University of Oulu, Finland
| | - Hongming Xu
- Department of Electrical and Computer Engineering, University of Alberta, Canada; School of Biomedical Engineering, Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China
| | - Tapio Seppänen
- Center for Machine Vision and Signal Analysis, Faculty of Information Technology and Electrical Engineering, University of Oulu, Finland
| |
Collapse
|
36
|
Zheng T, Chen W, Li S, Quan H, Zou M, Zheng S, Zhao Y, Gao X, Cui X. Learning how to detect: A deep reinforcement learning method for whole-slide melanoma histopathology images. Comput Med Imaging Graph 2023; 108:102275. [PMID: 37567046 DOI: 10.1016/j.compmedimag.2023.102275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2023] [Revised: 07/18/2023] [Accepted: 07/22/2023] [Indexed: 08/13/2023]
Abstract
Cutaneous melanoma represents one of the most life-threatening malignancies. Histopathological image analysis serves as a vital tool for early melanoma detection. Deep neural network (DNN) models are frequently employed to aid pathologists in enhancing the efficiency and accuracy of diagnoses. However, due to the paucity of well-annotated, high-resolution, whole-slide histopathology image (WSI) datasets, WSIs are typically fragmented into numerous patches during the model training and testing stages. This process disregards the inherent interconnectedness among patches, potentially impeding the models' performance. Additionally, the presence of excess, non-contributing patches extends processing times and introduces substantial computational burdens. To mitigate these issues, we draw inspiration from the clinical decision-making processes of dermatopathologists to propose an innovative, weakly supervised deep reinforcement learning framework, titled Fast medical decision-making in melanoma histopathology images (FastMDP-RL). This framework expedites model inference by reducing the number of irrelevant patches identified within WSIs. FastMDP-RL integrates two DNN-based agents: the search agent (SeAgent) and the decision agent (DeAgent). The SeAgent initiates actions, steered by the image features observed in the current viewing field at various magnifications. Simultaneously, the DeAgent provides labeling probabilities for each patch. We utilize multi-instance learning (MIL) to construct a teacher-guided model (MILTG), serving a dual purpose: rewarding the SeAgent and guiding the DeAgent. Our evaluations were conducted using two melanoma datasets: the publicly accessible TCIA-CM dataset and the proprietary MELSC dataset. Our experimental findings affirm FastMDP-RL's ability to expedite inference and accurately predict WSIs, even in the absence of pixel-level annotations. Moreover, our research investigates the WSI-based interactive environment, encompassing the design of agents, state and reward functions, and feature extractors suitable for melanoma tissue images. This investigation offers valuable insights and references for researchers engaged in related studies. The code is available at: https://github.com/titizheng/FastMDP-RL.
Collapse
Affiliation(s)
- Tingting Zheng
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Weixing Chen
- Shenzhen College of Advanced Technology, University of the Chinese Academy of Sciences, Beijing, China
| | - Shuqin Li
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Hao Quan
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Mingchen Zou
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Song Zheng
- National and Local Joint Engineering Research Center of Immunodermatological Theranostics, Department of Dermatology, The First Hospital of China Medical University, Shenyang, China
| | - Yue Zhao
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China; National and Local Joint Engineering Research Center of Immunodermatological Theranostics, Department of Dermatology, The First Hospital of China Medical University, Shenyang, China
| | - Xinghua Gao
- National and Local Joint Engineering Research Center of Immunodermatological Theranostics, Department of Dermatology, The First Hospital of China Medical University, Shenyang, China
| | - Xiaoyu Cui
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China.
| |
Collapse
|
37
|
Cooper M, Ji Z, Krishnan RG. Machine learning in computational histopathology: Challenges and opportunities. Genes Chromosomes Cancer 2023; 62:540-556. [PMID: 37314068 DOI: 10.1002/gcc.23177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 05/18/2023] [Accepted: 05/20/2023] [Indexed: 06/15/2023] Open
Abstract
Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.
Collapse
Affiliation(s)
- Michael Cooper
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- University Health Network, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Zongliang Ji
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | - Rahul G Krishnan
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
38
|
Shao J, Feng J, Li J, Liang S, Li W, Wang C. Novel tools for early diagnosis and precision treatment based on artificial intelligence. CHINESE MEDICAL JOURNAL PULMONARY AND CRITICAL CARE MEDICINE 2023; 1:148-160. [PMID: 39171128 PMCID: PMC11332840 DOI: 10.1016/j.pccm.2023.05.001] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Indexed: 08/23/2024]
Abstract
Lung cancer has the highest mortality rate among all cancers in the world. Hence, early diagnosis and personalized treatment plans are crucial to improving its 5-year survival rate. Chest computed tomography (CT) serves as an essential tool for lung cancer screening, and pathology images are the gold standard for lung cancer diagnosis. However, medical image evaluation relies on manual labor and suffers from missed diagnosis or misdiagnosis, and physician heterogeneity. The rapid development of artificial intelligence (AI) has brought a whole novel opportunity for medical task processing, demonstrating the potential for clinical application in lung cancer diagnosis and treatment. AI technologies, including machine learning and deep learning, have been deployed extensively for lung nodule detection, benign and malignant classification, and subtype identification based on CT images. Furthermore, AI plays a role in the non-invasive prediction of genetic mutations and molecular status to provide the optimal treatment regimen, and applies to the assessment of therapeutic efficacy and prognosis of lung cancer patients, enabling precision medicine to become a reality. Meanwhile, histology-based AI models assist pathologists in typing, molecular characterization, and prognosis prediction to enhance the efficiency of diagnosis and treatment. However, the leap to extensive clinical application still faces various challenges, such as data sharing, standardized label acquisition, clinical application regulation, and multimodal integration. Nevertheless, AI holds promising potential in the field of lung cancer to improve cancer care.
Collapse
Affiliation(s)
- Jun Shao
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Jiaming Feng
- West China School of Medicine, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Jingwei Li
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Shufan Liang
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Weimin Li
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Chengdi Wang
- Department of Pulmonary and Critical Care Medicine, Med-X Center for Manufacturing, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| |
Collapse
|
39
|
Davri A, Birbas E, Kanavos T, Ntritsos G, Giannakeas N, Tzallas AT, Batistatou A. Deep Learning for Lung Cancer Diagnosis, Prognosis and Prediction Using Histological and Cytological Images: A Systematic Review. Cancers (Basel) 2023; 15:3981. [PMID: 37568797 PMCID: PMC10417369 DOI: 10.3390/cancers15153981] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 07/27/2023] [Accepted: 08/03/2023] [Indexed: 08/13/2023] Open
Abstract
Lung cancer is one of the deadliest cancers worldwide, with a high incidence rate, especially in tobacco smokers. Lung cancer accurate diagnosis is based on distinct histological patterns combined with molecular data for personalized treatment. Precise lung cancer classification from a single H&E slide can be challenging for a pathologist, requiring most of the time additional histochemical and special immunohistochemical stains for the final pathology report. According to WHO, small biopsy and cytology specimens are the available materials for about 70% of lung cancer patients with advanced-stage unresectable disease. Thus, the limited available diagnostic material necessitates its optimal management and processing for the completion of diagnosis and predictive testing according to the published guidelines. During the new era of Digital Pathology, Deep Learning offers the potential for lung cancer interpretation to assist pathologists' routine practice. Herein, we systematically review the current Artificial Intelligence-based approaches using histological and cytological images of lung cancer. Most of the published literature centered on the distinction between lung adenocarcinoma, lung squamous cell carcinoma, and small cell lung carcinoma, reflecting the realistic pathologist's routine. Furthermore, several studies developed algorithms for lung adenocarcinoma predominant architectural pattern determination, prognosis prediction, mutational status characterization, and PD-L1 expression status estimation.
Collapse
Affiliation(s)
- Athena Davri
- Department of Pathology, Faculty of Medicine, School of Health Sciences, University of Ioannina, 45500 Ioannina, Greece;
| | - Effrosyni Birbas
- Faculty of Medicine, School of Health Sciences, University of Ioannina, 45110 Ioannina, Greece; (E.B.); (T.K.)
| | - Theofilos Kanavos
- Faculty of Medicine, School of Health Sciences, University of Ioannina, 45110 Ioannina, Greece; (E.B.); (T.K.)
| | - Georgios Ntritsos
- Department of Hygiene and Epidemiology, Faculty of Medicine, School of Health Sciences, University of Ioannina, 45110 Ioannina, Greece;
- Department of Informatics and Telecommunications, University of Ioannina, 47100 Arta, Greece;
| | - Nikolaos Giannakeas
- Department of Informatics and Telecommunications, University of Ioannina, 47100 Arta, Greece;
| | - Alexandros T. Tzallas
- Department of Informatics and Telecommunications, University of Ioannina, 47100 Arta, Greece;
| | - Anna Batistatou
- Department of Pathology, Faculty of Medicine, School of Health Sciences, University of Ioannina, 45500 Ioannina, Greece;
| |
Collapse
|
40
|
Mukashyaka P, Sheridan TB, Foroughi Pour A, Chuang JH. SAMPLER: Empirical distribution representations for rapid analysis of whole slide tissue images. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.01.551468. [PMID: 37577691 PMCID: PMC10418159 DOI: 10.1101/2023.08.01.551468] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Deep learning has revolutionized digital pathology, allowing for automatic analysis of hematoxylin and eosin (H&E) stained whole slide images (WSIs) for diverse tasks. In such analyses, WSIs are typically broken into smaller images called tiles, and a neural network backbone encodes each tile in a feature space. Many recent works have applied attention based deep learning models to aggregate tile-level features into a slide-level representation, which is then used for slide-level prediction tasks. However, training attention models is computationally intensive, necessitating hyperparameter optimization and specialized training procedures. Here, we propose SAMPLER, a fully statistical approach to generate efficient and informative WSI representations by encoding the empirical cumulative distribution functions (CDFs) of multiscale tile features. We demonstrate that SAMPLER-based classifiers are as accurate or better than state-of-the-art fully deep learning attention models for classification tasks including distinction of: subtypes of breast carcinoma (BRCA: AUC=0.911 ± 0.029); subtypes of non-small cell lung carcinoma (NSCLC: AUC=0.940±0.018); and subtypes of renal cell carcinoma (RCC: AUC=0.987±0.006). A major advantage of the SAMPLER representation is that predictive models are >100X faster compared to attention models. Histopathological review confirms that SAMPLER-identified high attention tiles contain tumor morphological features specific to the tumor type, while low attention tiles contain fibrous stroma, blood, or tissue folding artifacts. We further apply SAMPLER concepts to improve the design of attention-based neural networks, yielding a context aware multi-head attention model with increased accuracy for subtype classification within BRCA and RCC (BRCA: AUC=0.921±0.027, and RCC: AUC=0.988±0.010). Finally, we provide theoretical results identifying sufficient conditions for which SAMPLER is optimal. SAMPLER is a fast and effective approach for analyzing WSIs, with greatly improved scalability over attention methods to benefit digital pathology analysis.
Collapse
Affiliation(s)
- Patience Mukashyaka
- The Jackson Laboratory for Genomic Medicine, Farmington, CT
- University of Connecticut Health Center, Department of Genetics and Genome Sciences, Farmington, CT
| | - Todd B Sheridan
- The Jackson Laboratory for Genomic Medicine, Farmington, CT
- Department of Pathology, Hartford hospital, Hartford, CT
| | | | - Jeffrey H Chuang
- The Jackson Laboratory for Genomic Medicine, Farmington, CT
- University of Connecticut Health Center, Department of Genetics and Genome Sciences, Farmington, CT
| |
Collapse
|
41
|
Predicting EGFR gene mutation status in lung adenocarcinoma based on multifeature fusion. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104786] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/09/2023]
|
42
|
Fu Y, Zhou F, Shi X, Wang L, Li Y, Wu J, Huang H. Classification of adenoid cystic carcinoma in whole slide images by using deep learning. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/11/2023]
|
43
|
Hu W, Li X, Li C, Li R, Jiang T, Sun H, Huang X, Grzegorzek M, Li X. A state-of-the-art survey of artificial neural networks for Whole-slide Image analysis: From popular Convolutional Neural Networks to potential visual transformers. Comput Biol Med 2023; 161:107034. [PMID: 37230019 DOI: 10.1016/j.compbiomed.2023.107034] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 04/13/2023] [Accepted: 05/10/2023] [Indexed: 05/27/2023]
Abstract
In recent years, with the advancement of computer-aided diagnosis (CAD) technology and whole slide image (WSI), histopathological WSI has gradually played a crucial aspect in the diagnosis and analysis of diseases. To increase the objectivity and accuracy of pathologists' work, artificial neural network (ANN) methods have been generally needed in the segmentation, classification, and detection of histopathological WSI. However, the existing review papers only focus on equipment hardware, development status and trends, and do not summarize the art neural network used for full-slide image analysis in detail. In this paper, WSI analysis methods based on ANN are reviewed. Firstly, the development status of WSI and ANN methods is introduced. Secondly, we summarize the common ANN methods. Next, we discuss publicly available WSI datasets and evaluation metrics. These ANN architectures for WSI processing are divided into classical neural networks and deep neural networks (DNNs) and then analyzed. Finally, the application prospect of the analytical method in this field is discussed. The important potential method is Visual Transformers.
Collapse
Affiliation(s)
- Weiming Hu
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Xintong Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Chen Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China.
| | - Rui Li
- Microscopic Image and Medical Image Analysis Group, College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Tao Jiang
- School of Intelligent Medicine, Chengdu University of Traditional Chinese Medicine, Chengdu, China; International Joint Institute of Robotics and Intelligent Systems, Chengdu University of Information Technology, Chengdu, China
| | - Hongzan Sun
- Shengjing Hospital of China Medical University, Shenyang, China
| | - Xinyu Huang
- Institute for Medical Informatics, University of Luebeck, Luebeck, Germany
| | - Marcin Grzegorzek
- Institute for Medical Informatics, University of Luebeck, Luebeck, Germany; Department of Knowledge Engineering, University of Economics in Katowice, Katowice, Poland
| | - Xiaoyan Li
- Cancer Hospital of China Medical University, Shenyang, China.
| |
Collapse
|
44
|
Liu M, Li L, Wang H, Guo X, Liu Y, Li Y, Song K, Shao Y, Wu F, Zhang J, Sun N, Zhang T, Luan L. A multilayer perceptron-based model applied to histopathology image classification of lung adenocarcinoma subtypes. Front Oncol 2023; 13:1172234. [PMID: 37274249 PMCID: PMC10233124 DOI: 10.3389/fonc.2023.1172234] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Accepted: 05/05/2023] [Indexed: 06/06/2023] Open
Abstract
Objective Lung cancer is one of the most common malignant tumors in humans. Adenocarcinoma of the lung is another of the most common types of lung cancer. In clinical medicine, physicians rely on the information provided by pathology tests as an important reference for the fifinal diagnosis of many diseases. Thus, pathological diagnosis is known as the gold standard for disease diagnosis. However, the complexity of the information contained in pathology images and the increase in the number of patients far exceeds the number of pathologists, especially in the treatment of lung cancer in less-developed countries. Methods This paper proposes a multilayer perceptron model for lung cancer histopathology image detection, which enables the automatic detection of the degree of lung adenocarcinoma infifiltration. For the large amount of local information present in lung cancer histopathology images, MLP IN MLP (MIM) uses a dual data stream input method to achieve a modeling approach that combines global and local information to improve the classifification performance of the model. In our experiments, we collected 780 lung cancer histopathological images and prepared a lung histopathology image dataset to verify the effectiveness of MIM. Results The MIM achieves a diagnostic accuracy of 95.31% and has a precision, sensitivity, specificity and F1-score of 95.31%, 93.09%, 93.10%, 96.43% and 93.10% respectively, outperforming the diagnostic results of the common network model. In addition, a number of series of extension experiments demonstrated the scalability and stability of the MIM. Conclusions In summary, MIM has high classifification performance and substantial potential in lung cancer detection tasks.
Collapse
Affiliation(s)
- Mingyang Liu
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Liyuan Li
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Haoran Wang
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Xinyu Guo
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Yunpeng Liu
- Department of Thoracic Surgery, The First Hospital of Jilin University, Changchun, China
| | - Yuguang Li
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Kaiwen Song
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Yanbin Shao
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Fei Wu
- Department of Pathology, Central Hospital Affiliated to Shenyang Medical College, Shenyang, China
| | - Junjie Zhang
- Department of Pathology, Central Hospital Affiliated to Shenyang Medical College, Shenyang, China
| | - Nao Sun
- Center for Reproductive Medicine and Center for Prenatal Diagnosis, The First Hospital of Jilin University, Changchun, China
| | - Tianyu Zhang
- Key Laboratory of Geophysical Exploration Equipment, Ministry of Education, College of Instrumentation and Electrical Engineering, Jilin University, Changchun, China
| | - Lan Luan
- Department of Pathology, Central Hospital Affiliated to Shenyang Medical College, Shenyang, China
| |
Collapse
|
45
|
Cao L, Wang J, Zhang Y, Rong Z, Wang M, Wang L, Ji J, Qian Y, Zhang L, Wu H, Song J, Liu Z, Wang W, Li S, Wang P, Xu Z, Zhang J, Zhao L, Wang H, Sun M, Huang X, Yin R, Lu Y, Liu Z, Deng K, Wang G, Qiu M, Li K, Wang J, Hou Y. E2EFP-MIL: End-to-end and high-generalizability weakly supervised deep convolutional network for lung cancer classification from whole slide image. Med Image Anal 2023; 88:102837. [PMID: 37216736 DOI: 10.1016/j.media.2023.102837] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2022] [Revised: 03/11/2023] [Accepted: 05/05/2023] [Indexed: 05/24/2023]
Abstract
Efficient and accurate distinction of histopathological subtype of lung cancer is quite critical for the individualized treatment. So far, artificial intelligence techniques have been developed, whose performance yet remained debatable on more heterogenous data, hindering their clinical deployment. Here, we propose an end-to-end, well-generalized and data-efficient weakly supervised deep learning-based method. The method, end-to-end feature pyramid deep multi-instance learning model (E2EFP-MIL), contains an iterative sampling module, a trainable feature pyramid module and a robust feature aggregation module. E2EFP-MIL uses end-to-end learning to extract generalized morphological features automatically and identify discriminative histomorphological patterns. This method is trained with 1007 whole slide images (WSIs) of lung cancer from TCGA, with AUCs of 0.95-0.97 in test sets. We validated E2EFP-MIL in 5 real-world external heterogenous cohorts including nearly 1600 WSIs from both United States and China with AUCs of 0.94-0.97, and found that 100-200 training images are enough to achieve an AUC of >0.9. E2EFP-MIL overperforms multiple state-of-the-art MIL-based methods with high accuracy and low hardware requirements. Excellent and robust results prove generalizability and effectiveness of E2EFP-MIL in clinical practice. Our code is available at https://github.com/raycaohmu/E2EFP-MIL.
Collapse
Affiliation(s)
- Lei Cao
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Jie Wang
- Department of Tumor Biobank, Jiangsu Cancer Hospital, Jiangsu Institute of Cancer Research, Nanjing 210009, China
| | - Yuanyuan Zhang
- Department of Pathology, Peking University People's Hospital, Beijing 100044, China
| | - Zhiwei Rong
- Department of Biostatistics, School of Public Health, Peking University, Beijing 100191, China
| | - Meng Wang
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Liuying Wang
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Jianxin Ji
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Youhui Qian
- Department of Thoracic Surgery, The First Affiliated Hospital of Shenzhen University, Shenzhen 518000, China
| | - Liuchao Zhang
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Hao Wu
- Department of Thoracic Surgery, The First Affiliated Hospital of Shenzhen University, Shenzhen 518000, China
| | - Jiali Song
- Department of Biostatistics, School of Public Health, Peking University, Beijing 100191, China
| | - Zheng Liu
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China
| | - Wenjie Wang
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Shuang Li
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Peiyu Wang
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China
| | - Zhenyi Xu
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Jingyuan Zhang
- Department of Pathology, Jiangsu Cancer Hospital, The Affiliated Cancer Hospital of Nanjing Medical University, Nanjing 210009, China
| | - Liang Zhao
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China
| | - Hang Wang
- Department of Tumor Biobank, Jiangsu Cancer Hospital, Jiangsu Institute of Cancer Research, Nanjing 210009, China
| | - Mengting Sun
- Department of Tumor Biobank, Jiangsu Cancer Hospital, Jiangsu Institute of Cancer Research, Nanjing 210009, China
| | - Xing Huang
- Department of Pathology, Jiangsu Cancer Hospital, The Affiliated Cancer Hospital of Nanjing Medical University, Nanjing 210009, China
| | - Rong Yin
- Department of Thoracic Surgery, Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Jiangsu Cancer Hospital, Nanjing 210009, China
| | - Yuhong Lu
- Department of Biostatistics, School of Public Health, Peking University, Beijing 100191, China
| | - Ziqian Liu
- Biostatistics and SAS Programming, Clinical Sciences, Johnson & Johnson Vision Care, Inc., FL 32256, US
| | - Kui Deng
- Division of Epidemiology, Department of Medicine, Vanderbilt Epidemiology Center, Vanderbilt University Medical Center, Nashville, TN 37232, US
| | - Gongwei Wang
- Department of Pathology, Peking University People's Hospital, Beijing 100044, China
| | - Mantang Qiu
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China.
| | - Kang Li
- Department of Biostatistics, School of Public Health, Harbin Medical University, Harbin 150081, China.
| | - Jun Wang
- Department of Thoracic Surgery, Peking University People's Hospital, Beijing 100044, China.
| | - Yan Hou
- Department of Biostatistics, School of Public Health, Peking University, Beijing 100191, China.
| |
Collapse
|
46
|
Yan R, Shen Y, Zhang X, Xu P, Wang J, Li J, Ren F, Ye D, Zhou SK. Histopathological bladder cancer gene mutation prediction with hierarchical deep multiple-instance learning. Med Image Anal 2023; 87:102824. [PMID: 37126973 DOI: 10.1016/j.media.2023.102824] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2022] [Revised: 03/13/2023] [Accepted: 04/17/2023] [Indexed: 05/03/2023]
Abstract
Gene mutation detection is usually carried out by molecular biological methods, which is expensive and has a long-time cycle. In contrast, pathological images are ubiquitous. If clinically significant gene mutations can be predicted only through pathological images, it will greatly promote the widespread use of gene mutation detection in clinical practice. However, current gene mutation prediction methods based on pathological images are ineffective because of the inability to identify mutated regions in gigapixel Whole Slide Image (WSI). To address this challenge, hereby we propose a carefully designed framework for WSI-based gene mutation prediction, which consists of three parts. (i) The first part of cancerous area segmentation, based on supervised learning, quickly filters out a large number of non-mutated regions; (ii) the second part of cancerous patch clustering, based on the representations derived from contrastive learning, ensures the comprehensiveness of patch selection; and (iii) the third part of mutation classification, based on the proposed hierarchical deep multi-instance learning method (HDMIL), ensures that sufficient patches are considered and inaccurate selections are ignored. In addition, benefiting from a two-stage attention mechanism in HDMIL, the patches that are highly correlated with gene mutations can be identified. This interpretability can help a pathologist to analyze the correlation between gene mutation and histopathological morphology. Experimental results demonstrate that the proposed gene mutation prediction framework significantly outperforms the state-of-the-art methods. In the TCGA bladder cancer dataset, five clinically relevant gene mutations are well predicted.
Collapse
Affiliation(s)
- Rui Yan
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China; University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Yijun Shen
- Department of Urology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China; Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Xueyuan Zhang
- Zhijian Life Technology Co., Ltd., Beijing, 100036, China
| | - Peihang Xu
- Department of Urology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China; Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China
| | - Jun Wang
- Department of Urology, Sun Yat-sen University Cancer Center, Guangzhou, 510060, China
| | - Jintao Li
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China
| | - Fei Ren
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China; SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China.
| | - Dingwei Ye
- Department of Urology, Fudan University Shanghai Cancer Center, Shanghai, 200032, China; Department of Oncology, Shanghai Medical College, Fudan University, Shanghai, 200032, China.
| | - S Kevin Zhou
- Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China; School of Biomedical Engineering & Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, 215123, China.
| |
Collapse
|
47
|
Kang H, Yang M, Zhang F, Xu H, Ren S, Li J, Chen D, Wang F, Li D, Chen X. Identification lymph node metastasis in esophageal squamous cell carcinoma using whole slide images and a hybrid network of multiple instance and transfer learning. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104577] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
48
|
Shi Z, Zhu J, Yu L, Li X, Li J, Chen H, Chen L. A Two-Stage End-to-End Deep Learning Framework for Pathologic Examination in Skin Tumor Diagnosis. THE AMERICAN JOURNAL OF PATHOLOGY 2023:S0002-9440(23)00059-7. [PMID: 36868466 DOI: 10.1016/j.ajpath.2023.02.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Revised: 01/10/2023] [Accepted: 02/10/2023] [Indexed: 03/05/2023]
Abstract
Neurofibromas (NFs), Bowen disease (BD), and seborrheic keratosis (SK) are common skin tumors. Pathologic examination is the golden standard for diagnosis of these tumors. Current pathologic diagnosis is mainly based on the observation of naked eyes under microscope, which is laborious and time-consuming. Digitization of pathology brings the opportunity for artificial intelligence technology to improve the efficiency of diagnosis. This research aims to develop an end-to-end extendable framework for the diagnosis of skin tumor based on pathologic slide images. NF, BD, and SK were selected as target skin tumors. A two-stage skin cancer diagnosis framework is proposed in this article, which consists of two parts: patches-wise diagnosis and slide-wise diagnosis. Patches-wise diagnosis compares different convolutional neural networks to extract features and distinguish categories from patches generated in whole slide images. Slide-wise diagnosis combines attention graph gated network model prediction with post-processing algorithm. This approach can fuse information from feature-embedding learning and domain knowledge to draw a conclusion. Training, validation, and testing were performed on NF, BD, SK, and negative samples. Accuracy and receiver operating characteristic curves were used to evaluate the classification performance. This study investigated the feasibility of skin tumor diagnosis in pathologic image and may be the first time that deep learning is applied to address these three kinds of tumor diagnosis in skin pathology.
Collapse
Affiliation(s)
- Zhijie Shi
- Department of Dermatology, Huashan Hospital, Fudan University, Shanghai, China
| | - Jingyi Zhu
- Academy of Engineering and Technology, Fudan University, Shanghai, China
| | - Liheng Yu
- Academy of Engineering and Technology, Fudan University, Shanghai, China
| | - Xiaopeng Li
- School of Mathematics and Statistics, Central South University, Changsha, China
| | - Jiaxin Li
- Institute of Modern Physics, Chinese Academy of Sciences, Lanzhou, China; University of Chinese Academy of Sciences, Beijing, China
| | - Huyan Chen
- Department of Dermatology, Huashan Hospital, Fudan University, Shanghai, China.
| | - Lianjun Chen
- Department of Dermatology, Huashan Hospital, Fudan University, Shanghai, China.
| |
Collapse
|
49
|
Zhang Y, Lian H, Yang G, Zhao S, Ni P, Chen H, Li C. Inaccurate-Supervised Learning With Generative Adversarial Nets. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:1522-1536. [PMID: 34464286 DOI: 10.1109/tcyb.2021.3104848] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Inaccurate-supervised learning (ISL) is a weakly supervised learning framework for imprecise annotation, which is derived from some specific popular learning frameworks, mainly including partial label learning (PLL), partial multilabel learning (PML), and multiview PML (MVPML). While PLL, PML, and MVPML are each solved as independent models through different methods and no general framework can currently be applied to these frameworks, most existing methods for solving them were designed based on traditional machine-learning techniques, such as logistic regression, KNN, SVM, decision tree. Prior to this study, there was no single general framework that used adversarial networks to solve ISL problems. To narrow this gap, this study proposed an adversarial network structure to solve ISL problems, called ISL with generative adversarial nets (ISL-GANs). In ISL-GAN, fake samples, which are quite similar to real samples, gradually promote the Discriminator to disambiguate the noise labels of real samples. We also provide theoretical analyses for ISL-GAN in effectively handling ISL data. In this article, we propose a general framework to solve PLL, PML, and MVPML, while in the published conference version, we adopt the specific framework, which is a special case of the general one, to solve the PLL problem. Finally, the effectiveness is demonstrated through extensive experiments on various imprecise annotation learning tasks, including PLL, PML, and MVPML.
Collapse
|
50
|
Kim J, Tomita N, Suriawinata AA, Hassanpour S. Detection of Colorectal Adenocarcinoma and Grading Dysplasia on Histopathologic Slides Using Deep Learning. THE AMERICAN JOURNAL OF PATHOLOGY 2023; 193:332-340. [PMID: 36563748 PMCID: PMC10012966 DOI: 10.1016/j.ajpath.2022.12.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Revised: 10/28/2022] [Accepted: 12/01/2022] [Indexed: 12/24/2022]
Abstract
Colorectal cancer (CRC) is one of the most common types of cancer among men and women. The grading of dysplasia and the detection of adenocarcinoma are important clinical tasks in the diagnosis of CRC and shape the patients' follow-up plans. This study evaluated the feasibility of deep learning models for the classification of colorectal lesions into four classes: benign, low-grade dysplasia, high-grade dysplasia, and adenocarcinoma. To this end, a deep neural network was developed on a training set of 655 whole slide images of digitized colorectal resection slides from a tertiary medical institution; and the network was evaluated on an internal test set of 234 slides, as well as on an external test set of 606 adenocarcinoma slides from The Cancer Genome Atlas database. The model achieved an overall accuracy, sensitivity, and specificity of 95.5%, 91.0%, and 97.1%, respectively, on the internal test set, and an accuracy and sensitivity of 98.5% for adenocarcinoma detection task on the external test set. Results suggest that such deep learning models can potentially assist pathologists in grading colorectal dysplasia, detecting adenocarcinoma, prescreening, and prioritizing the reviewing of suspicious cases to improve the turnaround time for patients with a high risk of CRC. Furthermore, the high sensitivity on the external test set suggests the model's generalizability in detecting colorectal adenocarcinoma on whole slide images across different institutions.
Collapse
Affiliation(s)
- Junhwi Kim
- Department of Computer Science, Dartmouth College, Hanover, New Hampshire
| | - Naofumi Tomita
- Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire
| | - Arief A Suriawinata
- Department of Pathology and Laboratory Medicine, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire
| | - Saeed Hassanpour
- Department of Computer Science, Dartmouth College, Hanover, New Hampshire; Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire; Department of Epidemiology, Geisel School of Medicine at Dartmouth, Hanover, New Hampshire.
| |
Collapse
|