1
|
Gonzalez R, Saha A, Campbell CJ, Nejat P, Lokker C, Norgan AP. Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities. J Pathol Inform 2024; 15:100347. [PMID: 38162950 PMCID: PMC10755052 DOI: 10.1016/j.jpi.2023.100347] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Revised: 10/06/2023] [Accepted: 11/01/2023] [Indexed: 01/03/2024] Open
Abstract
This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.
Collapse
Affiliation(s)
- Ricardo Gonzalez
- DeGroote School of Business, McMaster University, Hamilton, Ontario, Canada
- Division of Computational Pathology and Artificial Intelligence, Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, United States
| | - Ashirbani Saha
- Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada
- Escarpment Cancer Research Institute, McMaster University and Hamilton Health Sciences, Hamilton, Ontario, Canada
| | - Clinton J.V. Campbell
- William Osler Health System, Brampton, Ontario, Canada
- Department of Pathology and Molecular Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada
| | - Peyman Nejat
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, United States
| | - Cynthia Lokker
- Health Information Research Unit, Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Andrew P. Norgan
- Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, United States
| |
Collapse
|
2
|
Jaspers TJM, Boers TGW, Kusters CHJ, Jong MR, Jukema JB, de Groof AJ, Bergman JJ, de With PHN, van der Sommen F. Robustness evaluation of deep neural networks for endoscopic image analysis: Insights and strategies. Med Image Anal 2024; 94:103157. [PMID: 38574544 DOI: 10.1016/j.media.2024.103157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 03/19/2024] [Accepted: 03/21/2024] [Indexed: 04/06/2024]
Abstract
Computer-aided detection and diagnosis systems (CADe/CADx) in endoscopy are commonly trained using high-quality imagery, which is not representative for the heterogeneous input typically encountered in clinical practice. In endoscopy, the image quality heavily relies on both the skills and experience of the endoscopist and the specifications of the system used for screening. Factors such as poor illumination, motion blur, and specific post-processing settings can significantly alter the quality and general appearance of these images. This so-called domain gap between the data used for developing the system and the data it encounters after deployment, and the impact it has on the performance of deep neural networks (DNNs) supportive endoscopic CAD systems remains largely unexplored. As many of such systems, for e.g. polyp detection, are already being rolled out in clinical practice, this poses severe patient risks in particularly community hospitals, where both the imaging equipment and experience are subject to considerable variation. Therefore, this study aims to evaluate the impact of this domain gap on the clinical performance of CADe/CADx for various endoscopic applications. For this, we leverage two publicly available data sets (KVASIR-SEG and GIANA) and two in-house data sets. We investigate the performance of commonly-used DNN architectures under synthetic, clinically calibrated image degradations and on a prospectively collected dataset including 342 endoscopic images of lower subjective quality. Additionally, we assess the influence of DNN architecture and complexity, data augmentation, and pretraining techniques for improved robustness. The results reveal a considerable decline in performance of 11.6% (±1.5) as compared to the reference, within the clinically calibrated boundaries of image degradations. Nevertheless, employing more advanced DNN architectures and self-supervised in-domain pre-training effectively mitigate this drop to 7.7% (±2.03). Additionally, these enhancements yield the highest performance on the manually collected test set including images with lower subjective quality. By comprehensively assessing the robustness of popular DNN architectures and training strategies across multiple datasets, this study provides valuable insights into their performance and limitations for endoscopic applications. The findings highlight the importance of including robustness evaluation when developing DNNs for endoscopy applications and propose strategies to mitigate performance loss.
Collapse
Affiliation(s)
- Tim J M Jaspers
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands.
| | - Tim G W Boers
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Carolus H J Kusters
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Martijn R Jong
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Jelmer B Jukema
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Albert J de Groof
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Jacques J Bergman
- Department of Gastroenterology and Hepatology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
| | - Peter H N de With
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Fons van der Sommen
- Department of Electrical Engineering, Video Coding & Architectures, Eindhoven University of Technology, Eindhoven, The Netherlands
| |
Collapse
|
3
|
Bülow RD, Lan YC, Amann K, Boor P. [Artificial intelligence in kidney transplant pathology]. PATHOLOGIE (HEIDELBERG, GERMANY) 2024:10.1007/s00292-024-01324-7. [PMID: 38598097 DOI: 10.1007/s00292-024-01324-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/12/2024] [Indexed: 04/11/2024]
Abstract
BACKGROUND Artificial intelligence (AI) systems have showed promising results in digital pathology, including digital nephropathology and specifically also kidney transplant pathology. AIM Summarize the current state of research and limitations in the field of AI in kidney transplant pathology diagnostics and provide a future outlook. MATERIALS AND METHODS Literature search in PubMed and Web of Science using the search terms "deep learning", "transplant", and "kidney". Based on these results and studies cited in the identified literature, a selection was made of studies that have a histopathological focus and use AI to improve kidney transplant diagnostics. RESULTS AND CONCLUSION Many studies have already made important contributions, particularly to the automation of the quantification of some histopathological lesions in nephropathology. This likely can be extended to automatically quantify all relevant lesions for a kidney transplant, such as Banff lesions. Important limitations and challenges exist in the collection of representative data sets and the updates of Banff classification, making large-scale studies challenging. The already positive study results make future AI support in kidney transplant pathology appear likely.
Collapse
Affiliation(s)
- Roman David Bülow
- Institut für Pathologie, Sektion Nephropathologie, Universitätsklinikum RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Deutschland
| | - Yu-Chia Lan
- Institut für Pathologie, Sektion Nephropathologie, Universitätsklinikum RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Deutschland
| | - Kerstin Amann
- Abteilung Nephropathologie, Institut für Pathologie, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Deutschland
| | - Peter Boor
- Institut für Pathologie, Sektion Nephropathologie, Universitätsklinikum RWTH Aachen, Pauwelsstraße 30, 52074, Aachen, Deutschland.
- Medizinische Klinik II, Universitätsklinikum RWTH Aachen, Aachen, Deutschland.
| |
Collapse
|
4
|
Wang W, Wang Y, Song D, Zhou Y, Luo R, Ying S, Yang L, Sun W, Cai J, Wang X, Bao Z, Zheng J, Zeng M, Gao Q, Wang X, Zhou J, Wang M, Shao G, Rao SX, Zhu K. A Transformer-Based microvascular invasion classifier enhances prognostic stratification in HCC following radiofrequency ablation. Liver Int 2024; 44:894-906. [PMID: 38263714 DOI: 10.1111/liv.15846] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/02/2023] [Revised: 12/15/2023] [Accepted: 01/07/2024] [Indexed: 01/25/2024]
Abstract
BACKGROUND & AIMS We aimed to develop a Transformer-based deep learning (DL) network for prognostic stratification in hepatocellular carcinoma (HCC) patients undergoing RFA. METHODS A Swin Transformer DL network was trained to establish associations between magnetic resonance imaging (MRI) datasets and the ground truth of microvascular invasion (MVI) based on 696 surgical resection (SR) patients with solitary HCC ≤3 cm, and was validated in an external cohort (n = 180). The multiphase MRI-based DL risk outputs using an optimal threshold of .5 was employed as a MVI classifier for prognosis stratification in the RFA cohort (n = 180). RESULTS Over 90% of all enrolled patients exhibited hepatitis B virus infection. Liver cirrhosis was significantly more prevalent in the RFA cohort compared to the SR cohort (72.2% vs. 44.1%, p < .001). The MVI risk outputs exhibited good performance (area under the curve values = .938 and .883) for predicting MVI in the training and validation cohort, respectively. The RFA patients at high risk of MVI classified by the MVI classifier demonstrated significantly lower recurrence-free survival (RFS) and overall survival rates at 1, 3 and 5 years compared to those classified as low risk (p < .001). Multivariate cox regression modelling of a-fetoprotein > 20 ng/mL [hazard ratio (HR) = 1.53; 95% confidence interval (95% CI): 1.02-2.33, p = .047], high risk of MVI (HR = 3.76; 95% CI: 2.40-5.88, p < .001) and unfavourable tumour location (HR = 2.15; 95% CI: 1.40-3.29, p = .001) yielded a c-index of .731 (bootstrapped 95% CI: .667-.778) for evaluating RFS after RFA. Among the three risk factors, MVI was the most powerful predictor for intrahepatic distance recurrence. CONCLUSIONS The proposed MVI classifier can serve as a valuable imaging biomarker for prognostic stratification in early-stage HCC patients undergoing RFA.
Collapse
Affiliation(s)
- Wentao Wang
- Department of Radiology, Cancer Center, Zhongshan Hospital, Fudan University, Shanghai, China
- Shanghai Institute of Medical Imaging, Shanghai, China
| | | | - Danjun Song
- Department of Interventional Therapy, Zhejiang Cancer Hospital, Hangzhou, China
- Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, China
| | - Yingting Zhou
- Department of Hepatic Oncology, Liver Cancer Institute, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Rongkui Luo
- Department of Pathology, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Siqi Ying
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China
- Shanghai Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention, Shanghai, China
| | - Li Yang
- Department of Radiology, Cancer Center, Zhongshan Hospital, Fudan University, Shanghai, China
- Shanghai Institute of Medical Imaging, Shanghai, China
| | - Wei Sun
- Department of Radiology, Cancer Center, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Jiabin Cai
- Department of Liver Surgery, Key Laboratory of Carcinogenesis and Cancer Invasion of Ministry of Education, Liver Cancer Institute, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Xi Wang
- Department of Ultrasound, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Zhen Bao
- Department of Pathology, Zhejiang Cancer Hospital, Hangzhou, China
| | - Jiaping Zheng
- Department of Interventional Therapy, Zhejiang Cancer Hospital, Hangzhou, China
- Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, China
| | - Mengsu Zeng
- Department of Radiology, Cancer Center, Zhongshan Hospital, Fudan University, Shanghai, China
- Shanghai Institute of Medical Imaging, Shanghai, China
| | - Qiang Gao
- Department of Liver Surgery, Key Laboratory of Carcinogenesis and Cancer Invasion of Ministry of Education, Liver Cancer Institute, Zhongshan Hospital, Fudan University, Shanghai, China
- Institutes of Biomedical Sciences, Fudan University, Shanghai, China
| | - Xiaoying Wang
- Department of Liver Surgery, Key Laboratory of Carcinogenesis and Cancer Invasion of Ministry of Education, Liver Cancer Institute, Zhongshan Hospital, Fudan University, Shanghai, China
| | - Jian Zhou
- Department of Liver Surgery, Key Laboratory of Carcinogenesis and Cancer Invasion of Ministry of Education, Liver Cancer Institute, Zhongshan Hospital, Fudan University, Shanghai, China
- Institutes of Biomedical Sciences, Fudan University, Shanghai, China
| | - Manning Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China
- Shanghai Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention, Shanghai, China
| | - Guoliang Shao
- Department of Interventional Therapy, Zhejiang Cancer Hospital, Hangzhou, China
- Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences, Hangzhou, China
| | - Sheng-Xiang Rao
- Department of Radiology, Cancer Center, Zhongshan Hospital, Fudan University, Shanghai, China
- Shanghai Institute of Medical Imaging, Shanghai, China
| | - Kai Zhu
- Department of Liver Surgery, Key Laboratory of Carcinogenesis and Cancer Invasion of Ministry of Education, Liver Cancer Institute, Zhongshan Hospital, Fudan University, Shanghai, China
| |
Collapse
|
5
|
Liu P, Ji L, Ye F, Fu B. AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images. Med Image Anal 2024; 91:103020. [PMID: 37926034 DOI: 10.1016/j.media.2023.103020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 04/05/2023] [Accepted: 10/31/2023] [Indexed: 11/07/2023]
Abstract
The survival analysis on histological whole-slide images (WSIs) is one of the most important means to estimate patient prognosis. Although many weakly-supervised deep learning models have been developed for gigapixel WSIs, their potential is generally restricted by classical survival analysis rules and fully-supervised learning requirements. As a result, these models provide patients only with a completely-certain point estimation of time-to-event, and they could only learn from the labeled WSI data currently at a small scale. To tackle these problems, we propose a novel adversarial multiple instance learning (AdvMIL) framework. This framework is based on adversarial time-to-event modeling, and integrates the multiple instance learning (MIL) that is much necessary for WSI representation learning. It is a plug-and-play one, so that most existing MIL-based end-to-end methods can be easily upgraded by applying this framework, gaining the improved abilities of survival distribution estimation and semi-supervised learning. Our extensive experiments show that AdvMIL not only could often bring performance improvement to mainstream WSI survival analysis methods at a relatively low computational cost, but also enables these methods to effectively utilize unlabeled data via semi-supervised learning. Moreover, it is observed that AdvMIL could help improving the robustness of models against patch occlusion and two representative image noises. The proposed AdvMIL framework could promote the research of survival analysis in computational pathology with its novel adversarial MIL paradigm.
Collapse
Affiliation(s)
- Pei Liu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| | - Luping Ji
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| | - Feng Ye
- Institute of Clinical Pathology, West China Hospital, Sichuan University, Guo Xue Xiang, Chengdu, 610041, Sichuan, China.
| | - Bo Fu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| |
Collapse
|
6
|
Vafaei Sadr A, Bülow R, von Stillfried S, Schmitz NEJ, Pilva P, Hölscher DL, Ha PP, Schweiker M, Boor P. Operational greenhouse-gas emissions of deep learning in digital pathology: a modelling study. Lancet Digit Health 2024; 6:e58-e69. [PMID: 37996339 PMCID: PMC10728828 DOI: 10.1016/s2589-7500(23)00219-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 10/04/2023] [Accepted: 10/16/2023] [Indexed: 11/25/2023]
Abstract
BACKGROUND Deep learning is a promising way to improve health care. Image-processing medical disciplines, such as pathology, are expected to be transformed by deep learning. The first clinically applicable deep-learning diagnostic support tools are already available in cancer pathology, and their number is increasing. However, data on the environmental sustainability of these tools are scarce. We aimed to conduct an environmental-sustainability analysis of a theoretical implementation of deep learning in patient-care pathology. METHODS For this modelling study, we first assembled and calculated relevant data and parameters of a digital-pathology workflow. Data were breast and prostate specimens from the university clinic at the Institute of Pathology of the Rheinisch-Westfälische Technische Hochschule Aachen (Aachen, Germany), for which commercially available deep learning was already available. Only specimens collected between Jan 1 and Dec 31, 2019 were used, to omit potential biases due to the COVID-19 pandemic. Our final selection was based on 2 representative weeks outside holidays, covering different types of specimens. To calculate carbon dioxide (CO2) or CO2 equivalent (CO2 eq) emissions of deep learning in pathology, we gathered relevant data for exact numbers and sizes of whole-slide images (WSIs), which were generated by scanning histopathology samples of prostate and breast specimens. We also evaluated different data input scenarios (including all slide tiles, only tiles containing tissue, or only tiles containing regions of interest). To convert estimated energy consumption from kWh to CO2 eq, we used the internet protocol address of the computational server and the Electricity Maps database to obtain information on the sources of the local electricity grid (ie, renewable vs non-renewable), and estimated the number of trees and proportion of the local and world's forests needed to sequester the CO2 eq emissions. We calculated the computational requirements and CO2 eq emissions of 30 deep-learning models that varied in task and size. The first scenario represented the use of one commercially available deep-learning model for one task in one case (1-task), the second scenario considered two deep-learning models for two tasks per case (2-task), the third scenario represented a future, potentially automated workflow that could handle 7 tasks per case (7-task), and the fourth scenario represented the use of a single potential, large, computer-vision model that could conduct multiple tasks (multitask). We also compared the performance (ie, accuracy) and CO2 eq emissions of different deep-learning models for the classification of renal cell carcinoma on WSIs, also from Rheinisch-Westfälische Technische Hochschule Aachen. We also tested other approaches to reducing CO2 eq emissions, including model pruning and an alternative method for histopathology analysis (pathomics). FINDINGS The pathology database contained 35 552 specimens (237 179 slides), 6420 of which were prostate specimens (10 115 slides) and 11 801 of which were breast specimens (19 763 slides). We selected and subsequently digitised 140 slides from eight breast-cancer cases and 223 slides from five prostate-cancer cases. Applying large deep-learning models on all WSI tiles of prostate and breast pathology cases would result in yearly CO2 eq emissions of 7·65 metric tons (t; 95% CI 7·62-7·68) with the use of a single deep-learning model per case; yearly CO2 eq emissions were up to 100·56 t (100·21-100·99) with the use of seven deep-learning models per case. CO2 eq emissions for different deep-learning model scenarios, data inputs, and deep-learning model sizes for all slides varied from 3·61 t (3·59-3·63) to 2795·30 t (1177·51-6482·13. For the estimated number of overall pathology cases worldwide, the yearly CO2 eq emissions varied, reaching up to 16 megatons (Mt) of CO2 eq, requiring up to 86 590 km2 (0·22%) of world forest to sequester the CO2 eq emissions. Use of the 7-task scenario and small deep-learning models on slides containing tissue only could substantially reduce CO2 eq emissions worldwide by up to 141 times (0·1 Mt, 95% CI 0·1-0·1). Considering the local environment in Aachen, Germany, the maximum CO2 eq emission from the use of deep learning in digital pathology only would require 32·8% (95% CI 13·8-76·6) of the local forest to sequester the CO2 eq emissions. A single pathomics run on a tissue could provide information that was comparable to or even better than the output of multitask deep-learning models, but with 147 times reduced CO2 eq emissions. INTERPRETATION Our findings suggest that widespread use of deep learning in pathology might have considerable global-warming potential. The medical community, policy decision makers, and the public should be aware of this potential and encourage the use of CO2 eq emissions reduction strategies where possible. FUNDING German Research Foundation, European Research Council, German Federal Ministry of Education and Research, Health, Economic Affairs and Climate Action, and the Innovation Fund of the Federal Joint Committee.
Collapse
Affiliation(s)
- Alireza Vafaei Sadr
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany; Department of Public Health Sciences, College of Medicine, Pennsylvania State University, Hershey, PA, USA
| | - Roman Bülow
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Saskia von Stillfried
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Nikolas E J Schmitz
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Pourya Pilva
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - David L Hölscher
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Peiman Pilehchi Ha
- Healthy Living Spaces Lab, Institute for Occupational, Social and Environmental Medicine, Medical Faculty, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Marcel Schweiker
- Healthy Living Spaces Lab, Institute for Occupational, Social and Environmental Medicine, Medical Faculty, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Peter Boor
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany; Department of Nephrology and Immunology, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany.
| |
Collapse
|
7
|
Solovev IA. [Artificial intelligence in pathological anatomy]. Arkh Patol 2024; 86:65-71. [PMID: 38591909 DOI: 10.17116/patol20248602165] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/10/2024]
Abstract
The review presents key concepts and global developments in the field of artificial intelligence used in pathological anatomy. The work examines two types of artificial intelligence (AI): weak and strong ones. A review of experimental algorithms using both deep machine learning and computer vision technologies to work with WSI images of preparations, diagnose and make a prognosis for various malignant neoplasms is carried out. It has been established that weak artificial intelligence at this stage of development of computer (digital) pathological anatomy shows significantly better results in speeding up and refining diagnostic procedures than strong artificial intelligence having signs of general intelligence. The article also discusses three options for the further development of AI assistants for pathologists based on the technologies of large language models (strong AI) ChatGPT (PathAsst), Flan-PaLM2 and LIMA. As a result of the analysis of the literature, key problems in the field were identified: the equipment of pathology institutions, the lack of experts in training neural networks, the lack of strict criteria for the clinical viability of AI diagnostic technologies.
Collapse
Affiliation(s)
- I A Solovev
- Pitirim Sorokin Syktyvkar State University, Syktyvkar, Russia
| |
Collapse
|
8
|
王 星, 李 茜, 马 彩, 张 铄, 林 钰, 李 建, 刘 澄. [Artificial intelligence in wearable electrocardiogram monitoring]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2023; 40:1084-1092. [PMID: 38151930 PMCID: PMC10753313 DOI: 10.7507/1001-5515.202301032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 10/31/2023] [Indexed: 12/29/2023]
Abstract
Electrocardiogram (ECG) monitoring owns important clinical value in diagnosis, prevention and rehabilitation of cardiovascular disease (CVD). With the rapid development of Internet of Things (IoT), big data, cloud computing, artificial intelligence (AI) and other advanced technologies, wearable ECG is playing an increasingly important role. With the aging process of the population, it is more and more urgent to upgrade the diagnostic mode of CVD. Using AI technology to assist the clinical analysis of long-term ECGs, and thus to improve the ability of early detection and prediction of CVD has become an important direction. Intelligent wearable ECG monitoring needs the collaboration between edge and cloud computing. Meanwhile, the clarity of medical scene is conducive for the precise implementation of wearable ECG monitoring. This paper first summarized the progress of AI-related ECG studies and the current technical orientation. Then three cases were depicted to illustrate how the AI in wearable ECG cooperate with the clinic. Finally, we demonstrated the two core issues-the reliability and worth of AI-related ECG technology and prospected the future opportunities and challenges.
Collapse
Affiliation(s)
- 星尧 王
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| | - 茜 李
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| | - 彩云 马
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| | - 铄 张
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| | - 钰洁 林
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| | - 建清 李
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| | - 澄玉 刘
- 东南大学 仪器科学与工程学院(南京 210096)School of Instrument Science and Engineering, Southeast University, Nanjing 210096, P. R. China
- 东南大学 生物电子学国家重点实验室(南京 210096)State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, P. R. China
| |
Collapse
|
9
|
Chauveau B, Merville P. Segment Anything by Meta as a foundation model for image segmentation: a new era for histopathological images. Pathology 2023; 55:1017-1020. [PMID: 37813761 DOI: 10.1016/j.pathol.2023.09.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 08/16/2023] [Accepted: 09/17/2023] [Indexed: 10/11/2023]
Affiliation(s)
- Bertrand Chauveau
- CHU de Bordeaux, Service de Pathologie, Hôpital Pellegrin, Bordeaux, France; ImmunoConcEpT, CNRS, Université Bordeaux, UMR 5164, Bordeaux, France.
| | - Pierre Merville
- ImmunoConcEpT, CNRS, Université Bordeaux, UMR 5164, Bordeaux, France; CHU de Bordeaux, Service de Néphrologie-Transplantation-Dialyse-Aphéréses, Hôpital Pellegrin, Bordeaux, France
| |
Collapse
|
10
|
Waqas A, Bui MM, Glassy EF, El Naqa I, Borkowski P, Borkowski AA, Rasool G. Revolutionizing Digital Pathology With the Power of Generative Artificial Intelligence and Foundation Models. J Transl Med 2023; 103:100255. [PMID: 37757969 DOI: 10.1016/j.labinv.2023.100255] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 09/06/2023] [Accepted: 09/19/2023] [Indexed: 09/29/2023] Open
Abstract
Digital pathology has transformed the traditional pathology practice of analyzing tissue under a microscope into a computer vision workflow. Whole-slide imaging allows pathologists to view and analyze microscopic images on a computer monitor, enabling computational pathology. By leveraging artificial intelligence (AI) and machine learning (ML), computational pathology has emerged as a promising field in recent years. Recently, task-specific AI/ML (eg, convolutional neural networks) has risen to the forefront, achieving above-human performance in many image-processing and computer vision tasks. The performance of task-specific AI/ML models depends on the availability of many annotated training datasets, which presents a rate-limiting factor for AI/ML development in pathology. Task-specific AI/ML models cannot benefit from multimodal data and lack generalization, eg, the AI models often struggle to generalize to new datasets or unseen variations in image acquisition, staining techniques, or tissue types. The 2020s are witnessing the rise of foundation models and generative AI. A foundation model is a large AI model trained using sizable data, which is later adapted (or fine-tuned) to perform different tasks using a modest amount of task-specific annotated data. These AI models provide in-context learning, can self-correct mistakes, and promptly adjust to user feedback. In this review, we provide a brief overview of recent advances in computational pathology enabled by task-specific AI, their challenges and limitations, and then introduce various foundation models. We propose to create a pathology-specific generative AI based on multimodal foundation models and present its potentially transformative role in digital pathology. We describe different use cases, delineating how it could serve as an expert companion of pathologists and help them efficiently and objectively perform routine laboratory tasks, including quantifying image analysis, generating pathology reports, diagnosis, and prognosis. We also outline the potential role that foundation models and generative AI can play in standardizing the pathology laboratory workflow, education, and training.
Collapse
Affiliation(s)
- Asim Waqas
- Department of Machine Learning, H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida; Department of Electrical Engineering, University of South Florida, Tampa, Florida.
| | - Marilyn M Bui
- Department of Machine Learning, H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida; Department of Pathology, H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida; University of South Florida, Morsani College of Medicine, Tampa, Florida
| | - Eric F Glassy
- Affiliated Pathologists Medical Group, Inc., Rancho Dominguez, California
| | - Issam El Naqa
- Department of Machine Learning, H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida
| | - Piotr Borkowski
- Quest Diagnostics/Ameripath, Tampa, Florida; Center of Excellence for Digital and AI-Empowered Pathology, Quest Diagnostics, Tampa, Florida
| | - Andrew A Borkowski
- University of South Florida, Morsani College of Medicine, Tampa, Florida; James A. Haley Veterans' Hospital, Tampa, Florida; National Artificial Intelligence Institute, Washington, District of Columbia
| | - Ghulam Rasool
- Department of Machine Learning, H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida; Department of Electrical Engineering, University of South Florida, Tampa, Florida; University of South Florida, Morsani College of Medicine, Tampa, Florida; Department of Neuro-Oncology, H. Lee Moffitt Cancer Center and Research Institute, Tampa, Florida
| |
Collapse
|
11
|
Wagner SJ, Reisenbüchler D, West NP, Niehues JM, Zhu J, Foersch S, Veldhuizen GP, Quirke P, Grabsch HI, van den Brandt PA, Hutchins GGA, Richman SD, Yuan T, Langer R, Jenniskens JCA, Offermans K, Mueller W, Gray R, Gruber SB, Greenson JK, Rennert G, Bonner JD, Schmolze D, Jonnagaddala J, Hawkins NJ, Ward RL, Morton D, Seymour M, Magill L, Nowak M, Hay J, Koelzer VH, Church DN, Matek C, Geppert C, Peng C, Zhi C, Ouyang X, James JA, Loughrey MB, Salto-Tellez M, Brenner H, Hoffmeister M, Truhn D, Schnabel JA, Boxberg M, Peng T, Kather JN. Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study. Cancer Cell 2023; 41:1650-1661.e4. [PMID: 37652006 PMCID: PMC10507381 DOI: 10.1016/j.ccell.2023.08.002] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 06/18/2023] [Accepted: 08/07/2023] [Indexed: 09/02/2023]
Abstract
Deep learning (DL) can accelerate the prediction of prognostic biomarkers from routine pathology slides in colorectal cancer (CRC). However, current approaches rely on convolutional neural networks (CNNs) and have mostly been validated on small patient cohorts. Here, we develop a new transformer-based pipeline for end-to-end biomarker prediction from pathology slides by combining a pre-trained transformer encoder with a transformer network for patch aggregation. Our transformer-based approach substantially improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training and evaluating on a large multicenter cohort of over 13,000 patients from 16 colorectal cancer cohorts, we achieve a sensitivity of 0.99 with a negative predictive value of over 0.99 for prediction of microsatellite instability (MSI) on surgical resection specimens. We demonstrate that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem.
Collapse
Affiliation(s)
- Sophia J Wagner
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany; Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany
| | - Daniel Reisenbüchler
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany
| | - Nicholas P West
- Institute of Pathology, University Medical Center Mainz, Mainz, Germany
| | - Jan Moritz Niehues
- Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany
| | - Jiefu Zhu
- Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany
| | - Sebastian Foersch
- Institute of Pathology, University Medical Center Mainz, Mainz, Germany
| | | | - Philip Quirke
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Heike I Grabsch
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Department of Pathology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Piet A van den Brandt
- Department of Epidemiology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Gordon G A Hutchins
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Susan D Richman
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Rupert Langer
- Institute of Pathology und Molecular Pathology, Johannes Kepler University Hospital Linz, Linz, Österreich
| | - Josien C A Jenniskens
- Department of Epidemiology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Kelly Offermans
- Department of Epidemiology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | | | - Richard Gray
- Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Stephen B Gruber
- Center for Precision Medicine and Department of Medical Oncology, City of Hope National Medical Center, Duarte, CA, USA
| | - Joel K Greenson
- Department of Pathology, City of Hope Comprehensive Cancer Center, Duarte, CA, USA
| | - Gad Rennert
- Department of Community Medicine & Epidemiology, Lady Davis Carmel Medical Center, Ruth & Bruce Rappaport Faculty of Medicine, Technion-Israel Institute of Technology, Haifa, Israel; Steve and Cindy Rasmussen Institute for Genomic Medicine, Lady Davis Carmel Medical Center and Technion Faculty of Medicine, Clalit National Cancer Control Center, Haifa, Israel
| | - Joseph D Bonner
- Department of Community Medicine & Epidemiology, Lady Davis Carmel Medical Center, Ruth & Bruce Rappaport Faculty of Medicine, Technion-Israel Institute of Technology, Haifa, Israel
| | - Daniel Schmolze
- Center for Precision Medicine and Department of Medical Oncology, City of Hope National Medical Center, Duarte, CA, USA
| | - Jitendra Jonnagaddala
- School of Population Health, Faculty of Medicine and Health, UNSW Sydney, Sydney, NSW, Australia
| | - Nicholas J Hawkins
- School of Medical Sciences, Faculty of Medicine and Health, UNSW Sydney, Sydney, NSW, Australia
| | - Robyn L Ward
- School of Medical Sciences, Faculty of Medicine and Health, UNSW Sydney, Sydney, NSW, Australia; Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia
| | - Dion Morton
- University Hospital Birmingham, Birmingham, UK
| | | | - Laura Magill
- University of Birmingham Clinical Trials Unit, Birmingham, UK
| | - Marta Nowak
- Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Jennifer Hay
- Glasgow Tissue Research Facility, University of Glasgow, Queen Elizabeth University Hospital, Glasgow, UK
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland; Department of Oncology, University of Oxford, Oxford, UK; Nuffield Department of Medicine, University of Oxford, Roosevelt Drive, Oxford, UK
| | - David N Church
- Nuffield Department of Medicine, University of Oxford, Roosevelt Drive, Oxford, UK; Oxford NIHR Comprehensive Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Christian Matek
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; Institute of Pathology, University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany; Comprehensive Cancer Center Erlangen-EMN (CCC), University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany
| | - Carol Geppert
- Institute of Pathology, University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany; Comprehensive Cancer Center Erlangen-EMN (CCC), University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany
| | - Chaolong Peng
- Medical School, Jianggang Shan University, Jiangxi, China
| | - Cheng Zhi
- Department of Pathology, the Second Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
| | - Xiaoming Ouyang
- Department of Pathology, the Second Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
| | - Jacqueline A James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK; Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, UK; The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK
| | - Maurice B Loughrey
- The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK; Department of Cellular Pathology, Belfast Health and Social Care Trust, Belfast, UK; Centre for Public Health, Queen's University Belfast, Belfast, UK
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK; Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, UK; Integrated Pathology Unit, Institute for Cancer Research and Royal Marsden Hospital, London, UK
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany; Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Julia A Schnabel
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany; School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK
| | - Melanie Boxberg
- Institute of Pathology, Technical University Munich, Munich, Germany; Institute of Pathology Munich-North, Munich, Germany
| | - Tingying Peng
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany.
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany; Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg.
| |
Collapse
|
12
|
Veldhuizen GP, Röcken C, Behrens HM, Cifci D, Muti HS, Yoshikawa T, Arai T, Oshima T, Tan P, Ebert MP, Pearson AT, Calderaro J, Grabsch HI, Kather JN. Deep learning-based subtyping of gastric cancer histology predicts clinical outcome: a multi-institutional retrospective study. Gastric Cancer 2023; 26:708-720. [PMID: 37269416 PMCID: PMC10361890 DOI: 10.1007/s10120-023-01398-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Accepted: 05/09/2023] [Indexed: 06/05/2023]
Abstract
INTRODUCTION The Laurén classification is widely used for Gastric Cancer (GC) histology subtyping. However, this classification is prone to interobserver variability and its prognostic value remains controversial. Deep Learning (DL)-based assessment of hematoxylin and eosin (H&E) stained slides is a potentially useful tool to provide an additional layer of clinically relevant information, but has not been systematically assessed in GC. OBJECTIVE We aimed to train, test and externally validate a deep learning-based classifier for GC histology subtyping using routine H&E stained tissue sections from gastric adenocarcinomas and to assess its potential prognostic utility. METHODS We trained a binary classifier on intestinal and diffuse type GC whole slide images for a subset of the TCGA cohort (N = 166) using attention-based multiple instance learning. The ground truth of 166 GC was obtained by two expert pathologists. We deployed the model on two external GC patient cohorts, one from Europe (N = 322) and one from Japan (N = 243). We assessed classification performance using the Area Under the Receiver Operating Characteristic Curve (AUROC) and prognostic value (overall, cancer specific and disease free survival) of the DL-based classifier with uni- and multivariate Cox proportional hazard models and Kaplan-Meier curves with log-rank test statistics. RESULTS Internal validation using the TCGA GC cohort using five-fold cross-validation achieved a mean AUROC of 0.93 ± 0.07. External validation showed that the DL-based classifier can better stratify GC patients' 5-year survival compared to pathologist-based Laurén classification for all survival endpoints, despite frequently divergent model-pathologist classifications. Univariate overall survival Hazard Ratios (HRs) of pathologist-based Laurén classification (diffuse type versus intestinal type) were 1.14 (95% Confidence Interval (CI) 0.66-1.44, p-value = 0.51) and 1.23 (95% CI 0.96-1.43, p-value = 0.09) in the Japanese and European cohorts, respectively. DL-based histology classification resulted in HR of 1.46 (95% CI 1.18-1.65, p-value < 0.005) and 1.41 (95% CI 1.20-1.57, p-value < 0.005), in the Japanese and European cohorts, respectively. In diffuse type GC (as defined by the pathologist), classifying patients using the DL diffuse and intestinal classifications provided a superior survival stratification, and demonstrated statistically significant survival stratification when combined with pathologist classification for both the Asian (overall survival log-rank test p-value < 0.005, HR 1.43 (95% CI 1.05-1.66, p-value = 0.03) and European cohorts (overall survival log-rank test p-value < 0.005, HR 1.56 (95% CI 1.16-1.76, p-value < 0.005)). CONCLUSION Our study shows that gastric adenocarcinoma subtyping using pathologist's Laurén classification as ground truth can be performed using current state of the art DL techniques. Patient survival stratification seems to be better by DL-based histology typing compared with expert pathologist histology typing. DL-based GC histology typing has potential as an aid in subtyping. Further investigations are warranted to fully understand the underlying biological mechanisms for the improved survival stratification despite apparent imperfect classification by the DL algorithm.
Collapse
Affiliation(s)
| | - Christoph Röcken
- Department of Pathology, Christian-Albrechts University, Kiel, Germany
| | | | - Didem Cifci
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Hannah Sophie Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, Technical University Dresden, University Hospital Carl Gustav Carus, Dresden, Germany
| | - Takaki Yoshikawa
- Department of Gastric Surgery, National Cancer Center Hospital, Tokyo, Japan
| | - Tomio Arai
- Department of Pathology, Tokyo Metropolitan Geriatric Hospital and Institute of Gerontology, Tokyo, Japan
| | - Takashi Oshima
- Department of Gastrointestinal Surgery, Kanagawa Cancer Center, Yokohama, Japan
| | - Patrick Tan
- Duke-NUS Medical School, Singapore, Singapore
| | - Matthias P Ebert
- Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
- DKFZ-Hector Cancer Institute at the University Medical Center, Mannheim, Germany
- Clinical Cooperation Unit Healthy Metabolism, Center for Preventive Medicine and Digital Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
- Mannheim Institute for Innate Immunoscience (MI3), Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | - Alexander T Pearson
- Department of Medicine, Section of Hematology/Oncology, The University of Chicago, Chicago, IL, USA
| | - Julien Calderaro
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Department of Pathology, Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Créteil, France
| | - Heike I Grabsch
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Department of Pathology, GROW School for Oncology and Reproduction, Maastricht University Medical Center+, Maastricht, The Netherlands
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
13
|
Müller-Franzes G, Müller-Franzes F, Huck L, Raaff V, Kemmer E, Khader F, Arasteh ST, Lemainque T, Kather JN, Nebelung S, Kuhl C, Truhn D. Fibroglandular tissue segmentation in breast MRI using vision transformers: a multi-institutional evaluation. Sci Rep 2023; 13:14207. [PMID: 37648728 PMCID: PMC10468506 DOI: 10.1038/s41598-023-41331-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Accepted: 08/24/2023] [Indexed: 09/01/2023] Open
Abstract
Accurate and automatic segmentation of fibroglandular tissue in breast MRI screening is essential for the quantification of breast density and background parenchymal enhancement. In this retrospective study, we developed and evaluated a transformer-based neural network for breast segmentation (TraBS) in multi-institutional MRI data, and compared its performance to the well established convolutional neural network nnUNet. TraBS and nnUNet were trained and tested on 200 internal and 40 external breast MRI examinations using manual segmentations generated by experienced human readers. Segmentation performance was assessed in terms of the Dice score and the average symmetric surface distance. The Dice score for nnUNet was lower than for TraBS on the internal testset (0.909 ± 0.069 versus 0.916 ± 0.067, P < 0.001) and on the external testset (0.824 ± 0.144 versus 0.864 ± 0.081, P = 0.004). Moreover, the average symmetric surface distance was higher (= worse) for nnUNet than for TraBS on the internal (0.657 ± 2.856 versus 0.548 ± 2.195, P = 0.001) and on the external testset (0.727 ± 0.620 versus 0.584 ± 0.413, P = 0.03). Our study demonstrates that transformer-based networks improve the quality of fibroglandular tissue segmentation in breast MRI compared to convolutional-based models like nnUNet. These findings might help to enhance the accuracy of breast density and parenchymal enhancement quantification in breast MRI screening.
Collapse
Affiliation(s)
- Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Fritz Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Luisa Huck
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Vanessa Raaff
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Eva Kemmer
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Teresa Lemainque
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University, Dresden, Germany
- Department of Medicine III, University Hospital RWTH, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany.
| |
Collapse
|
14
|
Joel MZ, Avesta A, Yang DX, Zhou JG, Omuro A, Herbst RS, Krumholz HM, Aneja S. Comparing Detection Schemes for Adversarial Images against Deep Learning Models for Cancer Imaging. Cancers (Basel) 2023; 15:1548. [PMID: 36900339 PMCID: PMC10000732 DOI: 10.3390/cancers15051548] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2023] [Revised: 02/27/2023] [Accepted: 02/27/2023] [Indexed: 03/05/2023] Open
Abstract
Deep learning (DL) models have demonstrated state-of-the-art performance in the classification of diagnostic imaging in oncology. However, DL models for medical images can be compromised by adversarial images, where pixel values of input images are manipulated to deceive the DL model. To address this limitation, our study investigates the detectability of adversarial images in oncology using multiple detection schemes. Experiments were conducted on thoracic computed tomography (CT) scans, mammography, and brain magnetic resonance imaging (MRI). For each dataset we trained a convolutional neural network to classify the presence or absence of malignancy. We trained five DL and machine learning (ML)-based detection models and tested their performance in detecting adversarial images. Adversarial images generated using projected gradient descent (PGD) with a perturbation size of 0.004 were detected by the ResNet detection model with an accuracy of 100% for CT, 100% for mammogram, and 90.0% for MRI. Overall, adversarial images were detected with high accuracy in settings where adversarial perturbation was above set thresholds. Adversarial detection should be considered alongside adversarial training as a defense technique to protect DL models for cancer imaging classification from the threat of adversarial images.
Collapse
Affiliation(s)
- Marina Z. Joel
- Department of Dermatology, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
- Department of Therapeutic Radiology, Yale School of Medicine, New Haven, CT 06510, USA
| | - Arman Avesta
- Department of Therapeutic Radiology, Yale School of Medicine, New Haven, CT 06510, USA
| | - Daniel X. Yang
- Department of Therapeutic Radiology, Yale School of Medicine, New Haven, CT 06510, USA
| | - Jian-Ge Zhou
- Department of Chemistry, Physics and Atmospheric Science, Jackson State University, Jackson, MS 39217, USA
| | - Antonio Omuro
- Department of Neurology, Yale School of Medicine, New Haven, CT 06510, USA
| | - Roy S. Herbst
- Department of Medicine, Yale School of Medicine, New Haven, CT 06510, USA
| | - Harlan M. Krumholz
- Department of Medicine, Yale School of Medicine, New Haven, CT 06510, USA
- Center for Outcomes Research and Evaluation (CORE), Yale School of Medicine, New Haven, CT 06510, USA
| | - Sanjay Aneja
- Department of Therapeutic Radiology, Yale School of Medicine, New Haven, CT 06510, USA
- Center for Outcomes Research and Evaluation (CORE), Yale School of Medicine, New Haven, CT 06510, USA
| |
Collapse
|