1
|
Petrella RJ. The AI Future of Emergency Medicine. Ann Emerg Med 2024:S0196-0644(24)00043-X. [PMID: 38795081 DOI: 10.1016/j.annemergmed.2024.01.031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 01/23/2024] [Accepted: 01/24/2024] [Indexed: 05/27/2024]
Abstract
In the coming years, artificial intelligence (AI) and machine learning will likely give rise to profound changes in the field of emergency medicine, and medicine more broadly. This article discusses these anticipated changes in terms of 3 overlapping yet distinct stages of AI development. It reviews some fundamental concepts in AI and explores their relation to clinical practice, with a focus on emergency medicine. In addition, it describes some of the applications of AI in disease diagnosis, prognosis, and treatment, as well as some of the practical issues that they raise, the barriers to their implementation, and some of the legal and regulatory challenges they create.
Collapse
Affiliation(s)
- Robert J Petrella
- Emergency Departments, CharterCARE Health Partners, Providence and North Providence, RI; Emergency Department, Boston VA Medical Center, Boston, MA; Emergency Departments, Steward Health Care System, Boston and Methuen, MA; Harvard Medical School, Boston, MA; Department of Chemistry and Chemical Biology, Harvard University, Cambridge, MA; Department of Medicine, Brigham and Women's Hospital, Boston, MA.
| |
Collapse
|
2
|
Sheng H, Ma L, Samson JF, Liu D. BarlowTwins-CXR: enhancing chest X-ray abnormality localization in heterogeneous data with cross-domain self-supervised learning. BMC Med Inform Decis Mak 2024; 24:126. [PMID: 38755563 PMCID: PMC11097466 DOI: 10.1186/s12911-024-02529-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2024] [Accepted: 05/07/2024] [Indexed: 05/18/2024] Open
Abstract
BACKGROUND Chest X-ray imaging based abnormality localization, essential in diagnosing various diseases, faces significant clinical challenges due to complex interpretations and the growing workload of radiologists. While recent advances in deep learning offer promising solutions, there is still a critical issue of domain inconsistency in cross-domain transfer learning, which hampers the efficiency and accuracy of diagnostic processes. This study aims to address the domain inconsistency problem and improve autonomic abnormality localization performance of heterogeneous chest X-ray image analysis, particularly in detecting abnormalities, by developing a self-supervised learning strategy called "BarlwoTwins-CXR". METHODS We utilized two publicly available datasets: the NIH Chest X-ray Dataset and the VinDr-CXR. The BarlowTwins-CXR approach was conducted in a two-stage training process. Initially, self-supervised pre-training was performed using an adjusted Barlow Twins algorithm on the NIH dataset with a Resnet50 backbone pre-trained on ImageNet. This was followed by supervised fine-tuning on the VinDr-CXR dataset using Faster R-CNN with Feature Pyramid Network (FPN). The study employed mean Average Precision (mAP) at an Intersection over Union (IoU) of 50% and Area Under the Curve (AUC) for performance evaluation. RESULTS Our experiments showed a significant improvement in model performance with BarlowTwins-CXR. The approach achieved a 3% increase in mAP50 accuracy compared to traditional ImageNet pre-trained models. In addition, the Ablation CAM method revealed enhanced precision in localizing chest abnormalities. The study involved 112,120 images from the NIH dataset and 18,000 images from the VinDr-CXR dataset, indicating robust training and testing samples. CONCLUSION BarlowTwins-CXR significantly enhances the efficiency and accuracy of chest X-ray image-based abnormality localization, outperforming traditional transfer learning methods and effectively overcoming domain inconsistency in cross-domain scenarios. Our experiment results demonstrate the potential of using self-supervised learning to improve the generalizability of models in medical settings with limited amounts of heterogeneous data. This approach can be instrumental in aiding radiologists, particularly in high-workload environments, offering a promising direction for future AI-driven healthcare solutions.
Collapse
Affiliation(s)
- Haoyue Sheng
- Département d'informatique et de recherche opérationnelle, Université de Montréal, 2920 chemin de la Tour, Montréal, H3T 1J4, QC, Canada.
- Mila - Quebec AI Institute, 6666 Rue Saint-Urbain, Montréal, H2S 3H1, QC, Canada.
- Direction des ressources informationnelles, CIUSSS du Centre-Sud-de-l'Île-de-Montréal, 400 Blvd. De Maisonneuve Ouest, Montréal, H3A 1L4, QC, Canada.
| | - Linrui Ma
- Département d'informatique et de recherche opérationnelle, Université de Montréal, 2920 chemin de la Tour, Montréal, H3T 1J4, QC, Canada
- Mila - Quebec AI Institute, 6666 Rue Saint-Urbain, Montréal, H2S 3H1, QC, Canada
| | - Jean-François Samson
- Direction des ressources informationnelles, CIUSSS du Centre-Sud-de-l'Île-de-Montréal, 400 Blvd. De Maisonneuve Ouest, Montréal, H3A 1L4, QC, Canada
| | - Dianbo Liu
- Mila - Quebec AI Institute, 6666 Rue Saint-Urbain, Montréal, H2S 3H1, QC, Canada
- School of Medicine and College of Design and Engineering, National University of Singapore, 21 Lower Kent Ridge Rd, Singapore, 119077, SG, Singapore
| |
Collapse
|
3
|
Kim H, Kim K, Oh SJ, Lee S, Woo JH, Kim JH, Cha YK, Kim K, Chung MJ. AI-assisted Analysis to Facilitate Detection of Humeral Lesions on Chest Radiographs. Radiol Artif Intell 2024; 6:e230094. [PMID: 38446041 DOI: 10.1148/ryai.230094] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/07/2024]
Abstract
Purpose To develop an artificial intelligence (AI) system for humeral tumor detection on chest radiographs (CRs) and evaluate the impact on reader performance. Materials and Methods In this retrospective study, 14 709 CRs (January 2000 to December 2021) were collected from 13 468 patients, including CT-proven normal (n = 13 116) and humeral tumor (n = 1593) cases. The data were divided into training and test groups. A novel training method called false-positive activation area reduction (FPAR) was introduced to enhance the diagnostic performance by focusing on the humeral region. The AI program and 10 radiologists were assessed using holdout test set 1, wherein the radiologists were tested twice (with and without AI test results). The performance of the AI system was evaluated using holdout test set 2, comprising 10 497 normal images. Receiver operating characteristic analyses were conducted for evaluating model performance. Results FPAR application in the AI program improved its performance compared with a conventional model based on the area under the receiver operating characteristic curve (0.87 vs 0.82, P = .04). The proposed AI system also demonstrated improved tumor localization accuracy (80% vs 57%, P < .001). In holdout test set 2, the proposed AI system exhibited a false-positive rate of 2%. AI assistance improved the radiologists' sensitivity, specificity, and accuracy by 8.9%, 1.2%, and 3.5%, respectively (P < .05 for all). Conclusion The proposed AI tool incorporating FPAR improved humeral tumor detection on CRs and reduced false-positive results in tumor visualization. It may serve as a supportive diagnostic tool to alert radiologists about humeral abnormalities. Keywords: Artificial Intelligence, Conventional Radiography, Humerus, Machine Learning, Shoulder, Tumor Supplemental material is available for this article. © RSNA, 2024.
Collapse
Affiliation(s)
- Harim Kim
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Kyungsu Kim
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Seong Je Oh
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Sungjoo Lee
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Jung Han Woo
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Jong Hee Kim
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Yoon Ki Cha
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Kyunga Kim
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| | - Myung Jin Chung
- From the Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, 81 Irwon-Ro, Gangnam-Gu, Seoul 06351, South Korea (H.K., J.H.W., J.H.K., Y.K.C., M.J.C.); Medical AI Research Center, Samsung Medical Center, Seoul, South Korea (Kyungsu Kim, M.J.C.); Department of Data Convergence and Future Medicine, Sungkyunkwan University School of Medicine, Seoul, South Korea (Kyungsu Kim, Kyunga Kim, M.J.C.); and Department of Health Sciences and Technology (S.J.O.) and Department of Digital Health (S.L., Kyunga Kim), Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Seoul, South Korea
| |
Collapse
|
4
|
Cohen I, Sorin V, Lekach R, Raskin D, Segev M, Klang E, Eshed I, Barash Y. Artificial intelligence for detection of effusion and lipo-hemarthrosis in X-rays and CT of the knee. Eur J Radiol 2024; 175:111460. [PMID: 38608501 DOI: 10.1016/j.ejrad.2024.111460] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 03/29/2024] [Accepted: 04/08/2024] [Indexed: 04/14/2024]
Abstract
BACKGROUND Traumatic knee injuries are challenging to diagnose accurately through radiography and to a lesser extent, through CT, with fractures sometimes overlooked. Ancillary signs like joint effusion or lipo-hemarthrosis are indicative of fractures, suggesting the need for further imaging. Artificial Intelligence (AI) can automate image analysis, improving diagnostic accuracy and help prioritizing clinically important X-ray or CT studies. OBJECTIVE To develop and evaluate an AI algorithm for detecting effusion of any kind in knee X-rays and selected CT images and distinguishing between simple effusion and lipo-hemarthrosis indicative of intra-articular fractures. METHODS This retrospective study analyzed post traumatic knee imaging from January 2016 to February 2023, categorizing images into lipo-hemarthrosis, simple effusion, or normal. It utilized the FishNet-150 algorithm for image classification, with class activation maps highlighting decision-influential regions. The AI's diagnostic accuracy was validated against a gold standard, based on the evaluations made by a radiologist with at least four years of experience. RESULTS Analysis included CT images from 515 patients and X-rays from 637 post traumatic patients, identifying lipo-hemarthrosis, simple effusion, and normal findings. The AI showed an AUC of 0.81 for detecting any effusion, 0.78 for simple effusion, and 0.83 for lipo-hemarthrosis in X-rays; and 0.89, 0.89, and 0.91, respectively, in CTs. CONCLUSION The AI algorithm effectively detects knee effusion and differentiates between simple effusion and lipo-hemarthrosis in post-traumatic patients for both X-rays and selected CT images further studies are needed to validate these results.
Collapse
Affiliation(s)
- Israel Cohen
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| | - Vera Sorin
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| | - Ruth Lekach
- Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel; Department of Nuclear Medicine, Sourasky Medical Center, Tel-Aviv, Israel.
| | - Daniel Raskin
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| | - Maria Segev
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| | - Eyal Klang
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| | - Iris Eshed
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| | - Yiftach Barash
- Department of Diagnostic Imaging, Sheba Medical Center, Tel Hashomer, Israel; Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel.
| |
Collapse
|
5
|
Alshammeri AF, Alhamaid YA, Alshakhs AM, Bohulaigah ZH, Eissa GA, Almutairi MS, Alhadi W, Algafly HA. X-ray interpretation in emergency department in the Kingdom of Saudi Arabia. Do we need the radiologist? Emerg Radiol 2024; 31:203-212. [PMID: 38499960 DOI: 10.1007/s10140-024-02217-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Accepted: 03/11/2024] [Indexed: 03/20/2024]
Abstract
INTRODUCTION Chest x-rays are widely used for diagnosing chest pathology worldwide. Pediatricians frequently interpret chest radiographs in the emergency department, guiding patient management. This study aims to assess the competency of non-radiologists in interpreting emergency chest x-rays and compare it with trainees of different levels to determine the necessity of radiologist input. METHODOLOGY A cross-sectional online survey was conducted in Saudi Arabia from September to October 2023, involving 385 participants, including pediatricians and medical interns from various regions. Carefully selected questions addressed a range of x-ray abnormalities in pediatric emergencies, assessing fundamental understanding of x-ray interpretation, such as inspiratory vs. expiratory and AP or PA films. RESULTS The study included 385 participants, primarily Saudi nationals in the eastern region, with an equal gender distribution and ages ranging from 20 to 29 years. Approximately 29.09% demonstrated fair knowledge, with 28% being Junior Pediatrics Residents, 18% Pediatric Consultants, and 15% Senior Pediatrics Residents. Fair knowledge was significantly associated with individuals aged 20-29 years, residents of the western region, and Junior Pediatrics Residents. Clinical knowledge varied among different groups, with 59% correctly identifying atypical pneumonia and 65% recognizing asymmetrical hyperinflation. However, rates for other conditions differed, with low identification of potential foreign body aspiration and film type. Accuracy in identifying tension pneumothorax and hyperlucency varied among clinicians. Pleural effusion films had a 65% identification rate for the diagnosis, but only 28% accurately described the X-ray and selected the correct answer for lung opacity. CONCLUSION The study concluded that 29.9% of the participating physicians exhibited fair knowledge of common pediatric emergency radiological films. Junior pediatric residents showed the best knowledge, and Tetralogy of Fallot, asymmetrical hyperinflation, and pleural effusion had the highest recognition rates. In conclusion, there is still a need for radiologists in the pediatric emergency department to ensure optimal functioning.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Wajd Alhadi
- College of Medicine, King Khalid University, Abha, Saudi Arabia
| | | |
Collapse
|
6
|
Hu B, Shi Z, Lu L, Miao Z, Wang H, Zhou Z, Zhang F, Wang R, Luo X, Xu F, Li S, Fang X, Wang X, Yan G, Lv F, Zhang M, Sun Q, Cui G, Liu Y, Zhang S, Pan C, Hou Z, Liang H, Pan Y, Chen X, Li X, Zhou F, Schoepf UJ, Varga-Szemes A, Garrison Moore W, Yu Y, Hu C, Zhang LJ. A deep-learning model for intracranial aneurysm detection on CT angiography images in China: a stepwise, multicentre, early-stage clinical validation study. Lancet Digit Health 2024; 6:e261-e271. [PMID: 38519154 DOI: 10.1016/s2589-7500(23)00268-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Revised: 10/23/2023] [Accepted: 12/29/2023] [Indexed: 03/24/2024]
Abstract
BACKGROUND Artificial intelligence (AI) models in real-world implementation are scarce. Our study aimed to develop a CT angiography (CTA)-based AI model for intracranial aneurysm detection, assess how it helps clinicians improve diagnostic performance, and validate its application in real-world clinical implementation. METHODS We developed a deep-learning model using 16 546 head and neck CTA examination images from 14 517 patients at eight Chinese hospitals. Using an adapted, stepwise implementation and evaluation, 120 certified clinicians from 15 geographically different hospitals were recruited. Initially, the AI model was externally validated with images of 900 digital subtraction angiography-verified CTA cases (examinations) and compared with the performance of 24 clinicians who each viewed 300 of these cases (stage 1). Next, as a further external validation a multi-reader multi-case study enrolled 48 clinicians to individually review 298 digital subtraction angiography-verified CTA cases (stage 2). The clinicians reviewed each CTA examination twice (ie, with and without the AI model), separated by a 4-week washout period. Then, a randomised open-label comparison study enrolled 48 clinicians to assess the acceptance and performance of this AI model (stage 3). Finally, the model was prospectively deployed and validated in 1562 real-world clinical CTA cases. FINDINGS The AI model in the internal dataset achieved a patient-level diagnostic sensitivity of 0·957 (95% CI 0·939-0·971) and a higher patient-level diagnostic sensitivity than clinicians (0·943 [0·921-0·961] vs 0·658 [0·644-0·672]; p<0·0001) in the external dataset. In the multi-reader multi-case study, the AI-assisted strategy improved clinicians' diagnostic performance both on a per-patient basis (the area under the receiver operating characteristic curves [AUCs]; 0·795 [0·761-0·830] without AI vs 0·878 [0·850-0·906] with AI; p<0·0001) and a per-aneurysm basis (the area under the weighted alternative free-response receiver operating characteristic curves; 0·765 [0·732-0·799] vs 0·865 [0·839-0·891]; p<0·0001). Reading time decreased with the aid of the AI model (87·5 s vs 82·7 s, p<0·0001). In the randomised open-label comparison study, clinicians in the AI-assisted group had a high acceptance of the AI model (92·6% adoption rate), and a higher AUC when compared with the control group (0·858 [95% CI 0·850-0·866] vs 0·789 [0·780-0·799]; p<0·0001). In the prospective study, the AI model had a 0·51% (8/1570) error rate due to poor-quality CTA images and recognition failure. The model had a high negative predictive value of 0·998 (0·994-1·000) and significantly improved the diagnostic performance of clinicians; AUC improved from 0·787 (95% CI 0·766-0·808) to 0·909 (0·894-0·923; p<0·0001) and patient-level sensitivity improved from 0·590 (0·511-0·666) to 0·825 (0·759-0·880; p<0·0001). INTERPRETATION This AI model demonstrated strong clinical potential for intracranial aneurysm detection with improved clinician diagnostic performance, high acceptance, and practical implementation in real-world clinical cases. FUNDING National Natural Science Foundation of China. TRANSLATION For the Chinese translation of the abstract see Supplementary Materials section.
Collapse
Affiliation(s)
- Bin Hu
- Department of Radiology, Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, China
| | - Zhao Shi
- Department of Radiology, Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, China
| | - Li Lu
- Department of Radiology, the Affiliated Hospital of Xuzhou Medical University, Xuzhou, Jiangsu, China
| | - Zhongchang Miao
- Department of Medical Imaging, the First People's Hospital of Lianyungang, Lianyungang, Jiangsu, China
| | - Hao Wang
- Deepwise Artificial Intelligence (AI) Lab, Deepwise, Beijing, China
| | - Zhen Zhou
- Deepwise Artificial Intelligence (AI) Lab, Deepwise, Beijing, China
| | - Fandong Zhang
- Deepwise Artificial Intelligence (AI) Lab, Deepwise, Beijing, China
| | - Rongpin Wang
- Department of Medical Imaging, Guizhou Province People's Hospital, Guiyang, Guizhou, China
| | - Xiao Luo
- Department of Radiology, Ma'anshan People's Hospital, Ma'anshan, Anhui, China
| | - Feng Xu
- Department of Medical Imaging, the Affiliated Suqian First People's Hospital of Nanjing Medical University, Suqian, Jiangsu, China
| | - Sheng Li
- Department of Radiology, People's Hospital, Hubei University of Medicine, Shiyan, Hubei, China
| | - Xiangming Fang
- Department of Medical Imaging, the Affiliated Wuxi People's Hospital of Nanjing Medical University, Wuxi, Jiangsu, China
| | - Xiaodong Wang
- Department of Radiology, General Hospital of Ningxia Medical University, Yinchuan, Ningxia, China
| | - Ge Yan
- Department of Medical Imaging, the First Affiliated Hospital of Xi'an Jiaotong University, Xi'an, Shaanxi, China
| | - Fajin Lv
- Department of Radiology, the First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Meng Zhang
- Department of Radiology, People's Hospital of Sanya, Sanya, Hainan, China
| | - Qiu Sun
- Department of Radiology, Lanzhou University Second Hospital, Lanzhou, Gansu, China
| | - Guangbin Cui
- Department of Radiology, Tangdu Hospital, Air Force Medical University (Fourth Military Medical University), Xi'an, Shaanxi, China
| | - Yubao Liu
- Medical Imaging Center, Shenzhen Hospital of Southern Medical University, Shenzhen, Guangdong, China
| | - Shu Zhang
- Deepwise Artificial Intelligence (AI) Lab, Deepwise, Beijing, China
| | - Chengwei Pan
- Institute of Artificial Intelligence, Beihang University, Beijing, China
| | - Zhibo Hou
- Department of Radiology, Medical Imaging Center, Peking University Shougang Hospital, Beijing, China
| | - Huiying Liang
- Medical Big Data Center, Guangdong Provincial People's Hospital, Guangzhou Guangdong, China
| | - Yuning Pan
- Department of Radiology, Ningbo First Hospital, Ningbo, Zhejiang, China
| | - Xiaoxia Chen
- Department of Radiology, Third Center Medical Center, Chinese PLA General Hospital, Beijing, China
| | - Xiaorong Li
- Department of Radiology, General Hospital of Southern Theater Command, PLA, Guangzhou, Guangdong, China
| | - Fei Zhou
- Department of Radiology, Central Hospital of Jilin City, Jilin, China
| | - U Joseph Schoepf
- Division of Cardiovascular Imaging, Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC, USA
| | - Akos Varga-Szemes
- Division of Cardiovascular Imaging, Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC, USA
| | - W Garrison Moore
- Division of Cardiovascular Imaging, Department of Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC, USA
| | - Yizhou Yu
- Department of Computer Science, The University of Hong Kong, Hong Kong Special Administrative Region, China
| | - Chunfeng Hu
- Department of Radiology, the Affiliated Hospital of Xuzhou Medical University, Xuzhou, Jiangsu, China
| | - Long Jiang Zhang
- Department of Radiology, Jinling Hospital, Affiliated Hospital of Medical School, Nanjing University, Nanjing, China.
| |
Collapse
|
7
|
Yim D, Khuntia J, Parameswaran V, Meyers A. Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review. JMIR Med Inform 2024; 12:e52073. [PMID: 38506918 PMCID: PMC10993141 DOI: 10.2196/52073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Revised: 10/12/2023] [Accepted: 01/30/2024] [Indexed: 03/21/2024] Open
Abstract
BACKGROUND Generative artificial intelligence tools and applications (GenAI) are being increasingly used in health care. Physicians, specialists, and other providers have started primarily using GenAI as an aid or tool to gather knowledge, provide information, train, or generate suggestive dialogue between physicians and patients or between physicians and patients' families or friends. However, unless the use of GenAI is oriented to be helpful in clinical service encounters that can improve the accuracy of diagnosis, treatment, and patient outcomes, the expected potential will not be achieved. As adoption continues, it is essential to validate the effectiveness of the infusion of GenAI as an intelligent technology in service encounters to understand the gap in actual clinical service use of GenAI. OBJECTIVE This study synthesizes preliminary evidence on how GenAI assists, guides, and automates clinical service rendering and encounters in health care The review scope was limited to articles published in peer-reviewed medical journals. METHODS We screened and selected 0.38% (161/42,459) of articles published between January 1, 2020, and May 31, 2023, identified from PubMed. We followed the protocols outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to select highly relevant studies with at least 1 element on clinical use, evaluation, and validation to provide evidence of GenAI use in clinical services. The articles were classified based on their relevance to clinical service functions or activities using the descriptive and analytical information presented in the articles. RESULTS Of 161 articles, 141 (87.6%) reported using GenAI to assist services through knowledge access, collation, and filtering. GenAI was used for disease detection (19/161, 11.8%), diagnosis (14/161, 8.7%), and screening processes (12/161, 7.5%) in the areas of radiology (17/161, 10.6%), cardiology (12/161, 7.5%), gastrointestinal medicine (4/161, 2.5%), and diabetes (6/161, 3.7%). The literature synthesis in this study suggests that GenAI is mainly used for diagnostic processes, improvement of diagnosis accuracy, and screening and diagnostic purposes using knowledge access. Although this solves the problem of knowledge access and may improve diagnostic accuracy, it is oriented toward higher value creation in health care. CONCLUSIONS GenAI informs rather than assisting or automating clinical service functions in health care. There is potential in clinical service, but it has yet to be actualized for GenAI. More clinical service-level evidence that GenAI is used to streamline some functions or provides more automated help than only information retrieval is needed. To transform health care as purported, more studies related to GenAI applications must automate and guide human-performed services and keep up with the optimism that forward-thinking health care organizations will take advantage of GenAI.
Collapse
Affiliation(s)
- Dobin Yim
- Loyola University, Maryland, MD, United States
| | - Jiban Khuntia
- University of Colorado Denver, Denver, CO, United States
| | | | - Arlen Meyers
- University of Colorado Denver, Denver, CO, United States
| |
Collapse
|
8
|
Bharadwaj P, Nicola L, Breau-Brunel M, Sensini F, Tanova-Yotova N, Atanasov P, Lobig F, Blankenburg M. Unlocking the Value: Quantifying the Return on Investment of Hospital Artificial Intelligence. J Am Coll Radiol 2024:S1546-1440(24)00292-8. [PMID: 38499053 DOI: 10.1016/j.jacr.2024.02.034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Revised: 02/23/2024] [Accepted: 02/28/2024] [Indexed: 03/20/2024]
Abstract
PURPOSE A comprehensive return on investment (ROI) calculator was developed to evaluate the monetary and nonmonetary benefits of an artificial intelligence (AI)-powered radiology diagnostic imaging platform to inform decision makers interested in adopting AI. METHODS A calculator was constructed to calculate comparative costs, estimated revenues, and quantify the clinical value of using an AI platform compared with no use of AI in radiology workflows of a US hospital over a 5-year time horizon. Parameters were determined on the basis of expert interviews and a literature review. Scenario and deterministic sensitivity analyses were conducted to evaluate calculator drivers. RESULTS In the calculator, the introduction of an AI platform into the hospital radiology workflow resulted in labor time reductions and delivery of an ROI of 451% over a 5-year period. The ROI was increased to 791% when radiologist time savings were considered. Time savings for radiologists included more than 15 8-hour working days of waiting time, 78 days in triage time, 10 days in reading time, and 41 days in reporting time. Using the platform also provided revenue benefits for the hospital in bringing in patients for clinically beneficial follow-up scans, hospitalizations, and treatment procedures. Results were sensitive to the time horizon, health center setting, and number of scans performed. Among those, the most influential outcome was the number of additional necessary treatments performed because of AI identification of patients. CONCLUSIONS The authors demonstrate a substantial 5-year ROI of implementing an AI platform in a stroke management-accredited hospital. The ROI calculator may be useful for decision makers evaluating AI-powered radiology platforms.
Collapse
Affiliation(s)
| | - Lauren Nicola
- CEO/Partner, Triad Radiology Associates; Chair, Ultrasound Commission, ACR; Chair, Reimbursement Committee, ACR
| | | | | | | | - Petar Atanasov
- Principal Consultant, Amaris Consulting, London, United Kingdom
| | | | | |
Collapse
|
9
|
Topff L, Steltenpool S, Ranschaert ER, Ramanauskas N, Menezes R, Visser JJ, Beets-Tan RGH, Hartkamp NS. Artificial intelligence-assisted double reading of chest radiographs to detect clinically relevant missed findings: a two-centre evaluation. Eur Radiol 2024:10.1007/s00330-024-10676-w. [PMID: 38466390 DOI: 10.1007/s00330-024-10676-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 01/21/2024] [Accepted: 02/01/2024] [Indexed: 03/13/2024]
Abstract
OBJECTIVES To evaluate an artificial intelligence (AI)-assisted double reading system for detecting clinically relevant missed findings on routinely reported chest radiographs. METHODS A retrospective study was performed in two institutions, a secondary care hospital and tertiary referral oncology centre. Commercially available AI software performed a comparative analysis of chest radiographs and radiologists' authorised reports using a deep learning and natural language processing algorithm, respectively. The AI-detected discrepant findings between images and reports were assessed for clinical relevance by an external radiologist, as part of the commercial service provided by the AI vendor. The selected missed findings were subsequently returned to the institution's radiologist for final review. RESULTS In total, 25,104 chest radiographs of 21,039 patients (mean age 61.1 years ± 16.2 [SD]; 10,436 men) were included. The AI software detected discrepancies between imaging and reports in 21.1% (5289 of 25,104). After review by the external radiologist, 0.9% (47 of 5289) of cases were deemed to contain clinically relevant missed findings. The institution's radiologists confirmed 35 of 47 missed findings (74.5%) as clinically relevant (0.1% of all cases). Missed findings consisted of lung nodules (71.4%, 25 of 35), pneumothoraces (17.1%, 6 of 35) and consolidations (11.4%, 4 of 35). CONCLUSION The AI-assisted double reading system was able to identify missed findings on chest radiographs after report authorisation. The approach required an external radiologist to review the AI-detected discrepancies. The number of clinically relevant missed findings by radiologists was very low. CLINICAL RELEVANCE STATEMENT The AI-assisted double reader workflow was shown to detect diagnostic errors and could be applied as a quality assurance tool. Although clinically relevant missed findings were rare, there is potential impact given the common use of chest radiography. KEY POINTS • A commercially available double reading system supported by artificial intelligence was evaluated to detect reporting errors in chest radiographs (n=25,104) from two institutions. • Clinically relevant missed findings were found in 0.1% of chest radiographs and consisted of unreported lung nodules, pneumothoraces and consolidations. • Applying AI software as a secondary reader after report authorisation can assist in reducing diagnostic errors without interrupting the radiologist's reading workflow. However, the number of AI-detected discrepancies was considerable and required review by a radiologist to assess their relevance.
Collapse
Affiliation(s)
- Laurens Topff
- Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands.
- GROW School for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands.
| | - Sanne Steltenpool
- Department of Radiology and Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
- Department of Radiology, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands
| | - Erik R Ranschaert
- Department of Radiology, St. Nikolaus Hospital, Eupen, Belgium
- Ghent University, Ghent, Belgium
| | - Naglis Ramanauskas
- Oxipit UAB, Vilnius, Lithuania
- Department of Radiology, Nuclear Medicine and Medical Physics, Institute of Biomedical Sciences, Faculty of Medicine, Vilnius University, Vilnius, Lithuania
| | - Renee Menezes
- Biostatistics Centre, Department of Psychosocial Research and Epidemiology, Netherlands Cancer Institute, Amsterdam, The Netherlands
| | - Jacob J Visser
- Department of Radiology and Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Regina G H Beets-Tan
- Department of Radiology, Netherlands Cancer Institute, Amsterdam, The Netherlands
- GROW School for Oncology and Reproduction, Maastricht University, Maastricht, The Netherlands
| | - Nolan S Hartkamp
- Department of Radiology, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands
| |
Collapse
|
10
|
Hwang EJ, Jeong WG, David PM, Arentz M, Ruhwald M, Yoon SH. AI for Detection of Tuberculosis: Implications for Global Health. Radiol Artif Intell 2024; 6:e230327. [PMID: 38197795 PMCID: PMC10982823 DOI: 10.1148/ryai.230327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 12/03/2023] [Accepted: 12/18/2023] [Indexed: 01/11/2024]
Abstract
Tuberculosis, which primarily affects developing countries, remains a significant global health concern. Since the 2010s, the role of chest radiography has expanded in tuberculosis triage and screening beyond its traditional complementary role in the diagnosis of tuberculosis. Computer-aided diagnosis (CAD) systems for tuberculosis detection on chest radiographs have recently made substantial progress in diagnostic performance, thanks to deep learning technologies. The current performance of CAD systems for tuberculosis has approximated that of human experts, presenting a potential solution to the shortage of human readers to interpret chest radiographs in low- or middle-income, high-tuberculosis-burden countries. This article provides a critical appraisal of developmental process reporting in extant CAD software for tuberculosis, based on the Checklist for Artificial Intelligence in Medical Imaging. It also explores several considerations to scale up CAD solutions, encompassing manufacturer-independent CAD validation, economic and political aspects, and ethical concerns, as well as the potential for broadening radiography-based diagnosis to other nontuberculosis diseases. Collectively, CAD for tuberculosis will emerge as a representative deep learning application, catalyzing advances in global health and health equity. Keywords: Computer-aided Diagnosis (CAD), Conventional Radiography, Thorax, Lung, Machine Learning Supplemental material is available for this article. © RSNA, 2024.
Collapse
Affiliation(s)
- Eui Jin Hwang
- From the Department of Radiology, Seoul National University Hospital
and Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu,
Seoul 03080, Korea (E.J.H., S.H.Y.); Department of Radiology, Chonnam National
University Hwasun Hospital, Hwasun, Korea (W.G.J.); Faculty of Pharmacy,
University of Montréal, Montréal, Canada (P.M.D.);
OBVIA–Observatoire sur les Impacts Sociétaux de l'IA et du
Numérique, Québec, Canada (P.M.D.); and FIND–The Global
Alliance for Diagnostics, Geneva, Switzerland (M.A., M.R.)
| | - Won Gi Jeong
- From the Department of Radiology, Seoul National University Hospital
and Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu,
Seoul 03080, Korea (E.J.H., S.H.Y.); Department of Radiology, Chonnam National
University Hwasun Hospital, Hwasun, Korea (W.G.J.); Faculty of Pharmacy,
University of Montréal, Montréal, Canada (P.M.D.);
OBVIA–Observatoire sur les Impacts Sociétaux de l'IA et du
Numérique, Québec, Canada (P.M.D.); and FIND–The Global
Alliance for Diagnostics, Geneva, Switzerland (M.A., M.R.)
| | - Pierre-Marie David
- From the Department of Radiology, Seoul National University Hospital
and Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu,
Seoul 03080, Korea (E.J.H., S.H.Y.); Department of Radiology, Chonnam National
University Hwasun Hospital, Hwasun, Korea (W.G.J.); Faculty of Pharmacy,
University of Montréal, Montréal, Canada (P.M.D.);
OBVIA–Observatoire sur les Impacts Sociétaux de l'IA et du
Numérique, Québec, Canada (P.M.D.); and FIND–The Global
Alliance for Diagnostics, Geneva, Switzerland (M.A., M.R.)
| | - Matthew Arentz
- From the Department of Radiology, Seoul National University Hospital
and Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu,
Seoul 03080, Korea (E.J.H., S.H.Y.); Department of Radiology, Chonnam National
University Hwasun Hospital, Hwasun, Korea (W.G.J.); Faculty of Pharmacy,
University of Montréal, Montréal, Canada (P.M.D.);
OBVIA–Observatoire sur les Impacts Sociétaux de l'IA et du
Numérique, Québec, Canada (P.M.D.); and FIND–The Global
Alliance for Diagnostics, Geneva, Switzerland (M.A., M.R.)
| | - Morten Ruhwald
- From the Department of Radiology, Seoul National University Hospital
and Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu,
Seoul 03080, Korea (E.J.H., S.H.Y.); Department of Radiology, Chonnam National
University Hwasun Hospital, Hwasun, Korea (W.G.J.); Faculty of Pharmacy,
University of Montréal, Montréal, Canada (P.M.D.);
OBVIA–Observatoire sur les Impacts Sociétaux de l'IA et du
Numérique, Québec, Canada (P.M.D.); and FIND–The Global
Alliance for Diagnostics, Geneva, Switzerland (M.A., M.R.)
| | - Soon Ho Yoon
- From the Department of Radiology, Seoul National University Hospital
and Seoul National University College of Medicine, 101 Daehak-ro, Jongno-gu,
Seoul 03080, Korea (E.J.H., S.H.Y.); Department of Radiology, Chonnam National
University Hwasun Hospital, Hwasun, Korea (W.G.J.); Faculty of Pharmacy,
University of Montréal, Montréal, Canada (P.M.D.);
OBVIA–Observatoire sur les Impacts Sociétaux de l'IA et du
Numérique, Québec, Canada (P.M.D.); and FIND–The Global
Alliance for Diagnostics, Geneva, Switzerland (M.A., M.R.)
| |
Collapse
|
11
|
Doherty G, McLaughlin L, Hughes C, McConnell J, Bond R, McFadden S. A scoping review of educational programmes on artificial intelligence (AI) available to medical imaging staff. Radiography (Lond) 2024; 30:474-482. [PMID: 38217933 DOI: 10.1016/j.radi.2023.12.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Revised: 12/29/2023] [Accepted: 12/30/2023] [Indexed: 01/15/2024]
Abstract
INTRODUCTION Medical imaging is arguably the most technologically advanced field in healthcare, encompassing a range of technologies which continually evolve as computing power and human knowledge expand. Artificial Intelligence (AI) is the next frontier which medical imaging is pioneering. The rapid development and implementation of AI has the potential to revolutionise healthcare, however, to do so, staff must be competent and confident in its application, hence AI readiness is an important precursor to AI adoption. Research to ascertain the best way to deliver this AI-enabled healthcare training is in its infancy. The aim of this scoping review is to compare existing studies which investigate and evaluate the efficacy of AI educational interventions for medical imaging staff. METHODS Following the creation of a search strategy and keyword searches, screening was conducted to determine study eligibility. This consisted of a title and abstract scan, then subsequently a full-text review. Articles were included if they were empirical studies wherein an educational intervention on AI for medical imaging staff was created, delivered, and evaluated. RESULTS Of the initial 1309 records returned, n = 5 (∼0.4 %) of studies met the eligibility criteria of the review. The curricula and delivery in each of the five studies shared similar aims and a 'flipped classroom' delivery was the most utilised method. However, the depth of content covered in the curricula of each varied and measured outcomes differed greatly. CONCLUSION The findings of this review will provide insights into the evaluation of existing AI educational interventions, which will be valuable when planning AI education for healthcare staff. IMPLICATIONS FOR PRACTICE This review highlights the need for standardised and comprehensive AI training programs for imaging staff.
Collapse
Affiliation(s)
- G Doherty
- Ulster University, School of Health Sciences, Faculty of Life and Health Sciences, Shore Road, Newtownabbey, Northern Ireland, United Kingdom.
| | - L McLaughlin
- Ulster University, School of Health Sciences, Faculty of Life and Health Sciences, Shore Road, Newtownabbey, Northern Ireland, United Kingdom
| | - C Hughes
- Ulster University, School of Health Sciences, Faculty of Life and Health Sciences, Shore Road, Newtownabbey, Northern Ireland, United Kingdom
| | - J McConnell
- Leeds Teaching Hospitals NHS Trust, United Kingdom
| | - R Bond
- Ulster University, School of Computing, Faculty of Computing, Engineering and the Built Environment, Shore Road, Newtownabbey, Northern Ireland, United Kingdom
| | - S McFadden
- Ulster University, School of Health Sciences, Faculty of Life and Health Sciences, Shore Road, Newtownabbey, Northern Ireland, United Kingdom
| |
Collapse
|
12
|
Yu F, Moehring A, Banerjee O, Salz T, Agarwal N, Rajpurkar P. Heterogeneity and predictors of the effects of AI assistance on radiologists. Nat Med 2024; 30:837-849. [PMID: 38504016 PMCID: PMC10957478 DOI: 10.1038/s41591-024-02850-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Accepted: 02/01/2024] [Indexed: 03/21/2024]
Abstract
The integration of artificial intelligence (AI) in medical image interpretation requires effective collaboration between clinicians and AI algorithms. Although previous studies demonstrated the potential of AI assistance in improving overall clinician performance, the individual impact on clinicians remains unclear. This large-scale study examined the heterogeneous effects of AI assistance on 140 radiologists across 15 chest X-ray diagnostic tasks and identified predictors of these effects. Surprisingly, conventional experience-based factors, such as years of experience, subspecialty and familiarity with AI tools, fail to reliably predict the impact of AI assistance. Additionally, lower-performing radiologists do not consistently benefit more from AI assistance, challenging prevailing assumptions. Instead, we found that the occurrence of AI errors strongly influences treatment outcomes, with inaccurate AI predictions adversely affecting radiologist performance on the aggregate of all pathologies and on half of the individual pathologies investigated. Our findings highlight the importance of personalized approaches to clinician-AI collaboration and the importance of accurate AI models. By understanding the factors that shape the effectiveness of AI assistance, this study provides valuable insights for targeted implementation of AI, enabling maximum benefits for individual clinicians in clinical practice.
Collapse
Affiliation(s)
- Feiyang Yu
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Alex Moehring
- Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Oishi Banerjee
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Tobias Salz
- Department of Economics, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Nikhil Agarwal
- Department of Economics, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Pranav Rajpurkar
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
13
|
Lee S, Park JS, Woo H, Yoo YK, Lee D, Chung S, Yoon DS, Lee KB, Lee JH. Rapid deep learning-assisted predictive diagnostics for point-of-care testing. Nat Commun 2024; 15:1695. [PMID: 38402240 PMCID: PMC10894262 DOI: 10.1038/s41467-024-46069-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Accepted: 02/12/2024] [Indexed: 02/26/2024] Open
Abstract
Prominent techniques such as real-time polymerase chain reaction (RT-PCR), enzyme-linked immunosorbent assay (ELISA), and rapid kits are currently being explored to both enhance sensitivity and reduce assay time for diagnostic tests. Existing commercial molecular methods typically take several hours, while immunoassays can range from several hours to tens of minutes. Rapid diagnostics are crucial in Point-of-Care Testing (POCT). We propose an approach that integrates a time-series deep learning architecture and AI-based verification, for the enhanced result analysis of lateral flow assays. This approach is applicable to both infectious diseases and non-infectious biomarkers. In blind tests using clinical samples, our method achieved diagnostic times as short as 2 minutes, exceeding the accuracy of human analysis at 15 minutes. Furthermore, our technique significantly reduces assay time to just 1-2 minutes in the POCT setting. This advancement has the potential to greatly enhance POCT diagnostics, enabling both healthcare professionals and non-experts to make rapid, accurate decisions.
Collapse
Affiliation(s)
- Seungmin Lee
- Department of Electrical Engineering, Kwangwoon University, 20 Kwangwoon-ro, Nowon, Seoul, 01897, Republic of Korea
- School of Biomedical Engineering, Korea University, 145 Anam-ro, Seongbuk, Seoul, 02841, Republic of Korea
| | - Jeong Soo Park
- Department of Electrical Engineering, Kwangwoon University, 20 Kwangwoon-ro, Nowon, Seoul, 01897, Republic of Korea
- School of Mechanical Engineering, Korea University, 145 Anam-ro, Seoungbuk-gu, Seoul, 02841, Republic of Korea
| | - Hyowon Woo
- Department of Electrical Engineering, Kwangwoon University, 20 Kwangwoon-ro, Nowon, Seoul, 01897, Republic of Korea
| | - Yong Kyoung Yoo
- Department of Electronic Engineering, Catholic Kwandong University, 24, Beomil-ro 579 beon-gil, Gangneung-si, Gangwon-do, 25601, Republic of Korea
| | - Dongho Lee
- CALTH Inc., Changeop-ro 54, Seongnam, Gyeonggi, 13449, Republic of Korea
| | - Seok Chung
- School of Mechanical Engineering, Korea University, 145 Anam-ro, Seoungbuk-gu, Seoul, 02841, Republic of Korea
| | - Dae Sung Yoon
- School of Biomedical Engineering, Korea University, 145 Anam-ro, Seongbuk, Seoul, 02841, Republic of Korea
- Interdisciplinary Program in Precision Public Health, Korea University, Seoul, 02841, Republic of Korea
- Astrion Inc, Seoul, 02841, Republic of Korea
| | - Ki-Baek Lee
- Department of Electrical Engineering, Kwangwoon University, 20 Kwangwoon-ro, Nowon, Seoul, 01897, Republic of Korea
| | - Jeong Hoon Lee
- Department of Electrical Engineering, Kwangwoon University, 20 Kwangwoon-ro, Nowon, Seoul, 01897, Republic of Korea.
- CALTH Inc., Changeop-ro 54, Seongnam, Gyeonggi, 13449, Republic of Korea.
| |
Collapse
|
14
|
Saab K, Tang S, Taha M, Lee-Messer C, Ré C, Rubin DL. Towards trustworthy seizure onset detection using workflow notes. NPJ Digit Med 2024; 7:42. [PMID: 38383884 PMCID: PMC10881468 DOI: 10.1038/s41746-024-01008-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 01/10/2024] [Indexed: 02/23/2024] Open
Abstract
A major barrier to deploying healthcare AI is trustworthiness. One form of trustworthiness is a model's robustness across subgroups: while models may exhibit expert-level performance on aggregate metrics, they often rely on non-causal features, leading to errors in hidden subgroups. To take a step closer towards trustworthy seizure onset detection from EEG, we propose to leverage annotations that are produced by healthcare personnel in routine clinical workflows-which we refer to as workflow notes-that include multiple event descriptions beyond seizures. Using workflow notes, we first show that by scaling training data to 68,920 EEG hours, seizure onset detection performance significantly improves by 12.3 AUROC (Area Under the Receiver Operating Characteristic) points compared to relying on smaller training sets with gold-standard labels. Second, we reveal that our binary seizure onset detection model underperforms on clinically relevant subgroups (e.g., up to a margin of 6.5 AUROC points between pediatrics and adults), while having significantly higher FPRs (False Positive Rates) on EEG clips showing non-epileptiform abnormalities (+19 FPR points). To improve model robustness to hidden subgroups, we train a multilabel model that classifies 26 attributes other than seizures (e.g., spikes and movement artifacts) and significantly improve overall performance (+5.9 AUROC points) while greatly improving performance among subgroups (up to +8.3 AUROC points) and decreasing false positives on non-epileptiform abnormalities (by 8 FPR points). Finally, we find that our multilabel model improves clinical utility (false positives per 24 EEG hours) by a factor of 2×.
Collapse
Affiliation(s)
- Khaled Saab
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
| | - Siyi Tang
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Mohamed Taha
- Department of Neurology, Stanford University, Stanford, CA, USA
| | | | - Christopher Ré
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Daniel L Rubin
- Department of Biomedical Data Science, Radiology, and Medicine, Stanford University, Stanford, CA, USA.
| |
Collapse
|
15
|
Hofmeister J, Garin N, Montet X, Scheffler M, Platon A, Poletti PA, Stirnemann J, Debray MP, Claessens YE, Duval X, Prendki V. Validating the accuracy of deep learning for the diagnosis of pneumonia on chest x-ray against a robust multimodal reference diagnosis: a post hoc analysis of two prospective studies. Eur Radiol Exp 2024; 8:20. [PMID: 38302850 PMCID: PMC10834924 DOI: 10.1186/s41747-023-00416-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 11/28/2023] [Indexed: 02/03/2024] Open
Abstract
BACKGROUND Artificial intelligence (AI) seems promising in diagnosing pneumonia on chest x-rays (CXR), but deep learning (DL) algorithms have primarily been compared with radiologists, whose diagnosis can be not completely accurate. Therefore, we evaluated the accuracy of DL in diagnosing pneumonia on CXR using a more robust reference diagnosis. METHODS We trained a DL convolutional neural network model to diagnose pneumonia and evaluated its accuracy in two prospective pneumonia cohorts including 430 patients, for whom the reference diagnosis was determined a posteriori by a multidisciplinary expert panel using multimodal data. The performance of the DL model was compared with that of senior radiologists and emergency physicians reviewing CXRs and that of radiologists reviewing computed tomography (CT) performed concomitantly. RESULTS Radiologists and DL showed a similar accuracy on CXR for both cohorts (p ≥ 0.269): cohort 1, radiologist 1 75.5% (95% confidence interval 69.1-80.9), radiologist 2 71.0% (64.4-76.8), DL 71.0% (64.4-76.8); cohort 2, radiologist 70.9% (64.7-76.4), DL 72.6% (66.5-78.0). The accuracy of radiologists and DL was significantly higher (p ≤ 0.022) than that of emergency physicians (cohort 1 64.0% [57.1-70.3], cohort 2 63.0% [55.6-69.0]). Accuracy was significantly higher for CT (cohort 1 79.0% [72.8-84.1], cohort 2 89.6% [84.9-92.9]) than for CXR readers including radiologists, clinicians, and DL (all p-values < 0.001). CONCLUSIONS When compared with a robust reference diagnosis, the performance of AI models to identify pneumonia on CXRs was inferior than previously reported but similar to that of radiologists and better than that of emergency physicians. RELEVANCE STATEMENT The clinical relevance of AI models for pneumonia diagnosis may have been overestimated. AI models should be benchmarked against robust reference multimodal diagnosis to avoid overestimating its performance. TRIAL REGISTRATION NCT02467192 , and NCT01574066 . KEY POINT • We evaluated an openly-access convolutional neural network (CNN) model to diagnose pneumonia on CXRs. • CNN was validated against a strong multimodal reference diagnosis. • In our study, the CNN performance (area under the receiver operating characteristics curve 0.74) was lower than that previously reported when validated against radiologists' diagnosis (0.99 in a recent meta-analysis). • The CNN performance was significantly higher than emergency physicians' (p ≤ 0.022) and comparable to that of board-certified radiologists (p ≥ 0.269).
Collapse
Affiliation(s)
- Jeremy Hofmeister
- Department of Diagnostics, Geneva University Hospitals, Geneva, Switzerland.
| | - Nicolas Garin
- Division of Internal Medicine, Riviera Chablais Hospital, Rennaz, Switzerland
- Department of Medicine, Geneva University Hospitals, Geneva, Switzerland
| | - Xavier Montet
- Department of Diagnostics, Geneva University Hospitals, Geneva, Switzerland
| | - Max Scheffler
- Department of Diagnostics, Geneva University Hospitals, Geneva, Switzerland
| | - Alexandra Platon
- Department of Diagnostics, Geneva University Hospitals, Geneva, Switzerland
| | | | - Jérôme Stirnemann
- Department of Medicine, Geneva University Hospitals, Geneva, Switzerland
| | - Marie-Pierre Debray
- Department of Radiology, APHP, Hôpital Bichat, University Paris Cité, Inserm UMR1152, Paris, France
| | - Yann-Erick Claessens
- Department of Emergency Medicine, Centre Hospitalier Princesse Grace, La Colle, Principality of Monaco, Monaco
| | - Xavier Duval
- Department of Epidemiology and Clinical ResearchInserm CIC 1425UMR 1138, APHP, Hôpital BichatUniversity Paris CitéIAME, Paris, France
| | - Virginie Prendki
- Department of Rehabilitation and Geriatrics, Geneva University Hospitals, Geneva, Switzerland.
- Division of Infectious Disease, Geneva University Hospital, 4 Rue Gabrielle Perret-Gentil, 1211, Geneva 14, Switzerland.
| |
Collapse
|
16
|
Häggström I, Leithner D, Alvén J, Campanella G, Abusamra M, Zhang H, Chhabra S, Beer L, Haug A, Salles G, Raderer M, Staber PB, Becker A, Hricak H, Fuchs TJ, Schöder H, Mayerhoefer ME. Deep learning for [ 18F]fluorodeoxyglucose-PET-CT classification in patients with lymphoma: a dual-centre retrospective analysis. Lancet Digit Health 2024; 6:e114-e125. [PMID: 38135556 DOI: 10.1016/s2589-7500(23)00203-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2023] [Revised: 08/29/2023] [Accepted: 09/26/2023] [Indexed: 12/24/2023]
Abstract
BACKGROUND The rising global cancer burden has led to an increasing demand for imaging tests such as [18F]fluorodeoxyglucose ([18F]FDG)-PET-CT. To aid imaging specialists in dealing with high scan volumes, we aimed to train a deep learning artificial intelligence algorithm to classify [18F]FDG-PET-CT scans of patients with lymphoma with or without hypermetabolic tumour sites. METHODS In this retrospective analysis we collected 16 583 [18F]FDG-PET-CTs of 5072 patients with lymphoma who had undergone PET-CT before or after treatment at the Memorial Sloa Kettering Cancer Center, New York, NY, USA. Using maximum intensity projection (MIP), three dimensional (3D) PET, and 3D CT data, our ResNet34-based deep learning model (Lymphoma Artificial Reader System [LARS]) for [18F]FDG-PET-CT binary classification (Deauville 1-3 vs 4-5), was trained on 80% of the dataset, and tested on 20% of this dataset. For external testing, 1000 [18F]FDG-PET-CTs were obtained from a second centre (Medical University of Vienna, Vienna, Austria). Seven model variants were evaluated, including MIP-based LARS-avg (optimised for accuracy) and LARS-max (optimised for sensitivity), and 3D PET-CT-based LARS-ptct. Following expert curation, areas under the curve (AUCs), accuracies, sensitivities, and specificities were calculated. FINDINGS In the internal test cohort (3325 PET-CTs, 1012 patients), LARS-avg achieved an AUC of 0·949 (95% CI 0·942-0·956), accuracy of 0·890 (0·879-0·901), sensitivity of 0·868 (0·851-0·885), and specificity of 0·913 (0·899-0·925); LARS-max achieved an AUC of 0·949 (0·942-0·956), accuracy of 0·868 (0·858-0·879), sensitivity of 0·909 (0·896-0·924), and specificity of 0·826 (0·808-0·843); and LARS-ptct achieved an AUC of 0·939 (0·930-0·948), accuracy of 0·875 (0·864-0·887), sensitivity of 0·836 (0·817-0·855), and specificity of 0·915 (0·901-0·927). In the external test cohort (1000 PET-CTs, 503 patients), LARS-avg achieved an AUC of 0·953 (0·938-0·966), accuracy of 0·907 (0·888-0·925), sensitivity of 0·874 (0·843-0·904), and specificity of 0·949 (0·921-0·960); LARS-max achieved an AUC of 0·952 (0·937-0·965), accuracy of 0·898 (0·878-0·916), sensitivity of 0·899 (0·871-0·926), and specificity of 0·897 (0·871-0·922); and LARS-ptct achieved an AUC of 0·932 (0·915-0·948), accuracy of 0·870 (0·850-0·891), sensitivity of 0·827 (0·793-0·863), and specificity of 0·913 (0·889-0·937). INTERPRETATION Deep learning accurately distinguishes between [18F]FDG-PET-CT scans of lymphoma patients with and without hypermetabolic tumour sites. Deep learning might therefore be potentially useful to rule out the presence of metabolically active disease in such patients, or serve as a second reader or decision support tool. FUNDING National Institutes of Health-National Cancer Institute Cancer Center Support Grant.
Collapse
Affiliation(s)
- Ida Häggström
- Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden; Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Doris Leithner
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Department of Radiology, NYU Langone Health, Grossman School of Medicine, New York, NY, USA
| | - Jennifer Alvén
- Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden
| | - Gabriele Campanella
- Hasso Plattner Institute for Digital Health, Mount Sinai Medical School, New York, NY, USA; Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Murad Abusamra
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Honglei Zhang
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Shalini Chhabra
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Lucian Beer
- Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria
| | - Alexander Haug
- Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria
| | - Gilles Salles
- Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Weill Cornell Medical College, Cornell University, New York, NY, USA
| | - Markus Raderer
- Department of Medicine I, Medical University of Vienna, Vienna, Austria
| | - Philipp B Staber
- Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Anton Becker
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Weill Cornell Medical College, Cornell University, New York, NY, USA; Department of Radiology, NYU Langone Health, Grossman School of Medicine, New York, NY, USA
| | - Hedvig Hricak
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Weill Cornell Medical College, Cornell University, New York, NY, USA
| | - Thomas J Fuchs
- Hasso Plattner Institute for Digital Health, Mount Sinai Medical School, New York, NY, USA; Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Heiko Schöder
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Weill Cornell Medical College, Cornell University, New York, NY, USA
| | - Marius E Mayerhoefer
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY, USA; Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria; Weill Cornell Medical College, Cornell University, New York, NY, USA; Department of Radiology, NYU Langone Health, Grossman School of Medicine, New York, NY, USA.
| |
Collapse
|
17
|
Buchlak QD, Tang CHM, Seah JCY, Johnson A, Holt X, Bottrell GM, Wardman JB, Samarasinghe G, Dos Santos Pinheiro L, Xia H, Ahmad HK, Pham H, Chiang JI, Ektas N, Milne MR, Chiu CHY, Hachey B, Ryan MK, Johnston BP, Esmaili N, Bennett C, Goldschlager T, Hall J, Vo DT, Oakden-Rayner L, Leveque JC, Farrokhi F, Abramson RG, Jones CM, Edelstein S, Brotchie P. Effects of a comprehensive brain computed tomography deep learning model on radiologist detection accuracy. Eur Radiol 2024; 34:810-822. [PMID: 37606663 PMCID: PMC10853361 DOI: 10.1007/s00330-023-10074-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Revised: 06/16/2023] [Accepted: 07/01/2023] [Indexed: 08/23/2023]
Abstract
OBJECTIVES Non-contrast computed tomography of the brain (NCCTB) is commonly used to detect intracranial pathology but is subject to interpretation errors. Machine learning can augment clinical decision-making and improve NCCTB scan interpretation. This retrospective detection accuracy study assessed the performance of radiologists assisted by a deep learning model and compared the standalone performance of the model with that of unassisted radiologists. METHODS A deep learning model was trained on 212,484 NCCTB scans drawn from a private radiology group in Australia. Scans from inpatient, outpatient, and emergency settings were included. Scan inclusion criteria were age ≥ 18 years and series slice thickness ≤ 1.5 mm. Thirty-two radiologists reviewed 2848 scans with and without the assistance of the deep learning system and rated their confidence in the presence of each finding using a 7-point scale. Differences in AUC and Matthews correlation coefficient (MCC) were calculated using a ground-truth gold standard. RESULTS The model demonstrated an average area under the receiver operating characteristic curve (AUC) of 0.93 across 144 NCCTB findings and significantly improved radiologist interpretation performance. Assisted and unassisted radiologists demonstrated an average AUC of 0.79 and 0.73 across 22 grouped parent findings and 0.72 and 0.68 across 189 child findings, respectively. When assisted by the model, radiologist AUC was significantly improved for 91 findings (158 findings were non-inferior), and reading time was significantly reduced. CONCLUSIONS The assistance of a comprehensive deep learning model significantly improved radiologist detection accuracy across a wide range of clinical findings and demonstrated the potential to improve NCCTB interpretation. CLINICAL RELEVANCE STATEMENT This study evaluated a comprehensive CT brain deep learning model, which performed strongly, improved the performance of radiologists, and reduced interpretation time. The model may reduce errors, improve efficiency, facilitate triage, and better enable the delivery of timely patient care. KEY POINTS • This study demonstrated that the use of a comprehensive deep learning system assisted radiologists in the detection of a wide range of abnormalities on non-contrast brain computed tomography scans. • The deep learning model demonstrated an average area under the receiver operating characteristic curve of 0.93 across 144 findings and significantly improved radiologist interpretation performance. • The assistance of the comprehensive deep learning model significantly reduced the time required for radiologists to interpret computed tomography scans of the brain.
Collapse
Affiliation(s)
- Quinlan D Buchlak
- Annalise.ai, Sydney, NSW, Australia.
- School of Medicine, University of Notre Dame Australia, Sydney, NSW, Australia.
- Department of Neurosurgery, Monash Health, Clayton, VIC, Australia.
| | | | - Jarrel C Y Seah
- Annalise.ai, Sydney, NSW, Australia
- Department of Radiology, Alfred Health, Melbourne, VIC, Australia
| | | | | | | | | | | | | | | | | | - Hung Pham
- Annalise.ai, Sydney, NSW, Australia
- Department of Radiology, University Medical Center, University of Medicine and Pharmacy, Ho Chi Minh City, Vietnam
| | - Jason I Chiang
- Annalise.ai, Sydney, NSW, Australia
- Department of General Practice, University of Melbourne, Melbourne, VIC, Australia
- Westmead Applied Research Centre, University of Sydney, Sydney, NSW, Australia
| | | | | | | | | | | | | | - Nazanin Esmaili
- School of Medicine, University of Notre Dame Australia, Sydney, NSW, Australia
- Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, Australia
| | - Christine Bennett
- School of Medicine, University of Notre Dame Australia, Sydney, NSW, Australia
| | - Tony Goldschlager
- Department of Neurosurgery, Monash Health, Clayton, VIC, Australia
- Department of Surgery, Monash University, Clayton, VIC, Australia
| | - Jonathan Hall
- Annalise.ai, Sydney, NSW, Australia
- Department of Radiology, St Vincent's Health Australia, Melbourne, VIC, Australia
- Department of Radiology, Austin Hospital, Melbourne, VIC, Australia
| | - Duc Tan Vo
- Department of Radiology, University Medical Center, University of Medicine and Pharmacy, Ho Chi Minh City, Vietnam
| | - Lauren Oakden-Rayner
- Australian Institute for Machine Learning, The University of Adelaide, Adelaide, SA, Australia
| | | | - Farrokh Farrokhi
- Center for Neurosciences and Spine, Virginia Mason Franciscan Health, Seattle, WA, USA
| | | | - Catherine M Jones
- Annalise.ai, Sydney, NSW, Australia
- I-MED Radiology Network, Brisbane, QLD, Australia
- School of Public and Preventive Health, Monash University, Clayton, VIC, Australia
- Department of Clinical Imaging Science, University of Sydney, Sydney, NSW, Australia
| | - Simon Edelstein
- Annalise.ai, Sydney, NSW, Australia
- I-MED Radiology Network, Brisbane, QLD, Australia
- Department of Radiology, Monash Health, Clayton, VIC, Australia
| | - Peter Brotchie
- Annalise.ai, Sydney, NSW, Australia
- Department of Radiology, St Vincent's Health Australia, Melbourne, VIC, Australia
| |
Collapse
|
18
|
Yoon SH, Park S, Jang S, Kim J, Lee KW, Lee W, Lee S, Yun G, Lee KH. Use of artificial intelligence in triaging of chest radiographs to reduce radiologists' workload. Eur Radiol 2024; 34:1094-1103. [PMID: 37615766 DOI: 10.1007/s00330-023-10124-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2022] [Revised: 06/22/2023] [Accepted: 07/03/2023] [Indexed: 08/25/2023]
Abstract
OBJECTIVES To evaluate whether deep learning-based detection algorithms (DLD)-based triaging can reduce outpatient chest radiograph interpretation workload while maintaining noninferior sensitivity. METHODS This retrospective study included patients who underwent initial chest radiography at the outpatient clinic between June 1 and June 30, 2017. Readers interpreted radiographs with/without a commercially available DLD that detects nine radiologic findings (atelectasis, calcification, cardiomegaly, consolidation, fibrosis, nodules, pneumothorax, pleural effusion, and pneumoperitoneum). The reading order was determined in a randomized, crossover manner. The radiographs were classified into negative and positive examinations. In a 50% worklist reduction scenario, radiographs were sorted in descending order of probability scores: the lower half was regarded as negative exams, while the remaining were read with DLD by radiologists. The primary analysis evaluated noninferiority in sensitivity between radiologists reading all radiographs and simulating a 50% worklist reduction, with the inferiority margin of 5%. The specificities were compared using McNemar's test. RESULTS The study included 1964 patients (median age [interquartile range], 55 years [40-67 years]). The sensitivity was 82.6% (195 of 236; 95% CI: 77.5%, 87.3%) when readers interpreted all chest radiographs without DLD and 83.5% (197 of 236; 95% CI: 78.8%, 88.1%) in the 50% worklist reduction scenario. The difference in sensitivity was 0.8% (95% CI: - 3.8%, 5.5%), establishing noninferiority of 50% worklist reduction (p = 0.01). The specificity increased from 86.7% (1498 of 1728) to 90.4% (1562 of 1728) (p < 0.001) with DLD-based triage. CONCLUSION Deep learning-based triaging may substantially reduce workload without lowering sensitivity while improving specificity. CLINICAL RELEVANCE STATEMENT Substantial workload reduction without lowering sensitivity was feasible using deep learning-based triaging of outpatient chest radiograph; however, the legal responsibility for incorrect diagnoses based on AI-standalone interpretation remains an issue that should be defined before clinical implementation. KEY POINTS • A 50% workload reduction simulation using deep learning-based detection algorithm maintained noninferior sensitivity while improving specificity. • The CT recommendation rate significantly decreased in the disease-negative patients, whereas it slightly increased in the disease-positive group without statistical significance. • In the exploratory analysis, the noninferiority of sensitivity was maintained until 70% of the workload was reduced; the difference in sensitivity was 0%.
Collapse
Affiliation(s)
- Sung Hyun Yoon
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Sunyoung Park
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Sowon Jang
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Junghoon Kim
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Kyung Won Lee
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea
| | - Woojoo Lee
- Department of Public Health Science, Graduate School of Public Health, Seoul National University, Seoul, Korea
| | - Seungjae Lee
- Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Korea
| | - Gabin Yun
- Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, PA, USA
| | - Kyung Hee Lee
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Korea.
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Bundang Hospital, Seongnam, Korea.
| |
Collapse
|
19
|
Sabottke C, Lee J, Chiang A, Spieler B, Mushtaq R. Text Report Analysis to Identify Opportunities for Optimizing Target Selection for Chest Radiograph Artificial Intelligence Models. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:402-411. [PMID: 38343239 DOI: 10.1007/s10278-023-00927-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/17/2023] [Revised: 09/06/2023] [Accepted: 09/06/2023] [Indexed: 03/02/2024]
Abstract
Our goal was to analyze radiology report text for chest radiographs (CXRs) to identify imaging findings that have the most impact on report length and complexity. Identifying these imaging findings can highlight opportunities for designing CXR AI systems which increase radiologist efficiency. We retrospectively analyzed text from 210,025 MIMIC-CXR reports and 168,949 reports from our local institution collected from 2019 to 2022. Fifty-nine categories of imaging finding keywords were extracted from reports using natural language processing (NLP), and their impact on report length was assessed using linear regression with and without LASSO regularization. Regression was also used to assess the impact of additional factors contributing to report length, such as the signing radiologist and use of terms of perception. For modeling CXR report word counts with regression, mean coefficient of determination, R2, was 0.469 ± 0.001 for local reports and 0.354 ± 0.002 for MIMIC-CXR when considering only imaging finding keyword features. Mean R2 was significantly less at 0.067 ± 0.001 for local reports and 0.086 ± 0.002 for MIMIC-CXR, when only considering use of terms of perception. For a combined model for the local report data accounting for the signing radiologist, imaging finding keywords, and terms of perception, the mean R2 was 0.570 ± 0.002. With LASSO, highest value coefficients pertained to endotracheal tubes and pleural drains for local data and masses, nodules, and cavitary and cystic lesions for MIMIC-CXR. Natural language processing and regression analysis of radiology report textual data can highlight imaging targets for AI models which offer opportunities to bolster radiologist efficiency.
Collapse
Affiliation(s)
- Carl Sabottke
- Department of Medical Imaging, University of Arizona College of Medicine, Tucson, AZ, USA.
| | - Jason Lee
- Department of Medical Imaging, University of Arizona College of Medicine, Tucson, AZ, USA
| | - Alan Chiang
- Department of Medical Imaging, University of Arizona College of Medicine, Tucson, AZ, USA
| | - Bradley Spieler
- Department of Radiology, Louisiana State University Health Sciences Center, New Orleans, LA, USA
| | - Raza Mushtaq
- Department of Neuroradiology, Barrow Neurological Institute, Phoenix, AZ, USA
| |
Collapse
|
20
|
van Leeuwen KG, Schalekamp S, Rutten MJCM, Huisman M, Schaefer-Prokop CM, de Rooij M, van Ginneken B, Maresch B, Geurts BHJ, van Dijke CF, Laupman-Koedam E, Hulleman EV, Verhoeff EL, Meys EMJ, Mohamed Hoesein FAA, Ter Brugge FM, van Hoorn F, van der Wel F, van den Berk IAH, Luyendijk JM, Meakin J, Habets J, Verbeke JIML, Nederend J, Meys KME, Deden LN, Langezaal LCM, Nasrollah M, Meij M, Boomsma MF, Vermeulen M, Vestering MM, Vijlbrief O, Algra P, Algra S, Bollen SM, Samson T, von Brucken Fock YHG. Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction. Radiology 2024; 310:e230981. [PMID: 38193833 DOI: 10.1148/radiol.230981] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2024]
Abstract
Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.
Collapse
Affiliation(s)
- Kicky G van Leeuwen
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Steven Schalekamp
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Matthieu J C M Rutten
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Merel Huisman
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Cornelia M Schaefer-Prokop
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Maarten de Rooij
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Bram van Ginneken
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Bas Maresch
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Bram H J Geurts
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Cornelius F van Dijke
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Emmeline Laupman-Koedam
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Enzo V Hulleman
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Eric L Verhoeff
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Evelyne M J Meys
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Firdaus A A Mohamed Hoesein
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Floor M Ter Brugge
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Francois van Hoorn
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Frank van der Wel
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Inge A H van den Berk
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Jacqueline M Luyendijk
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - James Meakin
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Jesse Habets
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Jonathan I M L Verbeke
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Joost Nederend
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Karlijn M E Meys
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Laura N Deden
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Lucianne C M Langezaal
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Mahtab Nasrollah
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Marleen Meij
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Martijn F Boomsma
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Matthijs Vermeulen
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Myrthe M Vestering
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Onno Vijlbrief
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Paul Algra
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Selma Algra
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Stijn M Bollen
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Tijs Samson
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| | - Yntor H G von Brucken Fock
- From the Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, the Netherlands (K.G.v.L., S.S., M.J.C.M.R., M.H., C.M.S.P., M.d.R., B.v.G., B.H.J.G., J.M.); Department of Radiology (M.J.C.M.R.) and Department of MICT and Imaging Techniques (T.S.), Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands; Department of Radiology, Meander Medical Centre, Amersfoort, the Netherlands (C.M.S.P., M.V.); Department of Radiology, Hospital Gelderse Vallei, Ede, the Netherlands (B.M., M.M.V.); Department of Radiology, Noordwest Ziekenhuisgroep, Alkmaar, the Netherlands (C.F.v.D., P.A.); Department of Radiology & Nuclear Medicine, Máxima Medical Center, Eindhoven, the Netherlands (E.L.K., F.v.d.W.); Department of Radiology, Ziekenhuisgroep Twente, Almelo, the Netherlands (E.V.H., F.M.t.B., M.M., O.V., Y.H.G.v.B.F.); Center for Radiology and Nuclear Medicine, Deventer Hospital, Deventer, the Netherlands (E.L.V., J.M.L., M.N.); Department of Radiology, Catharina Hospital, Eindhoven, the Netherlands (E.M.J.M., J.N., K.M.E.M.); Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands (F.A.A.M.H.); Department of Radiology, Zaans Medisch Centrum, Zaandam, the Netherlands (F.v.H.); Department of Radiology and Nuclear Medicine, Amsterdam UMC-Location University of Amsterdam, Amsterdam, the Netherlands (I.A.H.v.d.B.); Department of Radiology & Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (J.H.); Department of Radiology, Amsterdam University Medical Center, Amsterdam, the Netherlands (J.I.M.L.V.); Department of Radiology and Nuclear Medicine, Rijnstate, Arnhem, the Netherlands (L.N.D.); Department of Radiology, St Antonius Hospital, Nieuwegein, the Netherlands (L.C.M.L., S.A.); Department of Radiology, Isala Hospital, Zwolle, the Netherlands (M.F.B.); and Department of Radiology, Groene Hart Hospital, Gouda, the Netherlands (S.M.B.)
| |
Collapse
|
21
|
Pauling C, Kanber B, Arthurs OJ, Shelmerdine SC. Commercially available artificial intelligence tools for fracture detection: the evidence. BJR Open 2024; 6:tzad005. [PMID: 38352182 PMCID: PMC10860511 DOI: 10.1093/bjro/tzad005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Revised: 09/20/2023] [Accepted: 09/30/2023] [Indexed: 02/16/2024] Open
Abstract
Missed fractures are a costly healthcare issue, not only negatively impacting patient lives, leading to potential long-term disability and time off work, but also responsible for high medicolegal disbursements that could otherwise be used to improve other healthcare services. When fractures are overlooked in children, they are particularly concerning as opportunities for safeguarding may be missed. Assistance from artificial intelligence (AI) in interpreting medical images may offer a possible solution for improving patient care, and several commercial AI tools are now available for radiology workflow implementation. However, information regarding their development, evidence for performance and validation as well as the intended target population is not always clear, but vital when evaluating a potential AI solution for implementation. In this article, we review the range of available products utilizing AI for fracture detection (in both adults and children) and summarize the evidence, or lack thereof, behind their performance. This will allow others to make better informed decisions when deciding which product to procure for their specific clinical requirements.
Collapse
Affiliation(s)
- Cato Pauling
- UCL Great Ormond Street Institute of Child Health, University College London, London WC1E 6BT, United Kingdom
| | - Baris Kanber
- Queen Square Multiple Sclerosis Centre, Department of Neuroinflammation, University College London (UCL) Queen Square Institute of Neurology, Faculty of Brain Sciences, University College London, London WC1N 3BG, United Kingdom
- Department of Medical Physics and Biomedical Engineering, Centre for Medical Image Computing, University College London, London WC1E 6BT, United Kingdom
| | - Owen J Arthurs
- UCL Great Ormond Street Institute of Child Health, University College London, London WC1E 6BT, United Kingdom
- Department of Clinical Radiology, Great Ormond Street Hospital for Children NHS Foundation Trust, London WC1N 3JH, United Kingdom
- NIHR Great Ormond Street Hospital Biomedical Research Centre, Bloomsbury, London WC1N 1EH, United Kingdom
| | - Susan C Shelmerdine
- UCL Great Ormond Street Institute of Child Health, University College London, London WC1E 6BT, United Kingdom
- Department of Clinical Radiology, Great Ormond Street Hospital for Children NHS Foundation Trust, London WC1N 3JH, United Kingdom
- NIHR Great Ormond Street Hospital Biomedical Research Centre, Bloomsbury, London WC1N 1EH, United Kingdom
| |
Collapse
|
22
|
Bhatia BS, Morlese JF, Yusuf S, Xie Y, Schallhorn B, Gruen D. A real-world evaluation of the diagnostic accuracy of radiologists using positive predictive values verified from deep learning and natural language processing chest algorithms deployed retrospectively. BJR Open 2024; 6:tzad009. [PMID: 38352188 PMCID: PMC10860529 DOI: 10.1093/bjro/tzad009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 11/09/2023] [Accepted: 11/23/2023] [Indexed: 02/16/2024] Open
Abstract
Objectives This diagnostic study assessed the accuracy of radiologists retrospectively, using the deep learning and natural language processing chest algorithms implemented in Clinical Review version 3.2 for: pneumothorax, rib fractures in digital chest X-ray radiographs (CXR); aortic aneurysm, pulmonary nodules, emphysema, and pulmonary embolism in CT images. Methods The study design was double-blind (artificial intelligence [AI] algorithms and humans), retrospective, non-interventional, and at a single NHS Trust. Adult patients (≥18 years old) scheduled for CXR and CT were invited to enroll as participants through an opt-out process. Reports and images were de-identified, processed retrospectively, and AI-flagged discrepant findings were assigned to two lead radiologists, each blinded to patient identifiers and original radiologist. The radiologist's findings for each clinical condition were tallied as a verified discrepancy (true positive) or not (false positive). Results The missed findings were: 0.02% rib fractures, 0.51% aortic aneurysm, 0.32% pulmonary nodules, 0.92% emphysema, and 0.28% pulmonary embolism. The positive predictive values (PPVs) were: pneumothorax (0%), rib fractures (5.6%), aortic dilatation (43.2%), pulmonary emphysema (46.0%), pulmonary embolus (11.5%), and pulmonary nodules (9.2%). The PPV for pneumothorax was nil owing to lack of available studies that were analysed for outpatient activity. Conclusions The number of missed findings was far less than generally predicted. The chest algorithms deployed retrospectively were a useful quality tool and AI augmented the radiologists' workflow. Advances in knowledge The diagnostic accuracy of our radiologists generated missed findings of 0.02% for rib fractures CXR, 0.51% for aortic dilatation, 0.32% for pulmonary nodule, 0.92% for pulmonary emphysema, and 0.28% for pulmonary embolism for CT studies, all retrospectively evaluated with AI used as a quality tool to flag potential missed findings. It is important to account for prevalence of these chest conditions in clinical context and use appropriate clinical thresholds for decision-making, not relying solely on AI.
Collapse
Affiliation(s)
- Bahadar S Bhatia
- Directorate of Diagnostic Radiology, Sandwell & West Birmingham NHS Trust, Lyndon, West Bromwich B71 4HJ, United Kingdom
- Space Research Centre, Physics & Astronomy, University of Leicester, 92 Corporation Road, Leicester LE4 5SP, United Kingdom
| | - John F Morlese
- Directorate of Diagnostic Radiology, Sandwell & West Birmingham NHS Trust, Lyndon, West Bromwich B71 4HJ, United Kingdom
| | - Sarah Yusuf
- Directorate of Diagnostic Radiology, Sandwell & West Birmingham NHS Trust, Lyndon, West Bromwich B71 4HJ, United Kingdom
| | - Yiting Xie
- Merge, Merative (Formerly, IBM Watson Health Imaging), Ann Arbor, Michigan, MI 48108, United States
| | - Bob Schallhorn
- Merge, Merative (Formerly, IBM Watson Health Imaging), Ann Arbor, Michigan, MI 48108, United States
| | - David Gruen
- Jefferson Radiology and Radiology Partners, 111 Founders Plaza, East Hartford, Connecticut CT 06108, United States
| |
Collapse
|
23
|
Cid YD, Macpherson M, Gervais-Andre L, Zhu Y, Franco G, Santeramo R, Lim C, Selby I, Muthuswamy K, Amlani A, Hopewell H, Indrajeet D, Liakata M, Hutchinson CE, Goh V, Montana G. Development and validation of open-source deep neural networks for comprehensive chest x-ray reading: a retrospective, multicentre study. Lancet Digit Health 2024; 6:e44-e57. [PMID: 38071118 DOI: 10.1016/s2589-7500(23)00218-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Revised: 10/03/2023] [Accepted: 10/16/2023] [Indexed: 12/22/2023]
Abstract
BACKGROUND Artificial intelligence (AI) systems for automated chest x-ray interpretation hold promise for standardising reporting and reducing delays in health systems with shortages of trained radiologists. Yet, there are few freely accessible AI systems trained on large datasets for practitioners to use with their own data with a view to accelerating clinical deployment of AI systems in radiology. We aimed to contribute an AI system for comprehensive chest x-ray abnormality detection. METHODS In this retrospective cohort study, we developed open-source neural networks, X-Raydar and X-Raydar-NLP, for classifying common chest x-ray findings from images and their free-text reports. Our networks were developed using data from six UK hospitals from three National Health Service (NHS) Trusts (University Hospitals Coventry and Warwickshire NHS Trust, University Hospitals Birmingham NHS Foundation Trust, and University Hospitals Leicester NHS Trust) collectively contributing 2 513 546 chest x-ray studies taken from a 13-year period (2006-19), which yielded 1 940 508 usable free-text radiological reports written by the contemporary assessing radiologist (collectively referred to as the "historic reporters") and 1 896 034 frontal images. Chest x-rays were labelled using a taxonomy of 37 findings by a custom-trained natural language processing (NLP) algorithm, X-Raydar-NLP, from the original free-text reports. X-Raydar-NLP was trained on 23 230 manually annotated reports and tested on 4551 reports from all hospitals. 1 694 921 labelled images from the training set and 89 238 from the validation set were then used to train a multi-label image classifier. Our algorithms were evaluated on three retrospective datasets: a set of exams sampled randomly from the full NHS dataset reported during clinical practice and annotated using NLP (n=103 328); a consensus set sampled from all six hospitals annotated by three expert radiologists (two independent annotators for each image and a third consultant to facilitate disagreement resolution) under research conditions (n=1427); and an independent dataset, MIMIC-CXR, consisting of NLP-annotated exams (n=252 374). FINDINGS X-Raydar achieved a mean AUC of 0·919 (SD 0·039) on the auto-labelled set, 0·864 (0·102) on the consensus set, and 0·842 (0·074) on the MIMIC-CXR test, demonstrating similar performance to the historic clinical radiologist reporters, as assessed on the consensus set, for multiple clinically important findings, including pneumothorax, parenchymal opacification, and parenchymal mass or nodules. On the consensus set, X-Raydar outperformed historical reporter balanced accuracy with significance on 27 of 37 findings, was non-inferior on nine, and inferior on one finding, resulting in an average improvement of 13·3% (SD 13·1) to 0·763 (0·110), including a mean 5·6% (13·2) improvement in critical findings to 0·826 (0·119). INTERPRETATION Our study shows that automated classification of chest x-rays under a comprehensive taxonomy can achieve performance levels similar to those of historical reporters and exhibit robust generalisation to external data. The open-sourced neural networks can serve as foundation models for further research and are freely available to the research community. FUNDING Wellcome Trust.
Collapse
Affiliation(s)
| | - Matthew Macpherson
- WMG, University of Warwick, Coventry, UK; Mathematics Institute, University of Warwick, Coventry, UK
| | - Louise Gervais-Andre
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK
| | - Yuanyi Zhu
- WMG, University of Warwick, Coventry, UK; Mathematics Institute, University of Warwick, Coventry, UK
| | | | | | - Chee Lim
- Department of Radiology, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Ian Selby
- Department of Radiology, University of Cambridge, Cambridge, UK
| | | | - Ashik Amlani
- Department of Radiology, Guy's and St Thomas' NHS Foundation Trust, London, UK
| | - Heath Hopewell
- Department of Radiology, University Hospitals of Leicester NHS Trust, Leicester, UK
| | - Das Indrajeet
- Department of Radiology, University Hospitals of Leicester NHS Trust, Leicester, UK
| | - Maria Liakata
- The Alan Turing Institute, London, UK; Institute of Applied Data Science, Queen Mary University of London, London, UK
| | - Charles E Hutchinson
- Warwick Medical School, University of Warwick, Coventry, UK; Department of Radiology, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, UK
| | - Vicky Goh
- School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK; Department of Radiology, Guy's and St Thomas' NHS Foundation Trust, London, UK
| | - Giovanni Montana
- WMG, University of Warwick, Coventry, UK; Department of Statistics, University of Warwick, Coventry, UK; The Alan Turing Institute, London, UK.
| |
Collapse
|
24
|
Gefter WB, Prokop M, Seo JB, Raoof S, Langlotz CP, Hatabu H. Human-AI Symbiosis: A Path Forward to Improve Chest Radiography and the Role of Radiologists in Patient Care. Radiology 2024; 310:e232778. [PMID: 38259206 PMCID: PMC10831473 DOI: 10.1148/radiol.232778] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 12/08/2023] [Accepted: 12/18/2023] [Indexed: 01/24/2024]
Affiliation(s)
- Warren B. Gefter
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Mathias Prokop
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Joon Beom Seo
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Suhail Raoof
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Curtis P. Langlotz
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Hiroto Hatabu
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| |
Collapse
|
25
|
Zsidai B, Hilkert AS, Kaarre J, Narup E, Senorski EH, Grassi A, Ley C, Longo UG, Herbst E, Hirschmann MT, Kopf S, Seil R, Tischer T, Samuelsson K, Feldt R. A practical guide to the implementation of AI in orthopaedic research - part 1: opportunities in clinical application and overcoming existing challenges. J Exp Orthop 2023; 10:117. [PMID: 37968370 PMCID: PMC10651597 DOI: 10.1186/s40634-023-00683-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 10/21/2023] [Indexed: 11/17/2023] Open
Abstract
Artificial intelligence (AI) has the potential to transform medical research by improving disease diagnosis, clinical decision-making, and outcome prediction. Despite the rapid adoption of AI and machine learning (ML) in other domains and industry, deployment in medical research and clinical practice poses several challenges due to the inherent characteristics and barriers of the healthcare sector. Therefore, researchers aiming to perform AI-intensive studies require a fundamental understanding of the key concepts, biases, and clinical safety concerns associated with the use of AI. Through the analysis of large, multimodal datasets, AI has the potential to revolutionize orthopaedic research, with new insights regarding the optimal diagnosis and management of patients affected musculoskeletal injury and disease. The article is the first in a series introducing fundamental concepts and best practices to guide healthcare professionals and researcher interested in performing AI-intensive orthopaedic research studies. The vast potential of AI in orthopaedics is illustrated through examples involving disease- or injury-specific outcome prediction, medical image analysis, clinical decision support systems and digital twin technology. Furthermore, it is essential to address the role of human involvement in training unbiased, generalizable AI models, their explainability in high-risk clinical settings and the implementation of expert oversight and clinical safety measures for failure. In conclusion, the opportunities and challenges of AI in medicine are presented to ensure the safe and ethical deployment of AI models for orthopaedic research and clinical application. Level of evidence IV.
Collapse
Affiliation(s)
- Bálint Zsidai
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden.
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden.
| | - Ann-Sophie Hilkert
- Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden
- Medfield Diagnostics AB, Gothenburg, Sweden
| | - Janina Kaarre
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- Department of Orthopaedic Surgery, UPMC Freddie Fu Sports Medicine Center, University of Pittsburgh, Pittsburgh, USA
| | - Eric Narup
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Eric Hamrin Senorski
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Health and Rehabilitation, Institute of Neuroscience and Physiology, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- Sportrehab Sports Medicine Clinic, Gothenburg, Sweden
| | - Alberto Grassi
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- IIa Clinica Ortopedica E Traumatologica, IRCCS Istituto Ortopedico Rizzoli, Bologna, Italy
| | - Christophe Ley
- Department of Mathematics, University of Luxembourg, Esch-Sur-Alzette, Luxembourg
| | - Umile Giuseppe Longo
- Department of Orthopaedic and Trauma Surgery, Campus Bio-Medico University, Rome, Italy
| | - Elmar Herbst
- Department of Trauma, Hand and Reconstructive Surgery, University Hospital Münster, Münster, Germany
| | - Michael T Hirschmann
- Department of Orthopedic Surgery and Traumatology, Head Knee Surgery and DKF Head of Research, Kantonsspital Baselland, 4101, Bruderholz, Switzerland
| | - Sebastian Kopf
- Center of Orthopaedics and Traumatology, University Hospital Brandenburg a.d.H., Brandenburg Medical School Theodor Fontane, 14770, Brandenburg a.d.H., Germany
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, 14770, Brandenburg a.d.H., Germany
| | - Romain Seil
- Department of Orthopaedic Surgery, Centre Hospitalier Luxembourg and Luxembourg Institute of Health, Luxembourg, Luxembourg
| | - Thomas Tischer
- Clinic for Orthopaedics and Trauma Surgery, Malteser Waldkrankenhaus St. Marien, Erlangen, Germany
| | - Kristian Samuelsson
- Sahlgrenska Sports Medicine Center, Gothenburg, Sweden
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
- Department of Orthopaedics, Sahlgrenska University Hospital, Mölndal, Sweden
| | - Robert Feldt
- Department of Orthopaedics, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| |
Collapse
|
26
|
Shin H, Kim T, Park J, Raj H, Jabbar MS, Abebaw ZD, Lee J, Van CC, Kim H, Shin D. Pulmonary abnormality screening on chest x-rays from different machine specifications: a generalized AI-based image manipulation pipeline. Eur Radiol Exp 2023; 7:68. [PMID: 37940797 PMCID: PMC10632317 DOI: 10.1186/s41747-023-00386-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Accepted: 09/12/2023] [Indexed: 11/10/2023] Open
Abstract
BACKGROUND Chest x-ray is commonly used for pulmonary abnormality screening. However, since the image characteristics of x-rays highly depend on the machine specifications, an artificial intelligence (AI) model developed for specific equipment usually fails when clinically applied to various machines. To overcome this problem, we propose an image manipulation pipeline. METHODS A total of 15,010 chest x-rays from systems with different generators/detectors were retrospectively collected from five institutions from May 2020 to February 2021. We developed an AI model to classify pulmonary abnormalities using x-rays from a single system. Then, we externally tested its performance on chest x-rays from various machine specifications. We compared the area under the receiver operating characteristics curve (AUC) of AI models developed using conventional image processing pipelines (histogram equalization [HE], contrast-limited histogram equalization [CLAHE], and unsharp masking [UM] with common data augmentations) with that of the proposed manipulation pipeline (XM-pipeline). RESULTS The XM-pipeline model showed the highest performance for all the datasets of different machine specifications, such as chest x-rays acquired from a computed radiography system (n = 356, AUC 0.944 for XM-pipeline versus 0.917 for HE, 0.705 for CLAHE, 0.544 for UM, p [Formula: see text] 0.001, for all) and from a mobile x-ray generator (n = 204, AUC 0.949 for XM-pipeline versus 0.933 for HE, p = 0.042, 0.932 for CLAHE (p = 0.009), 0.925 for UM (p = 0.001). CONCLUSIONS Applying the XM-pipeline to AI training increased the diagnostic performance of the AI model on the chest x-rays of different machine configurations. RELEVANCE STATEMENT The proposed training pipeline would successfully promote a wide application of the AI model for abnormality screening when chest x-rays are acquired using various x-ray machines. KEY POINTS • AI models developed using x-rays of a specific machine suffer from generalization. • We proposed a new image processing pipeline to address the generalization problem. • AI models were tested using multicenter external x-ray datasets of various machines. • AI with our pipeline achieved the highest diagnostic performance than conventional methods.
Collapse
Affiliation(s)
- Heejun Shin
- Artificial Intelligence Engineering Division, RadiSen Co., Ltd, Seoul, Korea
| | - Taehee Kim
- Artificial Intelligence Engineering Division, RadiSen Co., Ltd, Seoul, Korea
| | - Juhyung Park
- Laboratory for Imaging Science and Technology, Department of Electrical and Computer Engineering, Seoul National University, Seoul, Korea
| | - Hruthvik Raj
- Artificial Intelligence Engineering Division, RadiSen Co., Ltd, Seoul, Korea
| | | | | | - Jongho Lee
- Laboratory for Imaging Science and Technology, Department of Electrical and Computer Engineering, Seoul National University, Seoul, Korea
| | - Cong Cung Van
- Department of Radiology, National Lung Hospital, Hanoi, Vietnam
| | - Hyungjin Kim
- Department of Radiology, Seoul National University Hospital, Seoul, Korea
| | - Dongmyung Shin
- Artificial Intelligence Engineering Division, RadiSen Co., Ltd, Seoul, Korea.
| |
Collapse
|
27
|
Macpherson MS, Hutchinson CE, Horst C, Goh V, Montana G. Patient Reidentification from Chest Radiographs: An Interpretable Deep Metric Learning Approach and Its Applications. Radiol Artif Intell 2023; 5:e230019. [PMID: 38074779 PMCID: PMC10698609 DOI: 10.1148/ryai.230019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2023] [Revised: 08/24/2023] [Accepted: 09/07/2023] [Indexed: 01/31/2024]
Abstract
Purpose To train an explainable deep learning model for patient reidentification in chest radiograph datasets and assess changes in model-perceived patient identity as a marker for emerging radiologic abnormalities in longitudinal image sets. Materials and Methods This retrospective study used a set of 1 094 537 frontal chest radiographs and free-text reports from 259 152 patients obtained from six hospitals between 2006 and 2019, with validation on the public ChestX-ray14, CheXpert, and MIMIC-CXR datasets. A deep learning model was trained for patient reidentification and assessed on patient identity confirmation, retrieval of patient images from a database based on a query image, and radiologic abnormality prediction in longitudinal image sets. The representation learned was incorporated into a generative adversarial network, allowing visual explanations of the relevant features. Performance was evaluated with sensitivity, specificity, F1 score, Precision at 1, R-Precision, and area under the receiver operating characteristic curve (AUC) for normal and abnormal prediction. Results Patient reidentification was achieved with a mean F1 score of 0.996 ± 0.001 (2 SD) on the internal test set (26 152 patients) and F1 scores of 0.947-0.993 on the external test data. Database retrieval yielded a mean Precision at 1 score of 0.976 ± 0.005 at 299 × 299 resolution on the internal test set and Precision at 1 scores between 0.868 and 0.950 on the external datasets. Patient sex, age, weight, and other factors were identified as key model features. The model achieved an AUC of 0.73 ± 0.01 for abnormality prediction versus an AUC of 0.58 ± 0.01 for age prediction error. Conclusion The image features used by a deep learning patient reidentification model for chest radiographs corresponded to intuitive human-interpretable characteristics, and changes in these identifying features over time may act as markers for an emerging abnormality.Keywords: Conventional Radiography, Thorax, Feature Detection, Supervised Learning, Convolutional Neural Network, Principal Component Analysis Supplemental material is available for this article. © RSNA, 2023See also the commentary by Raghu and Lu in this issue.
Collapse
Affiliation(s)
- Matthew S. Macpherson
- From the Mathematics Institute (M.S.M.), Warwick Medical School
(C.E.H.), Department of Statistics (G.M.), and Warwick Manufacturing Group
(G.M.), University of Warwick, Coventry CV4 7AL, United Kingdom;
Department of Radiology, University Hospitals Coventry and Warwickshire NHS
Trust, Coventry, United Kingdom (C.E.H.); School of Biomedical Engineering
& Imaging Sciences, King’s College London, London, United Kingdom
(C.H., V.G.); Department of Radiology, Guy’s and St Thomas’ NHS
Foundation Trust, London, United Kingdom (V.G.); and Alan Turing Institute,
London, United Kingdom (G.M.)
| | - Charles E. Hutchinson
- From the Mathematics Institute (M.S.M.), Warwick Medical School
(C.E.H.), Department of Statistics (G.M.), and Warwick Manufacturing Group
(G.M.), University of Warwick, Coventry CV4 7AL, United Kingdom;
Department of Radiology, University Hospitals Coventry and Warwickshire NHS
Trust, Coventry, United Kingdom (C.E.H.); School of Biomedical Engineering
& Imaging Sciences, King’s College London, London, United Kingdom
(C.H., V.G.); Department of Radiology, Guy’s and St Thomas’ NHS
Foundation Trust, London, United Kingdom (V.G.); and Alan Turing Institute,
London, United Kingdom (G.M.)
| | - Carolyn Horst
- From the Mathematics Institute (M.S.M.), Warwick Medical School
(C.E.H.), Department of Statistics (G.M.), and Warwick Manufacturing Group
(G.M.), University of Warwick, Coventry CV4 7AL, United Kingdom;
Department of Radiology, University Hospitals Coventry and Warwickshire NHS
Trust, Coventry, United Kingdom (C.E.H.); School of Biomedical Engineering
& Imaging Sciences, King’s College London, London, United Kingdom
(C.H., V.G.); Department of Radiology, Guy’s and St Thomas’ NHS
Foundation Trust, London, United Kingdom (V.G.); and Alan Turing Institute,
London, United Kingdom (G.M.)
| | - Vicky Goh
- From the Mathematics Institute (M.S.M.), Warwick Medical School
(C.E.H.), Department of Statistics (G.M.), and Warwick Manufacturing Group
(G.M.), University of Warwick, Coventry CV4 7AL, United Kingdom;
Department of Radiology, University Hospitals Coventry and Warwickshire NHS
Trust, Coventry, United Kingdom (C.E.H.); School of Biomedical Engineering
& Imaging Sciences, King’s College London, London, United Kingdom
(C.H., V.G.); Department of Radiology, Guy’s and St Thomas’ NHS
Foundation Trust, London, United Kingdom (V.G.); and Alan Turing Institute,
London, United Kingdom (G.M.)
| | - Giovanni Montana
- From the Mathematics Institute (M.S.M.), Warwick Medical School
(C.E.H.), Department of Statistics (G.M.), and Warwick Manufacturing Group
(G.M.), University of Warwick, Coventry CV4 7AL, United Kingdom;
Department of Radiology, University Hospitals Coventry and Warwickshire NHS
Trust, Coventry, United Kingdom (C.E.H.); School of Biomedical Engineering
& Imaging Sciences, King’s College London, London, United Kingdom
(C.H., V.G.); Department of Radiology, Guy’s and St Thomas’ NHS
Foundation Trust, London, United Kingdom (V.G.); and Alan Turing Institute,
London, United Kingdom (G.M.)
| |
Collapse
|
28
|
Bernstein MH, Atalay MK, Dibble EH, Maxwell AWP, Karam AR, Agarwal S, Ward RC, Healey TT, Baird GL. Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography. Eur Radiol 2023; 33:8263-8269. [PMID: 37266657 PMCID: PMC10235827 DOI: 10.1007/s00330-023-09747-1] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2022] [Revised: 04/16/2023] [Accepted: 04/18/2023] [Indexed: 06/03/2023]
Abstract
OBJECTIVE To examine whether incorrect AI results impact radiologist performance, and if so, whether human factors can be optimized to reduce error. METHODS Multi-reader design, 6 radiologists interpreted 90 identical chest radiographs (follow-up CT needed: yes/no) on four occasions (09/20-01/22). No AI result was provided for session 1. Sham AI results were provided for sessions 2-4, and AI for 12 cases were manipulated to be incorrect (8 false positives (FP), 4 false negatives (FN)) (0.87 ROC-AUC). In the Delete AI (No Box) condition, radiologists were told AI results would not be saved for the evaluation. In Keep AI (No Box) and Keep AI (Box), radiologists were told results would be saved. In Keep AI (Box), the ostensible AI program visually outlined the region of suspicion. AI results were constant between conditions. RESULTS Relative to the No AI condition (FN = 2.7%, FP = 51.4%), FN and FPs were higher in the Keep AI (No Box) (FN = 33.0%, FP = 86.0%), Delete AI (No Box) (FN = 26.7%, FP = 80.5%), and Keep AI (Box) (FN = to 20.7%, FP = 80.5%) conditions (all ps < 0.05). FNs were higher in the Keep AI (No Box) condition (33.0%) than in the Keep AI (Box) condition (20.7%) (p = 0.04). FPs were higher in the Keep AI (No Box) (86.0%) condition than in the Delete AI (No Box) condition (80.5%) (p = 0.03). CONCLUSION Incorrect AI causes radiologists to make incorrect follow-up decisions when they were correct without AI. This effect is mitigated when radiologists believe AI will be deleted from the patient's file or a box is provided around the region of interest. CLINICAL RELEVANCE STATEMENT When AI is wrong, radiologists make more errors than they would have without AI. Based on human factors psychology, our manuscript provides evidence for two AI implementation strategies that reduce the deleterious effects of incorrect AI. KEY POINTS • When AI provided incorrect results, false negative and false positive rates among the radiologists increased. • False positives decreased when AI results were deleted, versus kept, in the patient's record. • False negatives and false positives decreased when AI visually outlined the region of suspicion.
Collapse
Affiliation(s)
- Michael H Bernstein
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA.
- Rhode Island Hospital, Providence, RI, USA.
- Brown Radiology Human Factors Laboratory, Providence, RI, USA.
| | - Michael K Atalay
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
- Brown Radiology Human Factors Laboratory, Providence, RI, USA
| | - Elizabeth H Dibble
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Aaron W P Maxwell
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
- Brown Radiology Human Factors Laboratory, Providence, RI, USA
| | - Adib R Karam
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Saurabh Agarwal
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Robert C Ward
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Terrance T Healey
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
| | - Grayson L Baird
- Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Providence, RI, USA
- Rhode Island Hospital, Providence, RI, USA
- Brown Radiology Human Factors Laboratory, Providence, RI, USA
| |
Collapse
|
29
|
Kim D, Lee JH, Jang MJ, Park J, Hong W, Lee CS, Yang SY, Park CM. The Performance of a Deep Learning-Based Automatic Measurement Model for Measuring the Cardiothoracic Ratio on Chest Radiographs. Bioengineering (Basel) 2023; 10:1077. [PMID: 37760179 PMCID: PMC10525628 DOI: 10.3390/bioengineering10091077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 08/28/2023] [Accepted: 09/07/2023] [Indexed: 09/29/2023] Open
Abstract
OBJECTIVE Prior studies on models based on deep learning (DL) and measuring the cardiothoracic ratio (CTR) on chest radiographs have lacked rigorous agreement analyses with radiologists or reader tests. We validated the performance of a commercially available DL-based CTR measurement model with various thoracic pathologies, and performed agreement analyses with thoracic radiologists and reader tests using a probabilistic-based reference. MATERIALS AND METHODS This study included 160 posteroanterior view chest radiographs (no lung or pleural abnormalities, pneumothorax, pleural effusion, consolidation, and n = 40 in each category) to externally test a DL-based CTR measurement model. To assess the agreement between the model and experts, intraclass or interclass correlation coefficients (ICCs) were compared between the model and two thoracic radiologists. In the reader tests with a probabilistic-based reference standard (Dawid-Skene consensus), we compared diagnostic measures-including sensitivity and negative predictive value (NPV)-for cardiomegaly between the model and five other radiologists using the non-inferiority test. RESULTS For the 160 chest radiographs, the model measured a median CTR of 0.521 (interquartile range, 0.446-0.59) and a mean CTR of 0.522 ± 0.095. The ICC between the two thoracic radiologists and between the model and two thoracic radiologists was not significantly different (0.972 versus 0.959, p = 0.192), even across various pathologies (all p-values > 0.05). The model showed non-inferior diagnostic performance, including sensitivity (96.3% versus 97.8%) and NPV (95.6% versus 97.4%) (p < 0.001 in both), compared with the radiologists for all 160 chest radiographs. However, it showed inferior sensitivity in chest radiographs with consolidation (95.5% versus 99.9%; p = 0.082) and NPV in chest radiographs with pleural effusion (92.9% versus 94.6%; p = 0.079) and consolidation (94.1% versus 98.7%; p = 0.173). CONCLUSION While the sensitivity and NPV of this model for diagnosing cardiomegaly in chest radiographs with consolidation or pleural effusion were not as high as those of the radiologists, it demonstrated good agreement with the thoracic radiologists in measuring the CTR across various pathologies.
Collapse
Affiliation(s)
- Donguk Kim
- Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea;
| | - Jong Hyuk Lee
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea
| | - Myoung-jin Jang
- Medical Research Collaborating Center, Seoul National University Hospital, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea
| | - Jongsoo Park
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea
- Department of Radiology, College of Medicine, Yeungnam University 170, Hyeonchung-ro, Nam-gu, Daegu 42415, Republic of Korea
| | - Wonju Hong
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea
- Department of Radiology, Hallym University Sacred Heart Hospital, Anyang-si, Gyeonggi-do 14068, Republic of Korea
| | - Chan Su Lee
- Center for Artificial Intelligence in Medicine and Imaging, HealthHub Co. Ltd., 623, Gangnam-daero, Seocho-gu, Seoul 06524, Republic of Korea
| | - Si Yeong Yang
- Center for Artificial Intelligence in Medicine and Imaging, HealthHub Co. Ltd., 623, Gangnam-daero, Seocho-gu, Seoul 06524, Republic of Korea
| | - Chang Min Park
- Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea;
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea
- Institute of Radiation Medicine, Seoul National University Medical Research Center, 101, Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea
| |
Collapse
|
30
|
Lind Plesner L, Müller FC, Brejnebøl MW, Laustrup LC, Rasmussen F, Nielsen OW, Boesen M, Brun Andersen M. Commercially Available Chest Radiograph AI Tools for Detecting Airspace Disease, Pneumothorax, and Pleural Effusion. Radiology 2023; 308:e231236. [PMID: 37750768 DOI: 10.1148/radiol.231236] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/27/2023]
Abstract
Background Commercially available artificial intelligence (AI) tools can assist radiologists in interpreting chest radiographs, but their real-life diagnostic accuracy remains unclear. Purpose To evaluate the diagnostic accuracy of four commercially available AI tools for detection of airspace disease, pneumothorax, and pleural effusion on chest radiographs. Materials and Methods This retrospective study included consecutive adult patients who underwent chest radiography at one of four Danish hospitals in January 2020. Two thoracic radiologists (or three, in cases of disagreement) who had access to all previous and future imaging labeled chest radiographs independently for the reference standard. Area under the receiver operating characteristic curve, sensitivity, and specificity were calculated. Sensitivity and specificity were additionally stratified according to the severity of findings, number of findings on chest radiographs, and radiographic projection. The χ2 and McNemar tests were used for comparisons. Results The data set comprised 2040 patients (median age, 72 years [IQR, 58-81 years]; 1033 female), of whom 669 (32.8%) had target findings. The AI tools demonstrated areas under the receiver operating characteristic curve ranging 0.83-0.88 for airspace disease, 0.89-0.97 for pneumothorax, and 0.94-0.97 for pleural effusion. Sensitivities ranged 72%-91% for airspace disease, 63%-90% for pneumothorax, and 62%-95% for pleural effusion. Negative predictive values ranged 92%-100% for all target findings. In airspace disease, pneumothorax, and pleural effusion, specificity was high for chest radiographs with normal or single findings (range, 85%-96%, 99%-100%, and 95%-100%, respectively) and markedly lower for chest radiographs with four or more findings (range, 27%-69%, 96%-99%, 65%-92%, respectively) (P < .001). AI sensitivity was lower for vague airspace disease (range, 33%-61%) and small pneumothorax or pleural effusion (range, 9%-94%) compared with larger findings (range, 81%-100%; P value range, > .99 to < .001). Conclusion Current-generation AI tools showed moderate to high sensitivity for detecting airspace disease, pneumothorax, and pleural effusion on chest radiographs. However, they produced more false-positive findings than radiology reports, and their performance decreased for smaller-sized target findings and when multiple findings were present. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Yanagawa and Tomiyama in this issue.
Collapse
Affiliation(s)
- Louis Lind Plesner
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Felix C Müller
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Mathias W Brejnebøl
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Lene C Laustrup
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Finn Rasmussen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Olav W Nielsen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Mikael Boesen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| | - Michael Brun Andersen
- From the Department of Radiology, Herlev and Gentofte Hospital, Borgmester Ib, Juuls vej 1 Herlev, Copenhagen 2730, Denmark (L.L.P., F.C.M., M.W.B., L.C.L., M.B.A.); Faculty of Health Sciences, University of Copenhagen, Copenhagen, Denmark (L.L.P., M.W.B., O.W.N., M.B., M.B.A.); Radiological Artificial Intelligence Testcenter, RAIT.dk, Capital Region of Denmark (L.L.P., F.C.M., M.W.B., M.B., M.B.A.); Departments of Radiology (M.W.B., M.B.) and Cardiology (O.W.N.), Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark; and Department of Radiology, Aarhus University Hospital, Aarhus, Denmark (F.R.)
| |
Collapse
|
31
|
Prabhakaran S, Choong KWK, Prabhakaran S, Choy KT, Kong JC. Accuracy of deep neural learning models in the imaging prediction of pathological complete response after neoadjuvant chemoradiotherapy for locally advanced rectal cancer: a systematic review. Langenbecks Arch Surg 2023; 408:321. [PMID: 37594552 DOI: 10.1007/s00423-023-03039-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2023] [Accepted: 08/01/2023] [Indexed: 08/19/2023]
Abstract
PURPOSE Up to 15-27% of patients achieve pathologic complete response (pCR) following neoadjuvant chemoradiotherapy (CRT) for locally advanced rectal cancer (LARC). Deep neural learning (DL) algorithms have been suggested to be a useful adjunct to allow accurate prediction of pCR and to identify patients who could potentially avoid surgery. This systematic review aims to interrogate the accuracy of DL algorithms at predicting pCR. METHODS Embase (PubMed, MEDLINE) databases and Google Scholar were searched to identify eligible English-language studies, with the search concluding in July 2022. Studies reporting on the accuracy of DL models in predicting pCR were selected for review and information pertaining to study characteristics and diagnostic measures was extracted from relevant studies. Risk of bias was evaluated using the Newcastle-Ottawa scale (NOS). RESULTS Our search yielded 85 potential publications. Nineteen full texts were reviewed, and a total of 12 articles were included in this systematic review. There were six retrospective and six prospective cohort studies. The most common DL algorithm used was the Convolutional Neural Network (CNN). Performance comparison was carried out via single modality comparison. The median performance for each best-performing algorithm was an AUC of 0.845 (range 0.71-0.99) and Accuracy of 0.85 (0.83-0.98). CONCLUSIONS There is a promising role for DL models in the prediction of pCR following neoadjuvant-CRT for LARC. Further studies are needed to provide a standardised comparison in order to allow for large-scale clinical application. PROPERO REGISTRATION PROSPERO 2021 CRD42021269904 Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021269904 .
Collapse
Affiliation(s)
- Sowmya Prabhakaran
- Department of General Surgical Specialties, The Royal Melbourne Hospital, Melbourne, Victoria, Australia.
| | | | - Swetha Prabhakaran
- Department of Colorectal Surgery, Alfred Hospital, Melbourne, Victoria, Australia
| | - Kay Tai Choy
- Department of Surgery, Austin Health, Melbourne, Victoria, Australia
| | - Joseph Ch Kong
- Department of Colorectal Surgery, Alfred Hospital, Melbourne, Victoria, Australia
- Central Clinical School, Monash University, Melbourne, Victoria, Australia
- Department of Surgical Oncology, Peter MacCallum Cancer Centre, Melbourne, Victoria, Australia
| |
Collapse
|
32
|
Solomon C, Shmueli O, Shrot S, Blumenfeld-Katzir T, Radunsky D, Omer N, Stern N, Reichman DBA, Hoffmann C, Salti M, Greenspan H, Ben-Eliezer N. Psychophysical Evaluation of Visual vs. Computer-Aided Detection of Brain Lesions on Magnetic Resonance Images. J Magn Reson Imaging 2023; 58:642-649. [PMID: 36495014 DOI: 10.1002/jmri.28559] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Revised: 11/25/2022] [Accepted: 11/28/2022] [Indexed: 07/20/2023] Open
Abstract
BACKGROUND Magnetic resonance imaging (MRI) diagnosis is usually performed by analyzing contrast-weighted images, where pathology is detected once it reached a certain visual threshold. Computer-aided diagnosis (CAD) has been proposed as a way for achieving higher sensitivity to early pathology. PURPOSE To compare conventional (i.e., visual) MRI assessment of artificially generated multiple sclerosis (MS) lesions in the brain's white matter to CAD based on a deep neural network. STUDY TYPE Prospective. POPULATION A total of 25 neuroradiologists (15 males, age 39 ± 9, 9 ± 9.8 years of experience) independently assessed all synthetic lesions. FIELD STRENGTH/SEQUENCE A 3.0 T, T2 -weighted multi-echo spin-echo (MESE) sequence. ASSESSMENT MS lesions of varying severity levels were artificially generated in healthy volunteer MRI scans by manipulating T2 values. Radiologists and a neural network were tasked with detecting these lesions in a series of 48 MR images. Sixteen images presented healthy anatomy and the rest contained a single lesion at eight increasing severity levels (6%, 9%, 12%, 15%, 18%, 21%, 25%, and 30% elevation in T2 ). True positive (TP) rates, false positive (FP) rates, and odds ratios (ORs) were compared between radiological diagnosis and CAD across the range lesion severity levels. STATISTICAL TESTS Diagnostic performance of the two approaches was compared using z-tests on TP rates, FP rates, and the logarithm of ORs across severity levels. A P-value <0.05 was considered statistically significant. RESULTS ORs of identifying pathology were significantly higher for CAD vis-à-vis visual inspection for all lesions' severity levels. For a 6% change in T2 value (lowest severity), radiologists' TP and FP rates were not significantly different (P = 0.12), while the corresponding CAD results remained statistically significant. DATA CONCLUSION CAD is capable of detecting the presence or absence of more subtle lesions with greater precision than the representative group of 25 radiologists chosen in this study. LEVEL OF EVIDENCE 1 TECHNICAL EFFICACY: Stage 3.
Collapse
Affiliation(s)
- Chen Solomon
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
| | - Omer Shmueli
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
| | - Shai Shrot
- Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat-Gan, Israel
| | | | - Dvir Radunsky
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
| | - Noam Omer
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
| | - Neta Stern
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
| | | | - Chen Hoffmann
- Sackler School of Medicine, Tel-Aviv University, Tel-Aviv, Israel
- Department of Diagnostic Imaging, Sheba Medical Center, Ramat-Gan, Israel
| | - Moti Salti
- Brain Imaging Research Center (BIRC), Ben-Gurion University, Beer-Sheva, Israel
- Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beer-Sheva, Israel
| | - Hayit Greenspan
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
| | - Noam Ben-Eliezer
- Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv, Israel
- Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beer-Sheva, Israel
- Center for Advanced Imaging Innovation and Research (CAI2R), New York University, New York, New York, USA
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
33
|
Tang CHM, Seah JCY, Ahmad HK, Milne MR, Wardman JB, Buchlak QD, Esmaili N, Lambert JF, Jones CM. Analysis of Line and Tube Detection Performance of a Chest X-ray Deep Learning Model to Evaluate Hidden Stratification. Diagnostics (Basel) 2023; 13:2317. [PMID: 37510062 PMCID: PMC10378683 DOI: 10.3390/diagnostics13142317] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Revised: 07/05/2023] [Accepted: 07/07/2023] [Indexed: 07/30/2023] Open
Abstract
This retrospective case-control study evaluated the diagnostic performance of a commercially available chest radiography deep convolutional neural network (DCNN) in identifying the presence and position of central venous catheters, enteric tubes, and endotracheal tubes, in addition to a subgroup analysis of different types of lines/tubes. A held-out test dataset of 2568 studies was sourced from community radiology clinics and hospitals in Australia and the USA, and was then ground-truth labelled for the presence, position, and type of line or tube from the consensus of a thoracic specialist radiologist and an intensive care clinician. DCNN model performance for identifying and assessing the positioning of central venous catheters, enteric tubes, and endotracheal tubes over the entire dataset, as well as within each subgroup, was evaluated. The area under the receiver operating characteristic curve (AUC) was assessed. The DCNN algorithm displayed high performance in detecting the presence of lines and tubes in the test dataset with AUCs > 0.99, and good position classification performance over a subpopulation of ground truth positive cases with AUCs of 0.86-0.91. The subgroup analysis showed that model performance was robust across the various subtypes of lines or tubes, although position classification performance of peripherally inserted central catheters was relatively lower. Our findings indicated that the DCNN algorithm performed well in the detection and position classification of lines and tubes, supporting its use as an assistant for clinicians. Further work is required to evaluate performance in rarer scenarios, as well as in less common subgroups.
Collapse
Affiliation(s)
- Cyril H M Tang
- Annalise.ai, Sydney, NSW 2000, Australia
- Intensive Care Unit, Gosford Hospital, Sydney, NSW 2250, Australia
| | - Jarrel C Y Seah
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Radiology, Alfred Health, Melbourne, VIC 3004, Australia
| | | | | | | | - Quinlan D Buchlak
- Annalise.ai, Sydney, NSW 2000, Australia
- School of Medicine, The University of Notre Dame Australia, Sydney, NSW 2007, Australia
- Department of Neurosurgery, Monash Health, Melbourne, VIC 3168, Australia
| | - Nazanin Esmaili
- School of Medicine, The University of Notre Dame Australia, Sydney, NSW 2007, Australia
- Faculty of Engineering and Information Technology, University of Technology Sydney, Ultimo, NSW 2007, Australia
| | | | - Catherine M Jones
- Annalise.ai, Sydney, NSW 2000, Australia
- I-MED Radiology Network, Brisbane, QLD 4006, Australia
- School of Public and Preventive Health, Monash University, Clayton, VIC 3800, Australia
- Department of Clinical Imaging Science, University of Sydney, Sydney, NSW 2006, Australia
| |
Collapse
|
34
|
Arzamasov K, Vasilev Y, Vladzymyrskyy A, Omelyanskaya O, Shulkin I, Kozikhina D, Goncharova I, Gelezhe P, Kirpichev Y, Bobrovskaya T, Andreychenko A. An International Non-Inferiority Study for the Benchmarking of AI for Routine Radiology Cases: Chest X-ray, Fluorography and Mammography. Healthcare (Basel) 2023; 11:1684. [PMID: 37372802 DOI: 10.3390/healthcare11121684] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2023] [Revised: 06/01/2023] [Accepted: 06/04/2023] [Indexed: 06/29/2023] Open
Abstract
An international reader study was conducted to gauge an average diagnostic accuracy of radiologists interpreting chest X-ray images, including those from fluorography and mammography, and establish requirements for stand-alone radiological artificial intelligence (AI) models. The retrospective studies in the datasets were labelled as containing or not containing target pathological findings based on a consensus of two experienced radiologists, and the results of a laboratory test and follow-up examination, where applicable. A total of 204 radiologists from 11 countries with various experience performed an assessment of the dataset with a 5-point Likert scale via a web platform. Eight commercial radiological AI models analyzed the same dataset. The AI AUROC was 0.87 (95% CI:0.83-0.9) versus 0.96 (95% CI 0.94-0.97) for radiologists. The sensitivity and specificity of AI versus radiologists were 0.71 (95% CI 0.64-0.78) versus 0.91 (95% CI 0.86-0.95) and 0.93 (95% CI 0.89-0.96) versus 0.9 (95% CI 0.85-0.94) for AI. The overall diagnostic accuracy of radiologists was superior to AI for chest X-ray and mammography. However, the accuracy of AI was noninferior to the least experienced radiologists for mammography and fluorography, and to all radiologists for chest X-ray. Therefore, an AI-based first reading could be recommended to reduce the workload burden of radiologists for the most common radiological studies such as chest X-ray and mammography.
Collapse
Affiliation(s)
- Kirill Arzamasov
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Yuriy Vasilev
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
- Federal State Budgetary Institution "National Medical and Surgical Center Named after N.I. Pirogov" of the Ministry of Health of the Russian Federation, Nizhnyaya Pervomayskaya Street, 70, 105203 Moscow, Russia
| | - Anton Vladzymyrskyy
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
- Department of Information and Internet Technologies, I.M. Sechenov First Moscow State Medical University of the Ministry of Health of the Russian Federation (Sechenov University), Trubetskaya Street, 8, Building 2, 119991 Moscow, Russia
| | - Olga Omelyanskaya
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Igor Shulkin
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Darya Kozikhina
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Inna Goncharova
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Pavel Gelezhe
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Yury Kirpichev
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Tatiana Bobrovskaya
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| | - Anna Andreychenko
- State Budget-Funded Health Care Institution of the City of Moscow "Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department", Petrovka Street, 24, Building 1, 127051 Moscow, Russia
| |
Collapse
|
35
|
Azizi S, Culp L, Freyberg J, Mustafa B, Baur S, Kornblith S, Chen T, Tomasev N, Mitrović J, Strachan P, Mahdavi SS, Wulczyn E, Babenko B, Walker M, Loh A, Chen PHC, Liu Y, Bavishi P, McKinney SM, Winkens J, Roy AG, Beaver Z, Ryan F, Krogue J, Etemadi M, Telang U, Liu Y, Peng L, Corrado GS, Webster DR, Fleet D, Hinton G, Houlsby N, Karthikesalingam A, Norouzi M, Natarajan V. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nat Biomed Eng 2023:10.1038/s41551-023-01049-7. [PMID: 37291435 DOI: 10.1038/s41551-023-01049-7] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Accepted: 05/02/2023] [Indexed: 06/10/2023]
Abstract
Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Ting Chen
- Google Research, Mountain View, CA, USA
| | | | | | | | | | | | | | | | - Aaron Loh
- Google Research, Mountain View, CA, USA
| | | | - Yuan Liu
- Google Research, Mountain View, CA, USA
| | | | | | | | | | | | - Fiona Ryan
- Georgia Institute of Technology, Computer Science, Atlanta, GA, USA
| | | | - Mozziyar Etemadi
- School of Medicine/School of Engineering, Northwestern University, Chicago, IL, USA
| | | | - Yun Liu
- Google Research, Mountain View, CA, USA
| | - Lily Peng
- Google Research, Mountain View, CA, USA
| | | | | | | | | | | | | | | | | |
Collapse
|
36
|
Agrawal A, Khatri GD, Khurana B, Sodickson AD, Liang Y, Dreizin D. A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations. Emerg Radiol 2023; 30:267-277. [PMID: 36913061 PMCID: PMC10362990 DOI: 10.1007/s10140-023-02121-0] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2023] [Accepted: 02/28/2023] [Indexed: 03/14/2023]
Abstract
PURPOSE There is a growing body of diagnostic performance studies for emergency radiology-related artificial intelligence/machine learning (AI/ML) tools; however, little is known about user preferences, concerns, experiences, expectations, and the degree of penetration of AI tools in emergency radiology. Our aim is to conduct a survey of the current trends, perceptions, and expectations regarding AI among American Society of Emergency Radiology (ASER) members. METHODS An anonymous and voluntary online survey questionnaire was e-mailed to all ASER members, followed by two reminder e-mails. A descriptive analysis of the data was conducted, and results summarized. RESULTS A total of 113 members responded (response rate 12%). The majority were attending radiologists (90%) with greater than 10 years' experience (80%) and from an academic practice (65%). Most (55%) reported use of commercial AI CAD tools in their practice. Workflow prioritization based on pathology detection, injury or disease severity grading and classification, quantitative visualization, and auto-population of structured reports were identified as high-value tasks. Respondents overwhelmingly indicated a need for explainable and verifiable tools (87%) and the need for transparency in the development process (80%). Most respondents did not feel that AI would reduce the need for emergency radiologists in the next two decades (72%) or diminish interest in fellowship programs (58%). Negative perceptions pertained to potential for automation bias (23%), over-diagnosis (16%), poor generalizability (15%), negative impact on training (11%), and impediments to workflow (10%). CONCLUSION ASER member respondents are in general optimistic about the impact of AI in the practice of emergency radiology and its impact on the popularity of emergency radiology as a subspecialty. The majority expect to see transparent and explainable AI models with the radiologist as the decision-maker.
Collapse
Affiliation(s)
- Anjali Agrawal
- New Delhi operations, Teleradiology Solutions, Delhi, India
| | - Garvit D Khatri
- Nuclear Medicine, Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA
| | - Bharti Khurana
- Emergency Radiology, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Aaron D Sodickson
- Emergency Radiology, Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Yuanyuan Liang
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - David Dreizin
- Trauma and Emergency Radiology, Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
37
|
Busch F, Xu L, Sushko D, Weidlich M, Truhn D, Müller-Franzes G, Heimer MM, Niehues SM, Makowski MR, Hinsche M, Vahldiek JL, Aerts HJ, Adams LC, Bressem KK. Dual center validation of deep learning for automated multi-label segmentation of thoracic anatomy in bedside chest radiographs. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 234:107505. [PMID: 37003043 DOI: 10.1016/j.cmpb.2023.107505] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/29/2022] [Revised: 02/17/2023] [Accepted: 03/21/2023] [Indexed: 06/19/2023]
Abstract
BACKGROUND AND OBJECTIVES Bedside chest radiographs (CXRs) are challenging to interpret but important for monitoring cardiothoracic disease and invasive therapy devices in critical care and emergency medicine. Taking surrounding anatomy into account is likely to improve the diagnostic accuracy of artificial intelligence and bring its performance closer to that of a radiologist. Therefore, we aimed to develop a deep convolutional neural network for efficient automatic anatomy segmentation of bedside CXRs. METHODS To improve the efficiency of the segmentation process, we introduced a "human-in-the-loop" segmentation workflow with an active learning approach, looking at five major anatomical structures in the chest (heart, lungs, mediastinum, trachea, and clavicles). This allowed us to decrease the time needed for segmentation by 32% and select the most complex cases to utilize human expert annotators efficiently. After annotation of 2,000 CXRs from different Level 1 medical centers at Charité - University Hospital Berlin, there was no relevant improvement in model performance, and the annotation process was stopped. A 5-layer U-ResNet was trained for 150 epochs using a combined soft Dice similarity coefficient (DSC) and cross-entropy as a loss function. DSC, Jaccard index (JI), Hausdorff distance (HD) in mm, and average symmetric surface distance (ASSD) in mm were used to assess model performance. External validation was performed using an independent external test dataset from Aachen University Hospital (n = 20). RESULTS The final training, validation, and testing dataset consisted of 1900/50/50 segmentation masks for each anatomical structure. Our model achieved a mean DSC/JI/HD/ASSD of 0.93/0.88/32.1/5.8 for the lung, 0.92/0.86/21.65/4.85 for the mediastinum, 0.91/0.84/11.83/1.35 for the clavicles, 0.9/0.85/9.6/2.19 for the trachea, and 0.88/0.8/31.74/8.73 for the heart. Validation using the external dataset showed an overall robust performance of our algorithm. CONCLUSIONS Using an efficient computer-aided segmentation method with active learning, our anatomy-based model achieves comparable performance to state-of-the-art approaches. Instead of only segmenting the non-overlapping portions of the organs, as previous studies did, a closer approximation to actual anatomy is achieved by segmenting along the natural anatomical borders. This novel anatomy approach could be useful for developing pathology models for accurate and quantifiable diagnosis.
Collapse
Affiliation(s)
- Felix Busch
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany; Department of Anesthesiology, Division of Operative Intensive Care Medicine, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany.
| | - Lina Xu
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
| | - Dmitry Sushko
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
| | - Matthias Weidlich
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Maurice M Heimer
- Department of Radiology, Ludwig-Maximilians-University of Munich, Munich, Germany
| | - Stefan M Niehues
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
| | - Marcus R Makowski
- Department of Radiology, Technical University of Munich, Munich, Germany
| | - Markus Hinsche
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
| | - Janis L Vahldiek
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
| | - Hugo Jwl Aerts
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Berlin, Germany; Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA; Departments of Radiation Oncology and Radiology, Dana-Farber Cancer Institute and Brigham and Women's Hospital, Boston, MA, USA; Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University, Maastricht, the Netherlands
| | - Lisa C Adams
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany; Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Keno K Bressem
- Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany; Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Berlin, Germany; Artificial Intelligence in Medicine (AIM) Program, Mass General Brigham, Harvard Medical School, Boston, MA, USA
| |
Collapse
|
38
|
Dreizin D, Staziaki PV, Khatri GD, Beckmann NM, Feng Z, Liang Y, Delproposto ZS, Klug M, Spann JS, Sarkar N, Fu Y. Artificial intelligence CAD tools in trauma imaging: a scoping review from the American Society of Emergency Radiology (ASER) AI/ML Expert Panel. Emerg Radiol 2023; 30:251-265. [PMID: 36917287 PMCID: PMC10640925 DOI: 10.1007/s10140-023-02120-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Accepted: 02/27/2023] [Indexed: 03/16/2023]
Abstract
BACKGROUND AI/ML CAD tools can potentially improve outcomes in the high-stakes, high-volume model of trauma radiology. No prior scoping review has been undertaken to comprehensively assess tools in this subspecialty. PURPOSE To map the evolution and current state of trauma radiology CAD tools along key dimensions of technology readiness. METHODS Following a search of databases, abstract screening, and full-text document review, CAD tool maturity was charted using elements of data curation, performance validation, outcomes research, explainability, user acceptance, and funding patterns. Descriptive statistics were used to illustrate key trends. RESULTS A total of 4052 records were screened, and 233 full-text articles were selected for content analysis. Twenty-one papers described FDA-approved commercial tools, and 212 reported algorithm prototypes. Works ranged from foundational research to multi-reader multi-case trials with heterogeneous external data. Scalable convolutional neural network-based implementations increased steeply after 2016 and were used in all commercial products; however, options for explainability were narrow. Of FDA-approved tools, 9/10 performed detection tasks. Dataset sizes ranged from < 100 to > 500,000 patients, and commercialization coincided with public dataset availability. Cross-sectional torso datasets were uniformly small. Data curation methods with ground truth labeling by independent readers were uncommon. No papers assessed user acceptance, and no method included human-computer interaction. The USA and China had the highest research output and frequency of research funding. CONCLUSIONS Trauma imaging CAD tools are likely to improve patient care but are currently in an early stage of maturity, with few FDA-approved products for a limited number of uses. The scarcity of high-quality annotated data remains a major barrier.
Collapse
Affiliation(s)
- David Dreizin
- Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| | - Pedro V Staziaki
- Cardiothoracic Imaging, Department of Radiology, Larner College of Medicine, University of Vermont, Burlington, VT, USA
| | - Garvit D Khatri
- Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA
| | - Nicholas M Beckmann
- Memorial Hermann Orthopedic & Spine Hospital, McGovern Medical School at UTHealth, Houston, TX, USA
| | - Zhaoyong Feng
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Yuanyuan Liang
- Epidemiology & Public Health, University of Maryland School of Medicine, Baltimore, MD, USA
| | - Zachary S Delproposto
- Division of Emergency Radiology, Department of Radiology, University of Michigan, Ann Arbor, MI, USA
| | | | - J Stephen Spann
- Department of Radiology, University of Alabama at Birmingham Heersink School of Medicine, Birmingham, AL, USA
| | - Nathan Sarkar
- University of Maryland School of Medicine, Baltimore, MD, USA
| | - Yunting Fu
- Health Sciences and Human Services Library, University of Maryland, Baltimore, Baltimore, MD, USA
| |
Collapse
|
39
|
Rajpurkar P, Lungren MP. The Current and Future State of AI Interpretation of Medical Images. N Engl J Med 2023; 388:1981-1990. [PMID: 37224199 DOI: 10.1056/nejmra2301725] [Citation(s) in RCA: 72] [Impact Index Per Article: 72.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/26/2023]
Affiliation(s)
- Pranav Rajpurkar
- From the Department of Biomedical Informatics, Harvard Medical School, Boston (P.R.); the Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, and the Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco - both in California (M.P.L.); and Microsoft, Redmond, Washington (M.P.L.)
| | - Matthew P Lungren
- From the Department of Biomedical Informatics, Harvard Medical School, Boston (P.R.); the Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, and the Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco - both in California (M.P.L.); and Microsoft, Redmond, Washington (M.P.L.)
| |
Collapse
|
40
|
Ong ZZ, Sadek Y, Liu X, Qureshi R, Liu SH, Li T, Sounderajah V, Ashrafian H, Ting DSW, Said DG, Mehta JS, Burton MJ, Dua HS, Ting DSJ. Diagnostic performance of deep learning in infectious keratitis: a systematic review and meta-analysis protocol. BMJ Open 2023; 13:e065537. [PMID: 37164459 PMCID: PMC10173987 DOI: 10.1136/bmjopen-2022-065537] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 04/27/2023] [Indexed: 05/12/2023] Open
Abstract
INTRODUCTION Infectious keratitis (IK) represents the fifth-leading cause of blindness worldwide. A delay in diagnosis is often a major factor in progression to irreversible visual impairment and/or blindness from IK. The diagnostic challenge is further compounded by low microbiological culture yield, long turnaround time, poorly differentiated clinical features and polymicrobial infections. In recent years, deep learning (DL), a subfield of artificial intelligence, has rapidly emerged as a promising tool in assisting automated medical diagnosis, clinical triage and decision-making, and improving workflow efficiency in healthcare services. Recent studies have demonstrated the potential of using DL in assisting the diagnosis of IK, though the accuracy remains to be elucidated. This systematic review and meta-analysis aims to critically examine and compare the performance of various DL models with clinical experts and/or microbiological results (the current 'gold standard') in diagnosing IK, with an aim to inform practice on the clinical applicability and deployment of DL-assisted diagnostic models. METHODS AND ANALYSIS This review will consider studies that included application of any DL models to diagnose patients with suspected IK, encompassing bacterial, fungal, protozoal and/or viral origins. We will search various electronic databases, including EMBASE and MEDLINE, and trial registries. There will be no restriction to the language and publication date. Two independent reviewers will assess the titles, abstracts and full-text articles. Extracted data will include details of each primary studies, including title, year of publication, authors, types of DL models used, populations, sample size, decision threshold and diagnostic performance. We will perform meta-analyses for the included primary studies when there are sufficient similarities in outcome reporting. ETHICS AND DISSEMINATION No ethical approval is required for this systematic review. We plan to disseminate our findings via presentation/publication in a peer-reviewed journal. PROSPERO REGISTRATION NUMBER CRD42022348596.
Collapse
Affiliation(s)
- Zun Zheng Ong
- Department of Ophthalmology, Queen's Medical Centre, Nottingham, UK
| | - Youssef Sadek
- Department of Ophthalmology, Queen's Medical Centre, Nottingham, UK
| | - Xiaoxuan Liu
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
| | - Riaz Qureshi
- Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, USA
| | - Su-Hsun Liu
- Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, USA
| | - Tianjing Li
- Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, USA
| | - Viknesh Sounderajah
- Institute of Global Health Innovation, Imperial College London, London, UK
- Department of Surgery & Cancer, Imperial College London, London, UK
| | - Hutan Ashrafian
- Institute of Global Health Innovation, Imperial College London, London, UK
- Department of Surgery & Cancer, Imperial College London, London, UK
| | - Daniel Shu Wei Ting
- Duke-NUS Medical School, National University of Singapore, Singapore
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore
| | - Dalia G Said
- Department of Ophthalmology, Queen's Medical Centre, Nottingham, UK
- Academic Ophthalmology, School of Medicine, University of Nottingham, Nottingham, UK
- Research Institute of Ophthalmology, Cairo, Egypt
| | - Jodhbir S Mehta
- Duke-NUS Medical School, National University of Singapore, Singapore
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore
| | - Matthew J Burton
- International Centre for Eye Health, London School of Hygiene and Tropical Medicine, London, UK
- National Institute for Health Research (NIHR) Biomedical Research Centre (BRC) for Ophthalmology, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK
| | - Harminder Singh Dua
- Department of Ophthalmology, Queen's Medical Centre, Nottingham, UK
- Academic Ophthalmology, School of Medicine, University of Nottingham, Nottingham, UK
| | - Darren Shu Jeng Ting
- Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, University of Birmingham, Birmingham, UK
- Academic Ophthalmology, School of Medicine, University of Nottingham, Nottingham, UK
- Birmingham and Midland Eye Centre, Birmingham, UK
| |
Collapse
|
41
|
Kim C, Yang Z, Park SH, Hwang SH, Oh YW, Kang EY, Yong HS. Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence. Eur Radiol 2023; 33:3501-3509. [PMID: 36624227 DOI: 10.1007/s00330-022-09315-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 10/13/2022] [Accepted: 11/22/2022] [Indexed: 01/11/2023]
Abstract
OBJECTIVES To externally validate the performance of a commercial AI software program for interpreting CXRs in a large, consecutive, real-world cohort from primary healthcare centres. METHODS A total of 3047 CXRs were collected from two primary healthcare centres, characterised by low disease prevalence, between January and December 2018. All CXRs were labelled as normal or abnormal according to CT findings. Four radiology residents read all CXRs twice with and without AI assistance. The performances of the AI and readers with and without AI assistance were measured in terms of area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity. RESULTS The prevalence of clinically significant lesions was 2.2% (68 of 3047). The AUROC, sensitivity, and specificity of the AI were 0.648 (95% confidence interval [CI] 0.630-0.665), 35.3% (CI, 24.7-47.8), and 94.2% (CI, 93.3-95.0), respectively. AI detected 12 of 41 pneumonia, 3 of 5 tuberculosis, and 9 of 22 tumours. AI-undetected lesions tended to be smaller than true-positive lesions. The readers' AUROCs ranged from 0.534-0.676 without AI and 0.571-0.688 with AI (all p values < 0.05). For all readers, the mean reading time was 2.96-10.27 s longer with AI assistance (all p values < 0.05). CONCLUSIONS The performance of commercial AI in these high-volume, low-prevalence settings was poorer than expected, although it modestly boosted the performance of less-experienced readers. The technical prowess of AI demonstrated in experimental settings and approved by regulatory bodies may not directly translate to real-world practice, especially where the demand for AI assistance is highest. KEY POINTS • This study shows the limited applicability of commercial AI software for detecting abnormalities in CXRs in a health screening population. • When using AI software in a specific clinical setting that differs from the training setting, it is necessary to adjust the threshold or perform additional training with such data that reflects this environment well. • Prospective test accuracy studies, randomised controlled trials, or cohort studies are needed to examine AI software to be implemented in real clinical practice.
Collapse
Affiliation(s)
- Cherry Kim
- Department of Radiology, Ansan Hospital, Korea University College of Medicine, 123, Jeokgeum-ro, Danwon-gu, Ansan-si, Gyeonggi, 15355, South Korea
| | - Zepa Yang
- Biomedical Research Center, Guro Hospital, Korea University College of Medicine, Seoul, 08308, South Korea
| | - Seong Ho Park
- Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine, Seoul, 05505, South Korea
| | - Sung Ho Hwang
- Department of Radiology, Anam Hospital, Korea University College of Medicine, Seoul, 02841, South Korea
| | - Yu-Whan Oh
- Department of Radiology, Anam Hospital, Korea University College of Medicine, Seoul, 02841, South Korea
| | - Eun-Young Kang
- Department of Radiology, Guro Hospital, Korea University College of Medicine, 33-41, Gurodong-ro 28-gil, Guro-gu, Seoul, 08308, South Korea
| | - Hwan Seok Yong
- Department of Radiology, Guro Hospital, Korea University College of Medicine, 33-41, Gurodong-ro 28-gil, Guro-gu, Seoul, 08308, South Korea.
| |
Collapse
|
42
|
Ram S, Bodduluri S. Implementation of Artificial Intelligence-Assisted Chest X-ray Interpretation: It Is About Time. Ann Am Thorac Soc 2023; 20:641-642. [PMID: 37126001 PMCID: PMC10174129 DOI: 10.1513/annalsats.202303-195ed] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/02/2023] Open
Affiliation(s)
- Sundaresh Ram
- Department of Radiology and
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, Michigan; and
| | - Sandeep Bodduluri
- Division of Pulmonary, Allergy, and Critical Care Medicine, Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, Alabama
| |
Collapse
|
43
|
Tang JSN, Lai JKC, Bui J, Wang W, Simkin P, Gai D, Chan J, Pascoe DM, Heinze SB, Gaillard F, Lui E. Impact of Different Artificial Intelligence User Interfaces on Lung Nodule and Mass Detection on Chest Radiographs. Radiol Artif Intell 2023; 5:e220079. [PMID: 37293345 PMCID: PMC10245182 DOI: 10.1148/ryai.220079] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2022] [Revised: 02/07/2023] [Accepted: 03/02/2023] [Indexed: 06/10/2023]
Abstract
Purpose To explore the impact of different user interfaces (UIs) for artificial intelligence (AI) outputs on radiologist performance and user preference in detecting lung nodules and masses on chest radiographs. Materials and Methods A retrospective paired-reader study with a 4-week washout period was used to evaluate three different AI UIs compared with no AI output. Ten radiologists (eight radiology attending physicians and two trainees) evaluated 140 chest radiographs (81 with histologically confirmed nodules and 59 confirmed as normal with CT), with either no AI or one of three UI outputs: (a) text-only, (b) combined AI confidence score and text, or (c) combined text, AI confidence score, and image overlay. Areas under the receiver operating characteristic curve were calculated to compare radiologist diagnostic performance with each UI with their diagnostic performance without AI. Radiologists reported their UI preference. Results The area under the receiver operating characteristic curve improved when radiologists used the text-only output compared with no AI (0.87 vs 0.82; P < .001). There was no difference in performance for the combined text and AI confidence score output compared with no AI (0.77 vs 0.82; P = .46) and for the combined text, AI confidence score, and image overlay output compared with no AI (0.80 vs 0.82; P = .66). Eight of the 10 radiologists (80%) preferred the combined text, AI confidence score, and image overlay output over the other two interfaces. Conclusion Text-only UI output significantly improved radiologist performance compared with no AI in the detection of lung nodules and masses on chest radiographs, but user preference did not correspond with user performance.Keywords: Artificial Intelligence, Chest Radiograph, Conventional Radiography, Lung Nodule, Mass Detection© RSNA, 2023.
Collapse
|
44
|
Dreizin D. The American Society of Emergency Radiology (ASER) AI/ML expert panel: inception, mandate, work products, and goals. Emerg Radiol 2023; 30:279-283. [PMID: 37071272 DOI: 10.1007/s10140-023-02135-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Accepted: 04/11/2023] [Indexed: 04/19/2023]
Affiliation(s)
- David Dreizin
- Emergency and Trauma Imaging, Department of Diagnostic Radiology and Nuclear Medicine, R Adams Cowley Shock Trauma , Center, University of Maryland School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
45
|
van der Laan JJH, van der Putten JA, Zhao X, Karrenbeld A, Peters FTM, Westerhof J, de With PHN, van der Sommen F, Nagengast WB. Optical Biopsy of Dysplasia in Barrett's Oesophagus Assisted by Artificial Intelligence. Cancers (Basel) 2023; 15:cancers15071950. [PMID: 37046611 PMCID: PMC10093622 DOI: 10.3390/cancers15071950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Revised: 03/17/2023] [Accepted: 03/21/2023] [Indexed: 04/14/2023] Open
Abstract
Optical biopsy in Barrett's oesophagus (BE) using endocytoscopy (EC) could optimize endoscopic screening. However, the identification of dysplasia is challenging due to the complex interpretation of the highly detailed images. Therefore, we assessed whether using artificial intelligence (AI) as second assessor could help gastroenterologists in interpreting endocytoscopic BE images. First, we prospectively videotaped 52 BE patients with EC. Then we trained and tested the AI pm distinct datasets drawn from 83,277 frames, developed an endocytoscopic BE classification system, and designed online training and testing modules. We invited two successive cohorts for these online modules: 10 endoscopists to validate the classification system and 12 gastroenterologists to evaluate AI as second assessor by providing six of them with the option to request AI assistance. Training the endoscopists in the classification system established an improved sensitivity of 90.0% (+32.67%, p < 0.001) and an accuracy of 77.67% (+13.0%, p = 0.020) compared with the baseline. However, these values deteriorated at follow-up (-16.67%, p < 0.001 and -8.0%, p = 0.009). Contrastingly, AI-assisted gastroenterologists maintained high sensitivity and accuracy at follow-up, subsequently outperforming the unassisted gastroenterologists (+20.0%, p = 0.025 and +12.22%, p = 0.05). Thus, best diagnostic scores for the identification of dysplasia emerged through human-machine collaboration between trained gastroenterologists with AI as the second assessor. Therefore, AI could support clinical implementation of optical biopsies through EC.
Collapse
Affiliation(s)
- Jouke J H van der Laan
- Department of Gastroenterology and Hepatology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| | - Joost A van der Putten
- Department of Electrical Engineering, Video Coding and Architectures, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
| | - Xiaojuan Zhao
- Department of Gastroenterology and Hepatology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| | - Arend Karrenbeld
- Department of Pathology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| | - Frans T M Peters
- Department of Gastroenterology and Hepatology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| | - Jessie Westerhof
- Department of Gastroenterology and Hepatology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| | - Peter H N de With
- Department of Electrical Engineering, Video Coding and Architectures, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
| | - Fons van der Sommen
- Department of Electrical Engineering, Video Coding and Architectures, Eindhoven University of Technology, 5612 AZ Eindhoven, The Netherlands
| | - Wouter B Nagengast
- Department of Gastroenterology and Hepatology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
| |
Collapse
|
46
|
Domain-ensemble learning with cross-domain mixup for thoracic disease classification in unseen domains. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104488] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
|
47
|
Ahmad HK, Milne MR, Buchlak QD, Ektas N, Sanderson G, Chamtie H, Karunasena S, Chiang J, Holt X, Tang CHM, Seah JCY, Bottrell G, Esmaili N, Brotchie P, Jones C. Machine Learning Augmented Interpretation of Chest X-rays: A Systematic Review. Diagnostics (Basel) 2023; 13:diagnostics13040743. [PMID: 36832231 PMCID: PMC9955112 DOI: 10.3390/diagnostics13040743] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Revised: 02/13/2023] [Accepted: 02/14/2023] [Indexed: 02/18/2023] Open
Abstract
Limitations of the chest X-ray (CXR) have resulted in attempts to create machine learning systems to assist clinicians and improve interpretation accuracy. An understanding of the capabilities and limitations of modern machine learning systems is necessary for clinicians as these tools begin to permeate practice. This systematic review aimed to provide an overview of machine learning applications designed to facilitate CXR interpretation. A systematic search strategy was executed to identify research into machine learning algorithms capable of detecting >2 radiographic findings on CXRs published between January 2020 and September 2022. Model details and study characteristics, including risk of bias and quality, were summarized. Initially, 2248 articles were retrieved, with 46 included in the final review. Published models demonstrated strong standalone performance and were typically as accurate, or more accurate, than radiologists or non-radiologist clinicians. Multiple studies demonstrated an improvement in the clinical finding classification performance of clinicians when models acted as a diagnostic assistance device. Device performance was compared with that of clinicians in 30% of studies, while effects on clinical perception and diagnosis were evaluated in 19%. Only one study was prospectively run. On average, 128,662 images were used to train and validate models. Most classified less than eight clinical findings, while the three most comprehensive models classified 54, 72, and 124 findings. This review suggests that machine learning devices designed to facilitate CXR interpretation perform strongly, improve the detection performance of clinicians, and improve the efficiency of radiology workflow. Several limitations were identified, and clinician involvement and expertise will be key to driving the safe implementation of quality CXR machine learning systems.
Collapse
Affiliation(s)
- Hassan K. Ahmad
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Emergency Medicine, Royal North Shore Hospital, Sydney, NSW 2065, Australia
- Correspondence:
| | | | - Quinlan D. Buchlak
- Annalise.ai, Sydney, NSW 2000, Australia
- School of Medicine, University of Notre Dame Australia, Sydney, NSW 2007, Australia
- Department of Neurosurgery, Monash Health, Melbourne, VIC 3168, Australia
| | | | | | | | | | - Jason Chiang
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of General Practice, University of Melbourne, Melbourne, VIC 3010, Australia
- Westmead Applied Research Centre, University of Sydney, Sydney, NSW 2006, Australia
| | | | | | - Jarrel C. Y. Seah
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Radiology, Alfred Health, Melbourne, VIC 3004, Australia
| | | | - Nazanin Esmaili
- School of Medicine, University of Notre Dame Australia, Sydney, NSW 2007, Australia
- Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW 2007, Australia
| | - Peter Brotchie
- Annalise.ai, Sydney, NSW 2000, Australia
- Department of Radiology, St Vincent’s Health Australia, Melbourne, VIC 3065, Australia
| | - Catherine Jones
- Annalise.ai, Sydney, NSW 2000, Australia
- I-MED Radiology Network, Brisbane, QLD 4006, Australia
- School of Public and Preventive Health, Monash University, Clayton, VIC 3800, Australia
- Department of Clinical Imaging Science, University of Sydney, Sydney, NSW 2006, Australia
| |
Collapse
|
48
|
Nam JG, Hwang EJ, Kim J, Park N, Lee EH, Kim HJ, Nam M, Lee JH, Park CM, Goo JM. AI Improves Nodule Detection on Chest Radiographs in a Health Screening Population: A Randomized Controlled Trial. Radiology 2023; 307:e221894. [PMID: 36749213 DOI: 10.1148/radiol.221894] [Citation(s) in RCA: 23] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023]
Abstract
Background The impact of artificial intelligence (AI)-based computer-aided detection (CAD) software has not been prospectively explored in real-world populations. Purpose To investigate whether commercial AI-based CAD software could improve the detection rate of actionable lung nodules on chest radiographs in participants undergoing health checkups. Materials and Methods In this single-center, pragmatic, open-label randomized controlled trial, participants who underwent chest radiography between July 2020 and December 2021 in a health screening center were enrolled and randomized into intervention (AI group) and control (non-AI group) arms. One of three designated radiologists with 13-36 years of experience interpreted each radiograph, referring to the AI-based CAD results for the AI group. The primary outcome was the detection rate, that is, the number of true-positive radiographs divided by the total number of radiographs, of actionable lung nodules confirmed on CT scans obtained within 3 months. Actionable nodules were defined as solid nodules larger than 8 mm or subsolid nodules with a solid portion larger than 6 mm (Lung Imaging Reporting and Data System, or Lung-RADS, category 4). Secondary outcomes included the positive-report rate, sensitivity, false-referral rate, and malignant lung nodule detection rate. Clinical outcomes were compared between the two groups using univariable logistic regression analyses. Results A total of 10 476 participants (median age, 59 years [IQR, 50-66 years]; 5121 men) were randomized to an AI group (n = 5238) or non-AI group (n = 5238). The trial met the predefined primary outcome, demonstrating an improved detection rate of actionable nodules in the AI group compared with the non-AI group (0.59% [31 of 5238 participants] vs 0.25% [13 of 5238 participants], respectively; odds ratio, 2.4; 95% CI: 1.3, 4.7; P = .008). The detection rate for malignant lung nodules was higher in the AI group compared with the non-AI group (0.15% [eight of 5238 participants] vs 0.0% [0 of 5238 participants], respectively; P = .008). The AI and non-AI groups showed similar false-referral rates (45.9% [56 of 122 participants] vs 56.0% [56 of 100 participants], respectively; P = .14) and positive-report rates (2.3% [122 of 5238 participants] vs 1.9% [100 of 5238 participants]; P = .14). Conclusion In health checkup participants, artificial intelligence-based software improved the detection of actionable lung nodules on chest radiographs. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Auffermann in this isssue.
Collapse
Affiliation(s)
- Ju Gang Nam
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Eui Jin Hwang
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Jayoun Kim
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Nanhee Park
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Eun Hee Lee
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Hyun Jin Kim
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Miyeon Nam
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Jong Hyuk Lee
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Chang Min Park
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| | - Jin Mo Goo
- From the Department of Radiology (J.G.N., E.J.H., J.H.L., C.M.P., J.M.G.), Artificial Intelligence Collaborative Network (J.G.N.), Medical Research Collaborating Center (J.K., N.P.), and Center for Health Promotion and Optimal Aging (E.H.L., M.N.), Seoul National University Hospital and College of Medicine, 101 Daehak-ro, Jongno-gu, Seoul 03080, Republic of Korea; Department of Radiology, Ewha Womans University Seoul Hospital, Seoul, Republic of Korea (H.J.K.); Institute of Medical and Biological Engineering, Seoul National University Medical Research Center, Seoul, Republic of Korea (C.M.P.); and Cancer Research Institute, Seoul National University, Seoul, Republic of Korea (J.M.G.)
| |
Collapse
|
49
|
Towards precision medicine based on a continuous deep learning optimization and ensemble approach. NPJ Digit Med 2023; 6:18. [PMID: 36737644 PMCID: PMC9898519 DOI: 10.1038/s41746-023-00759-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Accepted: 01/17/2023] [Indexed: 02/05/2023] Open
Abstract
We developed a continuous learning system (CLS) based on deep learning and optimization and ensemble approach, and conducted a retrospective data simulated prospective study using ultrasound images of breast masses for precise diagnoses. We extracted 629 breast masses and 2235 images from 561 cases in the institution to train the model in six stages to diagnose benign and malignant tumors, pathological types, and diseases. We randomly selected 180 out of 3098 cases from two external institutions. The CLS was tested with seven independent datasets and compared with 21 physicians, and the system's diagnostic ability exceeded 20 physicians by training stage six. The optimal integrated method we developed is expected accurately diagnose breast masses. This method can also be extended to the intelligent diagnosis of masses in other organs. Overall, our findings have potential value in further promoting the application of AI diagnosis in precision medicine.
Collapse
|
50
|
Implementation of artificial intelligence in thoracic imaging-a what, how, and why guide from the European Society of Thoracic Imaging (ESTI). Eur Radiol 2023:10.1007/s00330-023-09409-2. [PMID: 36729173 PMCID: PMC9892666 DOI: 10.1007/s00330-023-09409-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2022] [Revised: 11/29/2022] [Accepted: 12/27/2022] [Indexed: 02/03/2023]
Abstract
This statement from the European Society of Thoracic imaging (ESTI) explains and summarises the essentials for understanding and implementing Artificial intelligence (AI) in clinical practice in thoracic radiology departments. This document discusses the current AI scientific evidence in thoracic imaging, its potential clinical utility, implementation and costs, training requirements and validation, its' effect on the training of new radiologists, post-implementation issues, and medico-legal and ethical issues. All these issues have to be addressed and overcome, for AI to become implemented clinically in thoracic radiology. KEY POINTS: • Assessing the datasets used for training and validation of the AI system is essential. • A departmental strategy and business plan which includes continuing quality assurance of AI system and a sustainable financial plan is important for successful implementation. • Awareness of the negative effect on training of new radiologists is vital.
Collapse
|