1
|
Baheti B, Pati S, Menze B, Bakas S. Leveraging 2D Deep Learning ImageNet-trained models for Native 3D Medical Image Analysis. BRAINLESION : GLIOMA, MULTIPLE SCLEROSIS, STROKE AND TRAUMATIC BRAIN INJURIES. BRAINLES (WORKSHOP) 2023; 13769:68-79. [PMID: 37928819 PMCID: PMC10623403 DOI: 10.1007/978-3-031-33842-7_6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2023]
Abstract
Convolutional neural networks (CNNs) have shown promising performance in various 2D computer vision tasks due to availability of large amounts of 2D training data. Contrarily, medical imaging deals with 3D data and usually lacks the equivalent extent and diversity of data, for developing AI models. Transfer learning provides the means to use models trained for one application as a starting point to another application. In this work, we leverage 2D pre-trained models as a starting point in 3D medical applications by exploring the concept of Axial-Coronal-Sagittal (ACS) convolutions. We have incorporated ACS as an alternative of native 3D convolutions in the Generally Nuanced Deep Learning Framework (GaNDLF), providing various well-established and state-of-the-art network architectures with the availability of pre-trained encoders from 2D data. Results of our experimental evaluation on 3D MRI data of brain tumor patients for i) tumor segmentation and ii) radiogenomic classification, show model size reduction by ~22% and improvement in validation accuracy by ~33%. Our findings support the advantage of ACS convolutions in pre-trained 2D CNNs over 3D CNN without pre-training, for 3D segmentation and classification tasks, democratizing existing models trained in datasets of unprecedented size and showing promise in the field of healthcare.
Collapse
Affiliation(s)
- Bhakti Baheti
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Sarthak Pati
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Informatics, Technical University of Munich, Munich, Germany
| | - Bjoern Menze
- Department of Informatics, Technical University of Munich, Munich, Germany
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
2
|
Karargyris A, Umeton R, Sheller MJ, Aristizabal A, George J, Wuest A, Pati S, Kassem H, Zenk M, Baid U, Narayana Moorthy P, Chowdhury A, Guo J, Nalawade S, Rosenthal J, Kanter D, Xenochristou M, Beutel DJ, Chung V, Bergquist T, Eddy J, Abid A, Tunstall L, Sanseviero O, Dimitriadis D, Qian Y, Xu X, Liu Y, Goh RSM, Bala S, Bittorf V, Reddy Puchala S, Ricciuti B, Samineni S, Sengupta E, Chaudhari A, Coleman C, Desinghu B, Diamos G, Dutta D, Feddema D, Fursin G, Huang X, Kashyap S, Lane N, Mallick I, Mascagni P, Mehta V, Ferro Moraes C, Natarajan V, Nikolov N, Padoy N, Pekhimenko G, Reddi VJ, Reina GA, Ribalta P, Singh A, Thiagarajan JJ, Albrecht J, Wolf T, Miller G, Fu H, Shah P, Xu D, Yadav P, Talby D, Awad MM, Howard JP, Rosenthal M, Marchionni L, Loda M, Johnson JM, Bakas S, Mattson P. Federated benchmarking of medical artificial intelligence with MedPerf. NAT MACH INTELL 2023; 5:799-810. [PMID: 38706981 PMCID: PMC11068064 DOI: 10.1038/s42256-023-00652-2] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2021] [Accepted: 04/06/2023] [Indexed: 05/07/2024]
Abstract
Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.
Collapse
Affiliation(s)
- Alexandros Karargyris
- IHU Strasbourg, Strasbourg, France
- University of Strasbourg, Strasbourg, France
- These authors contributed equally: Alexandros Karargyris, Renato Umeton, Micah J. Sheller
| | - Renato Umeton
- Dana-Farber Cancer Institute, Boston, MA, USA
- Weill Cornell Medicine, New York, NY, USA
- Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Massachusetts Institute of Technology, Cambridge, MA, USA
- These authors contributed equally: Alexandros Karargyris, Renato Umeton, Micah J. Sheller
| | - Micah J. Sheller
- Intel, Santa Clara, CA, USA
- These authors contributed equally: Alexandros Karargyris, Renato Umeton, Micah J. Sheller
| | | | | | - Anna Wuest
- Dana-Farber Cancer Institute, Boston, MA, USA
- Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Sarthak Pati
- Perelman School of Medicine, Philadelphia, PA, USA
- University of Pennsylvania, Philadelphia, PA, USA
| | | | - Maximilian Zenk
- German Cancer Research Center, Heidelberg, Germany
- University of Heidelberg, Heidelberg, Germany
| | - Ujjwal Baid
- Perelman School of Medicine, Philadelphia, PA, USA
- University of Pennsylvania, Philadelphia, PA, USA
| | | | | | - Junyi Guo
- Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | | | - Jacob Rosenthal
- Dana-Farber Cancer Institute, Boston, MA, USA
- Weill Cornell Medicine, New York, NY, USA
| | | | | | - Daniel J. Beutel
- University of Cambridge, Cambridge, UK
- Flower Labs, Hamburg, Germany
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Akshay Chaudhari
- Stanford University, Stanford, CA, USA
- Stanford University School of Medicine, Stanford, CA, USA
| | | | | | | | | | | | | | | | | | - Nicholas Lane
- University of Cambridge, Cambridge, UK
- Flower Labs, Hamburg, Germany
| | | | | | | | | | - Pietro Mascagni
- IHU Strasbourg, Strasbourg, France
- Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | | | | | | | | | - Nicolas Padoy
- IHU Strasbourg, Strasbourg, France
- University of Strasbourg, Strasbourg, France
| | - Gennady Pekhimenko
- University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | | | | | | | - Abhishek Singh
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | | | | | | | | | | | | | | | | | | | - Mark M. Awad
- Dana-Farber Cancer Institute, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Jeremy P. Howard
- fast.ai, San Francisco, CA, USA
- University of Queensland, Brisbane, Queensland, Australia
| | - Michael Rosenthal
- Dana-Farber Cancer Institute, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
- Brigham and Women’s Hospital, Boston, MA, USA
| | | | - Massimo Loda
- Dana-Farber Cancer Institute, Boston, MA, USA
- Weill Cornell Medicine, New York, NY, USA
- Harvard Medical School, Boston, MA, USA
- Broad Institute of MIT and Harvard, Cambridge, MA, USA
| | | | - Spyridon Bakas
- Perelman School of Medicine, Philadelphia, PA, USA
- University of Pennsylvania, Philadelphia, PA, USA
- These authors jointly supervised this work: Spyridon Bakas, Peter Mattson
| | - Peter Mattson
- MLCommons, San Francisco, CA, USA
- Google, Mountain View, CA, USA
- These authors jointly supervised this work: Spyridon Bakas, Peter Mattson
| |
Collapse
|
3
|
Foley P, Sheller MJ, Edwards B, Pati S, Riviera W, Sharma M, Narayana Moorthy P, Wang SH, Martin J, Mirhaji P, Shah P, Bakas S. OpenFL: the open federated learning library. Phys Med Biol 2022; 67:214001. [PMID: 36198326 PMCID: PMC9715347 DOI: 10.1088/1361-6560/ac97d9] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Accepted: 10/05/2022] [Indexed: 11/07/2022]
Abstract
Objective.Federated learning (FL) is a computational paradigm that enables organizations to collaborate on machine learning (ML) and deep learning (DL) projects without sharing sensitive data, such as patient records, financial data, or classified secrets.Approach.Open federated learning (OpenFL) framework is an open-source python-based tool for training ML/DL algorithms using the data-private collaborative learning paradigm of FL, irrespective of the use case. OpenFL works with training pipelines built with both TensorFlow and PyTorch, and can be easily extended to other ML and DL frameworks.Main results.In this manuscript, we present OpenFL and summarize its motivation and development characteristics, with the intention of facilitating its application to existing ML/DL model training in a production environment. We further provide recommendations to secure a federation using trusted execution environments to ensure explicit model security and integrity, as well as maintain data confidentiality. Finally, we describe the first real-world healthcare federations that use the OpenFL library, and highlight how it can be applied to other non-healthcare use cases.Significance.The OpenFL library is designed for real world scalability, trusted execution, and also prioritizes easy migration of centralized ML models into a federated training pipeline. Although OpenFL's initial use case was in healthcare, it is applicable beyond this domain and is now reaching wider adoption both in research and production settings. The tool is open-sourced atgithub.com/intel/openfl.
Collapse
Affiliation(s)
- Patrick Foley
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Micah J Sheller
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Brandon Edwards
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Sarthak Pati
- University of Pennsylvania, 3700 Hamilton Walk, Richards Medical Research Laboratories (7th Fl), Philadelphia, PA 19104, United States of America
| | - Walter Riviera
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Mansi Sharma
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | | | - Shih-Han Wang
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Jason Martin
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Parsa Mirhaji
- Albert Einstein College of Medicine, 1300 Morris Park Ave, Bronx, NY 10461, United States of America
| | - Prashant Shah
- Intel Corporation, Santa Clara, CA 95052, United States of America
| | - Spyridon Bakas
- University of Pennsylvania, 3700 Hamilton Walk, Richards Medical Research Laboratories (7th Fl), Philadelphia, PA 19104, United States of America
| |
Collapse
|
4
|
Pati S, Baid U, Edwards B, Sheller MJ, Foley P, Reina GA, Thakur S, Sako C, Bilello M, Davatzikos C, Martin J, Shah P, Menze B, Bakas S. The federated tumor segmentation (FeTS) tool: an open-source solution to further solid tumor research. Phys Med Biol 2022; 67:10.1088/1361-6560/ac9449. [PMID: 36137534 PMCID: PMC9592188 DOI: 10.1088/1361-6560/ac9449] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Accepted: 09/22/2022] [Indexed: 11/11/2022]
Abstract
Objective.De-centralized data analysis becomes an increasingly preferred option in the healthcare domain, as it alleviates the need for sharing primary patient data across collaborating institutions. This highlights the need for consistent harmonized data curation, pre-processing, and identification of regions of interest based on uniform criteria.Approach.Towards this end, this manuscript describes theFederatedTumorSegmentation (FeTS) tool, in terms of software architecture and functionality.Main results.The primary aim of the FeTS tool is to facilitate this harmonized processing and the generation of gold standard reference labels for tumor sub-compartments on brain magnetic resonance imaging, and further enable federated training of a tumor sub-compartment delineation model across numerous sites distributed across the globe, without the need to share patient data.Significance.Building upon existing open-source tools such as the Insight Toolkit and Qt, the FeTS tool is designed to enable training deep learning models targeting tumor delineation in either centralized or federated settings. The target audience of the FeTS tool is primarily the computational researcher interested in developing federated learning models, and interested in joining a global federation towards this effort. The tool is open sourced athttps://github.com/FETS-AI/Front-End.
Collapse
Affiliation(s)
- Sarthak Pati
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Informatics, Technical University of Munich, Munich, Germany
| | - Ujjwal Baid
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | | | | | | | | | - Siddhesh Thakur
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Chiharu Sako
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Michel Bilello
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Christos Davatzikos
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | | | | | - Bjoern Menze
- Department of Informatics, Technical University of Munich, Munich, Germany
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|