1
|
Shalom ES, Kim H, van der Heijden RA, Ahmed Z, Patel R, Hormuth DA, DiCarlo JC, Yankeelov TE, Sisco NJ, Dortch RD, Stokes AM, Inglese M, Grech-Sollars M, Toschi N, Sahoo P, Singh A, Verma SK, Rathore DK, Kazerouni AS, Partridge SC, LoCastro E, Paudyal R, Wolansky IA, Shukla-Dave A, Schouten P, Gurney-Champion OJ, Jiřík R, Macíček O, Bartoš M, Vitouš J, Das AB, Kim SG, Bokacheva L, Mikheev A, Rusinek H, Berks M, Hubbard Cristinacce PL, Little RA, Cheung S, O'Connor JPB, Parker GJM, Moloney B, LaViolette PS, Bobholz S, Duenweg S, Virostko J, Laue HO, Sung K, Nabavizadeh A, Saligheh Rad H, Hu LS, Sourbron S, Bell LC, Fathi Kazerooni A. The ISMRM Open Science Initiative for Perfusion Imaging (OSIPI): Results from the OSIPI-Dynamic Contrast-Enhanced challenge. Magn Reson Med 2024; 91:1803-1821. [PMID: 38115695 DOI: 10.1002/mrm.29909] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Revised: 08/22/2023] [Accepted: 10/16/2023] [Indexed: 12/21/2023]
Abstract
PURPOSE K trans $$ {K}^{\mathrm{trans}} $$ has often been proposed as a quantitative imaging biomarker for diagnosis, prognosis, and treatment response assessment for various tumors. None of the many software tools forK trans $$ {K}^{\mathrm{trans}} $$ quantification are standardized. The ISMRM Open Science Initiative for Perfusion Imaging-Dynamic Contrast-Enhanced (OSIPI-DCE) challenge was designed to benchmark methods to better help the efforts to standardizeK trans $$ {K}^{\mathrm{trans}} $$ measurement. METHODS A framework was created to evaluateK trans $$ {K}^{\mathrm{trans}} $$ values produced by DCE-MRI analysis pipelines to enable benchmarking. The perfusion MRI community was invited to apply their pipelines forK trans $$ {K}^{\mathrm{trans}} $$ quantification in glioblastoma from clinical and synthetic patients. Submissions were required to include the entrants'K trans $$ {K}^{\mathrm{trans}} $$ values, the applied software, and a standard operating procedure. These were evaluated using the proposedOSIP I gold $$ \mathrm{OSIP}{\mathrm{I}}_{\mathrm{gold}} $$ score defined with accuracy, repeatability, and reproducibility components. RESULTS Across the 10 received submissions, theOSIP I gold $$ \mathrm{OSIP}{\mathrm{I}}_{\mathrm{gold}} $$ score ranged from 28% to 78% with a 59% median. The accuracy, repeatability, and reproducibility scores ranged from 0.54 to 0.92, 0.64 to 0.86, and 0.65 to 1.00, respectively (0-1 = lowest-highest). Manual arterial input function selection markedly affected the reproducibility and showed greater variability inK trans $$ {K}^{\mathrm{trans}} $$ analysis than automated methods. Furthermore, provision of a detailed standard operating procedure was critical for higher reproducibility. CONCLUSIONS This study reports results from the OSIPI-DCE challenge and highlights the high inter-software variability withinK trans $$ {K}^{\mathrm{trans}} $$ estimation, providing a framework for ongoing benchmarking against the scores presented. Through this challenge, the participating teams were ranked based on the performance of their software tools in the particular setting of this challenge. In a real-world clinical setting, many of these tools may perform differently with different benchmarking methodology.
Collapse
Affiliation(s)
- Eve S Shalom
- School of Physics and Astronomy, University of Leeds, Leeds, UK
- Department of Infection, Immunity and Cardiovascular Disease, University of Sheffield, Sheffield, UK
| | - Harrison Kim
- Department of Radiology, University of Alabama, Birmingham, Alabama, USA
| | - Rianne A van der Heijden
- Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands
- Department of Radiology, University of Wisconsin-Madison, Madison, Wisconsin, USA
| | - Zaki Ahmed
- Corewell Health William Beaumont University Hospital, Royal Oak, Michigan, USA
| | - Reyna Patel
- Department of Radiology, Neuroradiology Division, Mayo Clinic, Scottsdale, Arizona, USA
| | - David A Hormuth
- Oden Institute for Computational Engineering and Sciences, The University of Texas, Austin, Texas, USA
| | - Julie C DiCarlo
- Biomedical Imaging Center, Livestrong Cancer Institutes, University of Texas at Austin, Austin, Texas, USA
| | - Thomas E Yankeelov
- Departments of Biomedical Engineering, Diagnostic Medicine, Oncology, Livestrong Cancer Institutes, Oden Institute for Computational Engineering and Sciences, The University of Texas, Austin, Texas, USA
- Department of Imaging Physics, MD Anderson Cancer Center, Houston, Texas, USA
| | - Nicholas J Sisco
- Department of Translational Neuroscience, Barrow Neurological Institute, Phoenix, Arizona, USA
| | - Richard D Dortch
- Department of Translational Neuroscience, Barrow Neurological Institute, Phoenix, Arizona, USA
| | - Ashley M Stokes
- Department of Translational Neuroscience, Barrow Neurological Institute, Phoenix, Arizona, USA
| | - Marianna Inglese
- Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Italy
- Department of Surgery and Cancer, Imperial College, London, UK
| | - Matthew Grech-Sollars
- Department of Surgery and Cancer, Imperial College, London, UK
- Department of Computer Science, University College London, London, UK
- Lysholm Department of Neuroradiology, National Hospital for Neurology and Neurosurgery, University College London Hospitals NHS Foundation Trust, London, UK
| | - Nicola Toschi
- Department of Biomedicine and Prevention, University of Rome, Tor Vergata, Italy
- Athinoula A. Martinos Center for Biomedical Imaging, Harvard Medical School, Boston, Massachusetts, USA
| | - Prativa Sahoo
- University Medical Center Göttingen, Göttingen, Germany
| | - Anup Singh
- Center for Biomedical Engineering, Indian Institute of Technology Delhi, New Delhi, India
| | - Sanjay K Verma
- Institute of Bioengineering and Bioimaging, Singapore, Singapore
| | - Divya K Rathore
- Institute of Psychiatry, Psychology & Neuroscience, King's College, London, UK
| | - Anum S Kazerouni
- Department of Radiology, University of Washington, Seattle, Washington, USA
| | | | - Eve LoCastro
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Ramesh Paudyal
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Ivan A Wolansky
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Amita Shukla-Dave
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
- Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Pepijn Schouten
- Department of Radiology and Nuclear Medicine, University of Amsterdam, Amsterdam, The Netherlands
| | - Oliver J Gurney-Champion
- Department of Radiology and Nuclear Medicine, University of Amsterdam, Amsterdam, The Netherlands
- Cancer Center Amsterdam, Imaging and Biomarkers, Amsterdam, The Netherlands
| | - Radovan Jiřík
- Czech Academy of Sciences, Institute of Scientific Instruments, Brno, Czech Republic
| | - Ondřej Macíček
- Czech Academy of Sciences, Institute of Scientific Instruments, Brno, Czech Republic
| | - Michal Bartoš
- Czech Academy of Sciences, Institute of Information Theory and Automation, Praha, Czech Republic
| | - Jiří Vitouš
- Czech Academy of Sciences, Institute of Scientific Instruments, Brno, Czech Republic
| | | | - S Gene Kim
- Department of Radiology, Weill Cornell Medical College, New York, New York, USA
| | - Louisa Bokacheva
- Department of Radiology, Grossman School of Medicine, New York University, New York, New York, USA
| | - Artem Mikheev
- Department of Radiology, Grossman School of Medicine, New York University, New York, New York, USA
| | - Henry Rusinek
- Department of Radiology, Grossman School of Medicine, New York University, New York, New York, USA
| | - Michael Berks
- Division of Cancer Sciences, University of Manchester, Manchester, UK
| | | | - Ross A Little
- Division of Cancer Sciences, University of Manchester, Manchester, UK
| | - Susan Cheung
- Division of Cancer Sciences, University of Manchester, Manchester, UK
| | - James P B O'Connor
- Division of Cancer Sciences, University of Manchester, Manchester, UK
- Department of Radiology, The Christie Hospital NHS Trust, Manchester, UK
- Division of Radiotherapy and Imaging, The Institute of Cancer Research, London, UK
| | - Geoff J M Parker
- Center for Medical Image Computing, Department of Medical Physics and Biomedical Engineering, University College London, London, UK
- Bioxydyn Ltd, Manchester, UK
| | - Brendan Moloney
- Advanced Imaging Research Center, Oregon Health & Science Institute, Portland, Oregon, USA
| | - Peter S LaViolette
- Department of Radiology, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Samuel Bobholz
- Department of Radiology, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Savannah Duenweg
- Department of Radiology, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - John Virostko
- Department of Diagnostic Medicine, University of Texas, Austin, Texas, USA
| | - Hendrik O Laue
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Kyunghyun Sung
- Department of Radiological Sciences, University of California, Los Angeles, California, USA
| | - Ali Nabavizadeh
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Center for Data-Driven Discovery, Division of Neurosurgery, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, USA
| | - Hamidreza Saligheh Rad
- Quantitative MR Imaging and Spectroscopy Group, Research Center for Molecular and Cellular Imaging, Tehran University of Medical Sciences, Tehran, Iran
- Center for Computational Imaging & Simulation Technologies in Biomedicine, School of Computing/School of Medicine, University of Leeds, Leeds, UK
| | - Leland S Hu
- Neuroradiology Division, Department of Radiology, Mayo Clinic, Phoenix, Arizona, USA
| | - Steven Sourbron
- Department of Infection, Immunity and Cardiovascular Disease, University of Sheffield, Sheffield, UK
| | - Laura C Bell
- Clinical Imaging Group, Genentech, Inc., South San Francisco, California, USA
| | - Anahita Fathi Kazerooni
- Quantitative MR Imaging and Spectroscopy Group, Research Center for Molecular and Cellular Imaging, Tehran University of Medical Sciences, Tehran, Iran
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Pennsylvania, USA
| |
Collapse
|