1
|
Viar-Hernandez D, Molina-Maza JM, Vera-Sánchez JA, Perez-Moreno JM, Mazal A, Rodriguez-Vila B, Malpica N, Torrado-Carvajal A. Enhancing adaptive proton therapy through CBCT images: Synthetic head and neck CT generation based on 3D vision transformers. Med Phys 2024; 51:4922-4935. [PMID: 38569141 DOI: 10.1002/mp.17057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2023] [Revised: 03/01/2024] [Accepted: 03/17/2024] [Indexed: 04/05/2024] Open
Abstract
BACKGROUND Proton therapy is a form of radiotherapy commonly used to treat various cancers. Due to its high conformality, minor variations in patient anatomy can lead to significant alterations in dose distribution, making adaptation crucial. While cone-beam computed tomography (CBCT) is a well-established technique for adaptive radiation therapy (ART), it cannot be directly used for adaptive proton therapy (APT) treatments because the stopping power ratio (SPR) cannot be estimated from CBCT images. PURPOSE To address this limitation, Deep Learning methods have been suggested for converting pseudo-CT (pCT) images from CBCT images. In spite of convolutional neural networks (CNNs) have shown consistent improvement in pCT literature, there is still a need for further enhancements to make them suitable for clinical applications. METHODS The authors introduce the 3D vision transformer (ViT) block, studying its performance at various stages of the proposed architectures. Additionally, they conduct a retrospective analysis of a dataset that includes 259 image pairs from 59 patients who underwent treatment for head and neck cancer. The dataset is partitioned into 80% for training, 10% for validation, and 10% for testing purposes. RESULTS The SPR maps obtained from the pCT using the proposed method present an absolute relative error of less than 5% from those computed from the planning CT, thus improving the results of CBCT. CONCLUSIONS We introduce an enhanced ViT3D architecture for pCT image generation from CBCT images, reducing SPR error within clinical margins for APT workflows. The new method minimizes bias compared to CT-based SPR estimation and dose calculation, signaling a promising direction for future research in this field. However, further research is needed to assess the robustness and generalizability across different medical imaging applications.
Collapse
Affiliation(s)
- David Viar-Hernandez
- Universidad Rey Juan Carlos, Medical Image Analysis and Biometry Laboratory, Madrid, Spain
| | | | | | | | - Alejandro Mazal
- Centro de Protonterapia Quironsalud, Servicio de física médica, Madrid, Spain
| | - Borja Rodriguez-Vila
- Universidad Rey Juan Carlos, Medical Image Analysis and Biometry Laboratory, Madrid, Spain
| | - Norberto Malpica
- Universidad Rey Juan Carlos, Medical Image Analysis and Biometry Laboratory, Madrid, Spain
| | - Angel Torrado-Carvajal
- Universidad Rey Juan Carlos, Medical Image Analysis and Biometry Laboratory, Madrid, Spain
| |
Collapse
|
2
|
Emin S, Rossi E, Myrvold Rooth E, Dorniok T, Hedman M, Gagliardi G, Villegas F. Clinical implementation of a commercial synthetic computed tomography solution for radiotherapy treatment of glioblastoma. Phys Imaging Radiat Oncol 2024; 30:100589. [PMID: 38818305 PMCID: PMC11137592 DOI: 10.1016/j.phro.2024.100589] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Revised: 05/12/2024] [Accepted: 05/13/2024] [Indexed: 06/01/2024] Open
Abstract
Background and Purpose Magnetic resonance (MR)-only radiotherapy (RT) workflow eliminates uncertainties due to computed tomography (CT)-MR image registration, by using synthetic CT (sCT) images generated from MR. This study describes the clinical implementation process, from retrospective commissioning to prospective validation stage of a commercial artificial intelligence (AI)-based sCT product. Evaluation of the dosimetric performance of the sCT is presented, with emphasis on the impact of voxel size differences between image modalities. Materials and methods sCT performance was assessed in glioblastoma RT planning. Dose differences for 30 patients in both commissioning and validation cohorts were calculated at various dose-volume-histogram (DVH) points for target and organs-at-risk (OAR). A gamma analysis was conducted on regridded image plans. Quality assurance (QA) guidelines were established based on commissioning phase results. Results Mean dose difference to target structures was found to be within ± 0.7 % regardless of image resolution and cohort. OARs' mean dose differences were within ± 1.3 % for plans calculated on regridded images for both cohorts, while differences were higher for plans with original voxel size, reaching up to -4.2 % for chiasma D2% in the commissioning cohort. Gamma passing rates for the brain structure using the criteria 1 %/1mm, 2 %/2mm and 3 %/3mm were 93.6 %/99.8 %/100 % and 96.6 %/99.9 %/100 % for commissioning and validation cohorts, respectively. Conclusions Dosimetric outcomes in both commissioning and validation stages confirmed sCT's equivalence to CT. The large patient cohort in this study aided in establishing a robust QA program for the MR-only workflow, now applied in glioblastoma RT at our center.
Collapse
Affiliation(s)
- Sevgi Emin
- Department of Medical Radiation Physics and Nuclear Medicine, Karolinska University Hospital, 171 76 Stockholm, Sweden
| | - Elia Rossi
- Department of Radiation Oncology, Karolinska University Hospital, 171 76 Stockholm, Sweden
| | | | - Torsten Dorniok
- Department of Medical Radiation Physics and Nuclear Medicine, Karolinska University Hospital, 171 76 Stockholm, Sweden
| | - Mattias Hedman
- Department of Radiation Oncology, Karolinska University Hospital, 171 76 Stockholm, Sweden
- Department of Oncology-Pathology, Karolinska Institute, 171 77 Stockholm, Sweden
| | - Giovanna Gagliardi
- Department of Medical Radiation Physics and Nuclear Medicine, Karolinska University Hospital, 171 76 Stockholm, Sweden
- Department of Oncology-Pathology, Karolinska Institute, 171 77 Stockholm, Sweden
| | - Fernanda Villegas
- Department of Medical Radiation Physics and Nuclear Medicine, Karolinska University Hospital, 171 76 Stockholm, Sweden
- Department of Oncology-Pathology, Karolinska Institute, 171 77 Stockholm, Sweden
| |
Collapse
|
3
|
Sherwani MK, Gopalakrishnan S. A systematic literature review: deep learning techniques for synthetic medical image generation and their applications in radiotherapy. FRONTIERS IN RADIOLOGY 2024; 4:1385742. [PMID: 38601888 PMCID: PMC11004271 DOI: 10.3389/fradi.2024.1385742] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Accepted: 03/11/2024] [Indexed: 04/12/2024]
Abstract
The aim of this systematic review is to determine whether Deep Learning (DL) algorithms can provide a clinically feasible alternative to classic algorithms for synthetic Computer Tomography (sCT). The following categories are presented in this study: ∙ MR-based treatment planning and synthetic CT generation techniques. ∙ Generation of synthetic CT images based on Cone Beam CT images. ∙ Low-dose CT to High-dose CT generation. ∙ Attenuation correction for PET images. To perform appropriate database searches, we reviewed journal articles published between January 2018 and June 2023. Current methodology, study strategies, and results with relevant clinical applications were analyzed as we outlined the state-of-the-art of deep learning based approaches to inter-modality and intra-modality image synthesis. This was accomplished by contrasting the provided methodologies with traditional research approaches. The key contributions of each category were highlighted, specific challenges were identified, and accomplishments were summarized. As a final step, the statistics of all the cited works from various aspects were analyzed, which revealed that DL-based sCTs have achieved considerable popularity, while also showing the potential of this technology. In order to assess the clinical readiness of the presented methods, we examined the current status of DL-based sCT generation.
Collapse
Affiliation(s)
- Moiz Khan Sherwani
- Section for Evolutionary Hologenomics, Globe Institute, University of Copenhagen, Copenhagen, Denmark
| | | |
Collapse
|
4
|
Gianoli C, De Bernardi E, Parodi K. "Under the hood": artificial intelligence in personalized radiotherapy. BJR Open 2024; 6:tzae017. [PMID: 39104573 PMCID: PMC11299549 DOI: 10.1093/bjro/tzae017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 05/10/2024] [Accepted: 07/10/2024] [Indexed: 08/07/2024] Open
Abstract
This review presents and discusses the ways in which artificial intelligence (AI) tools currently intervene, or could potentially intervene in the future, to enhance the diverse tasks involved in the radiotherapy workflow. The radiotherapy framework is presented on 2 different levels for the personalization of the treatment, distinct in tasks and methodologies. The first level is the clinically well-established anatomy-based workflow, known as adaptive radiation therapy. The second level is referred to as biology-driven workflow, explored in the research literature and recently appearing in some preliminary clinical trials for personalized radiation treatments. A 2-fold role for AI is defined according to these 2 different levels. In the anatomy-based workflow, the role of AI is to streamline and improve the tasks in terms of time and variability reductions compared to conventional methodologies. The biology-driven workflow instead fully relies on AI, which introduces decision-making tools opening uncharted frontiers that were in the past deemed challenging to explore. These methodologies are referred to as radiomics and dosiomics, handling imaging and dosimetric information, or multiomics, when complemented by clinical and biological parameters (ie, biomarkers). The review explicitly highlights the methodologies that are currently incorporated into clinical practice or still in research, with the aim of presenting the AI's growing role in personalized radiotherapy.
Collapse
Affiliation(s)
- Chiara Gianoli
- Department of Experimental Physics – Medical Physics, Faculty for Physics of the Ludwig-Maximilians-Universität München (LMU Munich), Geschwister-Scholl-Platz 1, München, 80539, Germany
| | - Elisabetta De Bernardi
- School of Medicine and Surgery, Università degli Studi di Milano-Bicocca, Piazza dell'Ateneo Nuovo 1, Milano, 20126, Italy
| | - Katia Parodi
- Department of Experimental Physics – Medical Physics, Faculty for Physics of the Ludwig-Maximilians-Universität München (LMU Munich), Geschwister-Scholl-Platz 1, München, 80539, Germany
| |
Collapse
|
5
|
Park TY, Koh H, Lee W, Park SH, Chang WS, Kim H. Real-Time Acoustic Simulation Framework for tFUS: A Feasibility Study Using Navigation System. Neuroimage 2023; 282:120411. [PMID: 37844771 DOI: 10.1016/j.neuroimage.2023.120411] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2023] [Revised: 10/10/2023] [Accepted: 10/13/2023] [Indexed: 10/18/2023] Open
Abstract
Transcranial focused ultrasound (tFUS), in which acoustic energy is focused on a small region in the brain through the skull, is a non-invasive therapeutic method with high spatial resolution and depth penetration. Image-guided navigation has been widely utilized to visualize the location of acoustic focus in the cranial cavity. However, this system is often inaccurate because of the significant aberrations caused by the skull. Therefore, acoustic simulations using a numerical solver have been widely adopted to compensate for this inaccuracy. Although the simulation can predict the intracranial acoustic pressure field, real-time application during tFUS treatment is almost impossible due to the high computational cost. In this study, we propose a neural network-based real-time acoustic simulation framework and test its feasibility by implementing a simulation-guided navigation (SGN) system. Real-time acoustic simulation is performed using a 3D conditional generative adversarial network (3D-cGAN) model featuring residual blocks and multiple loss functions. This network was trained by the conventional numerical acoustic simulation program (i.e., k-Wave). The SGN system is then implemented by integrating real-time acoustic simulation with a conventional image-guided navigation system. The proposed system can provide simulation results with a frame rate of 5 Hz (i.e., about 0.2 s), including all processing times. In numerical validation (3D-cGAN vs. k-Wave), the average peak intracranial pressure error was 6.8 ± 5.5%, and the average acoustic focus position error was 5.3 ± 7.7 mm. In experimental validation using a skull phantom (3D-cGAN vs. actual measurement), the average peak intracranial pressure error was 4.5%, and the average acoustic focus position error was 6.6 mm. These results demonstrate that the SGN system can predict the intracranial acoustic field according to transducer placement in real-time.
Collapse
Affiliation(s)
- Tae Young Park
- Bionics Research Center, Biomedical Research Division, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea; Division of Bio-Medical Science and Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Republic of Korea
| | - Heekyung Koh
- Bionics Research Center, Biomedical Research Division, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea
| | - Wonhye Lee
- Bionics Research Center, Biomedical Research Division, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea; Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - So Hee Park
- Department of Neurosurgery, Yeungnam University Medical Center, Daegu 42415, Republic of Korea
| | - Won Seok Chang
- Department of Neurosurgery, Brain Research Institute, Yonsei University College of Medicine, Seoul 04527, Republic of Korea
| | - Hyungmin Kim
- Bionics Research Center, Biomedical Research Division, Korea Institute of Science and Technology, Seoul 02792, Republic of Korea; Division of Bio-Medical Science and Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Republic of Korea.
| |
Collapse
|
6
|
Tian L, Lühr A. Proton range uncertainty caused by synthetic computed tomography generated with deep learning from pelvic magnetic resonance imaging. Acta Oncol 2023; 62:1461-1469. [PMID: 37703314 DOI: 10.1080/0284186x.2023.2256967] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Accepted: 09/04/2023] [Indexed: 09/15/2023]
Abstract
BACKGROUND In proton therapy, it is disputed whether synthetic computed tomography (sCT), derived from magnetic resonance imaging (MRI), permits accurate dose calculations. On the one hand, an MRI-only workflow could eliminate errors caused by, e.g., MRI-CT registration. On the other hand, the extra error would be induced due to an sCT generation model. This work investigated the systematic and random model error induced by sCT generation of a widely discussed deep learning model, pix2pix. MATERIAL AND METHODS An open-source image dataset of 19 patients with cancer in the pelvis was employed and split into 10, 5, and 4 for training, testing, and validation of the model, respectively. Proton pencil beams (200 MeV) were simulated on the real CT and generated sCT using the tool for particle simulation (TOPAS). Monte Carlo (MC) dropout was used for error estimation (50 random sCT samples). Systematic and random model errors were investigated for sCT generation and dose calculation on sCT. RESULTS For sCT generation, random model error near the edge of the body (∼200 HU) was higher than that within the body (∼100 HU near the bone edge and <10 HU in soft tissue). The mean absolute error (MAE) was 49 ± 5, 191 ± 23, and 503 ± 70 HU for the whole body, bone, and air in the patient, respectively. Random model errors of the proton range were small (<0.2 mm) for all spots and evenly distributed throughout the proton fields. Systematic errors of the proton range were -1.0(±2.2) mm and 0.4(±0.9)%, respectively, and were unevenly distributed within the proton fields. For 4.5% of the spots, large errors (>5 mm) were found, which may relate to MRI-CT mismatch due to, e.g., registration, MRI distortion anatomical changes, etc. CONCLUSION The sCT model was shown to be robust, i.e., had a low random model error. However, further investigation to reduce and even predict and manage systematic error is still needed for future MRI-only proton therapy.
Collapse
Affiliation(s)
- Liheng Tian
- Department of Physics, TU Dortmund University, Dortmund, Germany
| | - Armin Lühr
- Department of Physics, TU Dortmund University, Dortmund, Germany
| |
Collapse
|
7
|
Kaushik SS, Bylund M, Cozzini C, Shanbhag D, Petit SF, Wyatt JJ, Menzel MI, Pirkl C, Mehta B, Chauhan V, Chandrasekharan K, Jonsson J, Nyholm T, Wiesinger F, Menze B. Region of interest focused MRI to synthetic CT translation using regression and segmentation multi-task network. Phys Med Biol 2023; 68:195003. [PMID: 37567235 DOI: 10.1088/1361-6560/acefa3] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2022] [Accepted: 08/10/2023] [Indexed: 08/13/2023]
Abstract
Objective. In MR-only clinical workflow, replacing CT with MR image is of advantage for workflow efficiency and reduces radiation to the patient. An important step required to eliminate CT scan from the workflow is to generate the information provided by CT via an MR image. In this work, we aim to demonstrate a method to generate accurate synthetic CT (sCT) from an MR image to suit the radiation therapy (RT) treatment planning workflow. We show the feasibility of the method and make way for a broader clinical evaluation.Approach. We present a machine learning method for sCT generation from zero-echo-time (ZTE) MRI aimed at structural and quantitative accuracies of the image, with a particular focus on the accurate bone density value prediction. The misestimation of bone density in the radiation path could lead to unintended dose delivery to the target volume and results in suboptimal treatment outcome. We propose a loss function that favors a spatially sparse bone region in the image. We harness the ability of the multi-task network to produce correlated outputs as a framework to enable localization of region of interest (RoI) via segmentation, emphasize regression of values within RoI and still retain the overall accuracy via global regression. The network is optimized by a composite loss function that combines a dedicated loss from each task.Main results. We have included 54 brain patient images in this study and tested the sCT images against reference CT on a subset of 20 cases. A pilot dose evaluation was performed on 9 of the 20 test cases to demonstrate the viability of the generated sCT in RT planning. The average quantitative metrics produced by the proposed method over the test set were-(a) mean absolute error (MAE) of 70 ± 8.6 HU; (b) peak signal-to-noise ratio (PSNR) of 29.4 ± 2.8 dB; structural similarity metric (SSIM) of 0.95 ± 0.02; and (d) Dice coefficient of the body region of 0.984 ± 0.Significance. We demonstrate that the proposed method generates sCT images that resemble visual characteristics of a real CT image and has a quantitative accuracy that suits RT dose planning application. We compare the dose calculation from the proposed sCT and the real CT in a radiation therapy treatment planning setup and show that sCT based planning falls within 0.5% target dose error. The method presented here with an initial dose evaluation makes an encouraging precursor to a broader clinical evaluation of sCT based RT planning on different anatomical regions.
Collapse
Affiliation(s)
- Sandeep S Kaushik
- GE Healthcare, Munich, Germany
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Mikael Bylund
- Department of Radiation Sciences, UmeåUniversity, Umea, Sweden
| | | | | | - Steven F Petit
- Department of Radiotherapy, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Jonathan J Wyatt
- Translational and Clinical Research Institute, Newcastle University and Northern Centre for Cancer Care, Newcastle upon Tyne Hospitals NHS Foundation Trust, United Kingdom
| | - Marion I Menzel
- GE Healthcare, Munich, Germany
- Dept. of Physics, Technical University of Munich, Munich, Germany
| | | | | | - Vikas Chauhan
- Sree Chitra Tirunal Institute of Medical Sciences and Technology (SCTIMST), Trivandrum, India
| | | | - Joakim Jonsson
- Department of Radiation Sciences, UmeåUniversity, Umea, Sweden
| | - Tufve Nyholm
- Department of Radiation Sciences, UmeåUniversity, Umea, Sweden
| | | | - Bjoern Menze
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| |
Collapse
|
8
|
Mori S, Hirai R, Sakata Y, Tachibana Y, Koto M, Ishikawa H. Deep neural network-based synthetic image digital fluoroscopy using digitally reconstructed tomography. Phys Eng Sci Med 2023; 46:1227-1237. [PMID: 37349631 DOI: 10.1007/s13246-023-01290-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2023] [Accepted: 06/16/2023] [Indexed: 06/24/2023]
Abstract
We developed a deep neural network (DNN) to generate X-ray flat panel detector (FPD) images from digitally reconstructed radiographic (DRR) images. FPD and treatment planning CT images were acquired from patients with prostate and head and neck (H&N) malignancies. The DNN parameters were optimized for FPD image synthesis. The synthetic FPD images' features were evaluated to compare to the corresponding ground-truth FPD images using mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM). The image quality of the synthetic FPD image was also compared with that of the DRR image to understand the performance of our DNN. For the prostate cases, the MAE of the synthetic FPD image was improved (= 0.12 ± 0.02) from that of the input DRR image (= 0.35 ± 0.08). The synthetic FPD image showed higher PSNRs (= 16.81 ± 1.54 dB) than those of the DRR image (= 8.74 ± 1.56 dB), while SSIMs for both images (= 0.69) were almost the same. All metrics for the synthetic FPD images of the H&N cases were improved (MAE 0.08 ± 0.03, PSNR 19.40 ± 2.83 dB, and SSIM 0.80 ± 0.04) compared to those for the DRR image (MAE 0.48 ± 0.11, PSNR 5.74 ± 1.63 dB, and SSIM 0.52 ± 0.09). Our DNN successfully generated FPD images from DRR images. This technique would be useful to increase throughput when images from two different modalities are compared by visual inspection.
Collapse
Affiliation(s)
- Shinichiro Mori
- National Institutes for Quantum Science and Technology, Quantum Life and Medical Science Directorate, Institute for Quantum Medical Science, Inage-ku, Chiba, 263-8555, Japan.
| | - Ryusuke Hirai
- Corporate Research and Development Center, Toshiba Corporation, Kanagawa, 212-8582, Japan
| | - Yukinobu Sakata
- Corporate Research and Development Center, Toshiba Corporation, Kanagawa, 212-8582, Japan
| | - Yasuhiko Tachibana
- National Institutes for Quantum Science and Technology, Quantum Life and Medical Science Directorate, Institute for Quantum Medical Science, Inage-ku, Chiba, 263-8555, Japan
| | - Masashi Koto
- QST hospital, National Institutes for Quantum Science and Technology, Inage-ku, Chiba, 263-8555, Japan
| | - Hitoshi Ishikawa
- QST hospital, National Institutes for Quantum Science and Technology, Inage-ku, Chiba, 263-8555, Japan
| |
Collapse
|
9
|
Eidex Z, Ding Y, Wang J, Abouei E, Qiu RL, Liu T, Wang T, Yang X. Deep Learning in MRI-guided Radiation Therapy: A Systematic Review. ARXIV 2023:arXiv:2303.11378v2. [PMID: 36994167 PMCID: PMC10055493] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 03/31/2023]
Abstract
MRI-guided radiation therapy (MRgRT) offers a precise and adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed. MRI-guided radiation therapy offers a precise, adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed with emphasis placed on underlying methods. Studies are further categorized into the areas of segmentation, synthesis, radiomics, and real time MRI. Finally, clinical implications, current challenges, and future directions are discussed.
Collapse
Affiliation(s)
- Zach Eidex
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
- School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA
| | - Yifu Ding
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Jing Wang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Elham Abouei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Richard L.J. Qiu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
| | - Tian Liu
- Department of Radiation Oncology, Icahn School of Medicine at Mount Sinai, New York, NY
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA
- School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA
| |
Collapse
|
10
|
A Systematic Literature Review on Applications of GAN-Synthesized Images for Brain MRI. FUTURE INTERNET 2022. [DOI: 10.3390/fi14120351] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022] Open
Abstract
With the advances in brain imaging, magnetic resonance imaging (MRI) is evolving as a popular radiological tool in clinical diagnosis. Deep learning (DL) methods can detect abnormalities in brain images without an extensive manual feature extraction process. Generative adversarial network (GAN)-synthesized images have many applications in this field besides augmentation, such as image translation, registration, super-resolution, denoising, motion correction, segmentation, reconstruction, and contrast enhancement. The existing literature was reviewed systematically to understand the role of GAN-synthesized dummy images in brain disease diagnosis. Web of Science and Scopus databases were extensively searched to find relevant studies from the last 6 years to write this systematic literature review (SLR). Predefined inclusion and exclusion criteria helped in filtering the search results. Data extraction is based on related research questions (RQ). This SLR identifies various loss functions used in the above applications and software to process brain MRIs. A comparative study of existing evaluation metrics for GAN-synthesized images helps choose the proper metric for an application. GAN-synthesized images will have a crucial role in the clinical sector in the coming years, and this paper gives a baseline for other researchers in the field.
Collapse
|
11
|
Possibilities and challenges when using synthetic computed tomography in an adaptive carbon-ion treatment workflow. Z Med Phys 2022:S0939-3889(22)00064-2. [PMID: 35764469 DOI: 10.1016/j.zemedi.2022.05.003] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Revised: 05/29/2022] [Accepted: 05/29/2022] [Indexed: 11/23/2022]
Abstract
BACKGROUND AND PURPOSE Anatomical surveillance during ion-beam therapy is the basis for an effective tumor treatment and optimal organ at risk (OAR) sparing. Synthetic computed tomography (sCT) based on magnetic resonance imaging (MRI) can replace the X-ray based planning CT (X-rayCT) in photon radiotherapy and improve the workflow efficiency without additional imaging dose. The extension to carbon-ion radiotherapy is highly challenging; complex patient positioning, unique anatomical situations, distinct horizontal and vertical beam incidence directions, and limited training data are only few problems. This study gives insight into the possibilities and challenges of using sCTs in carbon-ion therapy. MATERIALS AND METHODS For head and neck patients immobilised with thermoplastic masks 30 clinically applied actively scanned carbon-ion treatment plans on 15 CTs comprising 60 beams were analyzed. Those treatment plans were re-calculated on MRI based sCTs which were created employing a 3D U-Net. Dose differences and carbon-ion spot displacements between sCT and X-rayCT were evaluated on a patient specific basis. RESULTS Spot displacement analysis showed a peak displacement by 0.2 cm caused by the immobilisation mask not measurable with the MRI. 95.7% of all spot displacements were located within 1 cm. For the clinical target volume (CTV) the median D50% agreed within -0.2% (-1.3 to 1.4%), while the median D0.01cc differed up to 4.2% (-1.3 to 25.3%) comparing the dose distribution on the X-rayCT and the sCT. OAR deviations depended strongly on the position and the dose gradient. For three patients no deterioration of the OAR parameters was observed. Other patients showed large deteriorations, e.g. for one patient D2% of the chiasm differed by 28.1%. CONCLUSION The usage of sCTs opens several new questions, concluding that we are not ready yet for an MR-only workflow in carbon-ion therapy, as envisaged in photon therapy. Although omitting the X-rayCT seems unfavourable in the case of carbon-ion therapy, an sCT could be advantageous for monitoring, re-planning, and adaptation.
Collapse
|
12
|
Ma X, Chen X, Wang Y, Qin S, Yan X, Cao Y, Chen Y, Dai J, Men K. Personalized modeling to improve pseudo-CT images for magnetic resonance imaging-guided adaptive radiotherapy. Int J Radiat Oncol Biol Phys 2022; 113:885-892. [PMID: 35462026 DOI: 10.1016/j.ijrobp.2022.03.032] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2021] [Revised: 02/24/2022] [Accepted: 03/25/2022] [Indexed: 10/18/2022]
Abstract
PURPOSE Magnetic resonance imaging-guided adaptive radiotherapy (MRIgART) greatly improves daily tumor localization and enables online re-planning to obtain maximum dosimetric benefits. However, accurately predicting patient-specific electron density maps for adaptive radiotherapy (ART) planning remains a challenge. Therefore, this study proposes a personalized modeling framework for generating pseudo-computed tomography (pCT) in MRIgART. METHODS AND MATERIALS Eighty-three patients who received MRIgART were included and CT simulations were performed on all the patients. Daily T2-weighted 1.5 T MRI was acquired using the Unity MR-linac for adaptive planning. Pairs of co-registered CT and daily MRI images of the randomly selected training set (68 patients) were inputted into a generative adversarial network (GAN) to establish a population model. The personalized model for each patient in the test set (15 patients) was acquired using model fine-tuning, which adopted the pair of the deformable-registered CT and the first daily MRI to fine-tune the population model. The pCT quality was quantitatively evaluated in the second and the last fractions with three metrics: intensity accuracy using mean absolute error (MAE); anatomical structure similarity using dice similarity coefficient (DSC); and dosimetric consistency using gamma-passing rate (GPR). RESULTS The image generation speed was 65 slices per second. For the last fractions, and for head-neck, thoracoabdominal, and pelvic cases, the average MAEs were 76.8 HU vs. 123.6 HU, 38.1 HU vs. 52.0 HU, and 29.5 HU vs. 39.7 HU, respectively. Furthermore, the average DSCs of bone were 0.92 vs. 0.80, 0.85 vs. 0.73, and 0.94 vs. 0.88; and the average GPRs (1%/1 mm) were 95.5% vs. 84.7%, 97.7% vs. 92.8%, and 95.5% vs. 88.7%, for personalized vs. population models, respectively. Results of the second fractions were similar. CONCLUSIONS The proposed personalized modeling framework remarkably improved pCT quality for multiple treatment sites and was well suited for the MRIgART clinical setting.
Collapse
Affiliation(s)
- Xiangyu Ma
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China..
| | - Xinyuan Chen
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yu Wang
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Shirui Qin
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Xuena Yan
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Ying Cao
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yan Chen
- Elekta Technology Co., Shanghai, China
| | - Jianrong Dai
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Kuo Men
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China..
| |
Collapse
|
13
|
Zimmermann L, Knäusl B, Stock M, Lütgendorf-Caucig C, Georg D, Kuess P. An MRI sequence independent convolutional neural network for synthetic head CT generation in proton therapy. Z Med Phys 2021; 32:218-227. [PMID: 34920940 PMCID: PMC9948837 DOI: 10.1016/j.zemedi.2021.10.003] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 10/11/2021] [Accepted: 10/19/2021] [Indexed: 12/11/2022]
Abstract
A magnetic resonance imaging (MRI) sequence independent deep learning technique was developed and validated to generate synthetic computed tomography (sCT) scans for MR guided proton therapy. 47 meningioma patients previously undergoing proton therapy based on pencil beam scanning were divided into training (33), validation (6), and test (8) cohorts. T1, T2, and contrast enhanced T1 (T1CM) MRI sequences were used in combination with the planning CT (pCT) data to train a 3D U-Net architecture with ResNet-Blocks. A hyperparameter search was performed including two loss functions, two group sizes of normalisation, and depth of the network. Training outcome was compared between models trained for each individual MRI sequence and for all sequences combined. The performance was evaluated based on a metric and dosimetric analysis as well as spot difference maps. Furthermore, the influence of immobilisation masks that are not visible on MRIs was investigated. Based on the hyperparameter search, the final model was trained with fixed features per group for the group normalisation, six down-convolution steps, an input size of 128×192×192, and feature loss. For the test dataset for body/bone the mean absolute error (MAE) values were on average 79.8/216.3Houndsfield unit (HU) when trained using T1 images, 71.1/186.1HU for T2, and 82.9/236.4HU for T1CM. The structural similarity metric (SSIM) ranged from 0.95 to 0.98 for all sequences. The investigated dose parameters of the target structures agreed within 1% between original proton treatment plans and plans recalculated on sCTs. The spot difference maps had peaks at ±0.2cm and for 98% of all spots the difference was less than 1cm. A novel MRI sequence independent sCT generator was developed, which suggests that the training phase of neural networks can be disengaged from specific MRI acquisition protocols. In contrast to previous studies, the patient cohort consisted exclusively of actual proton therapy patients (i.e. "real-world data").
Collapse
Affiliation(s)
- Lukas Zimmermann
- Medical University of Vienna, Department of Radiation Oncology, Vienna, Austria,Faculty of Engineering, University of Applied Sciences Wiener Neustadt, Austria,Competence Center for Preclinical Imaging and Biomedical Engineering, University of Applied Sciences Wiener Neustadt, Austria
| | - Barbara Knäusl
- Medical University of Vienna, Department of Radiation Oncology, Vienna, Austria,MedAustron Ion Therapy Center, Wiener Neustadt, Austria
| | - Markus Stock
- MedAustron Ion Therapy Center, Wiener Neustadt, Austria
| | | | - Dietmar Georg
- Medical University of Vienna, Department of Radiation Oncology, Vienna, Austria
| | - Peter Kuess
- Medical University of Vienna, Department of Radiation Oncology, Vienna, Austria; MedAustron Ion Therapy Center, Wiener Neustadt, Austria.
| |
Collapse
|
14
|
Boulanger M, Nunes JC, Chourak H, Largent A, Tahri S, Acosta O, De Crevoisier R, Lafond C, Barateau A. Deep learning methods to generate synthetic CT from MRI in radiotherapy: A literature review. Phys Med 2021; 89:265-281. [PMID: 34474325 DOI: 10.1016/j.ejmp.2021.07.027] [Citation(s) in RCA: 74] [Impact Index Per Article: 24.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/11/2021] [Revised: 07/15/2021] [Accepted: 07/19/2021] [Indexed: 01/04/2023] Open
Abstract
PURPOSE In radiotherapy, MRI is used for target volume and organs-at-risk delineation for its superior soft-tissue contrast as compared to CT imaging. However, MRI does not provide the electron density of tissue necessary for dose calculation. Several methods of synthetic-CT (sCT) generation from MRI data have been developed for radiotherapy dose calculation. This work reviewed deep learning (DL) sCT generation methods and their associated image and dose evaluation, in the context of MRI-based dose calculation. METHODS We searched the PubMed and ScienceDirect electronic databases from January 2010 to March 2021. For each paper, several items were screened and compiled in figures and tables. RESULTS This review included 57 studies. The DL methods were either generator-only based (45% of the reviewed studies), or generative adversarial network (GAN) architecture and its variants (55% of the reviewed studies). The brain and pelvis were the most commonly investigated anatomical localizations (39% and 28% of the reviewed studies, respectively), and more rarely, the head-and-neck (H&N) (15%), abdomen (10%), liver (5%) or breast (3%). All the studies performed an image evaluation of sCTs with a diversity of metrics, with only 36 studies performing dosimetric evaluations of sCT. CONCLUSIONS The median mean absolute errors were around 76 HU for the brain and H&N sCTs and 40 HU for the pelvis sCTs. For the brain, the mean dose difference between the sCT and the reference CT was <2%. For the H&N and pelvis, the mean dose difference was below 1% in most of the studies. Recent GAN architectures have advantages compared to generator-only, but no superiority was found in term of image or dose sCT uncertainties. Key challenges of DL-based sCT generation methods from MRI in radiotherapy is the management of movement for abdominal and thoracic localizations, the standardization of sCT evaluation, and the investigation of multicenter impacts.
Collapse
Affiliation(s)
- M Boulanger
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France
| | - Jean-Claude Nunes
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France.
| | - H Chourak
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France; CSIRO Australian e-Health Research Centre, Herston, Queensland, Australia
| | - A Largent
- Developing Brain Institute, Department of Diagnostic Imaging and Radiology, Children's National Hospital, Washington, DC, USA
| | - S Tahri
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France
| | - O Acosta
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France
| | - R De Crevoisier
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France
| | - C Lafond
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France
| | - A Barateau
- Univ. Rennes 1, CLCC Eugène Marquis, INSERM, LTSI - UMR 1099, F-35000 Rennes, France
| |
Collapse
|
15
|
Spadea MF, Maspero M, Zaffino P, Seco J. Deep learning based synthetic-CT generation in radiotherapy and PET: A review. Med Phys 2021; 48:6537-6566. [PMID: 34407209 DOI: 10.1002/mp.15150] [Citation(s) in RCA: 82] [Impact Index Per Article: 27.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2021] [Revised: 06/06/2021] [Accepted: 07/13/2021] [Indexed: 01/22/2023] Open
Abstract
Recently,deep learning (DL)-based methods for the generation of synthetic computed tomography (sCT) have received significant research attention as an alternative to classical ones. We present here a systematic review of these methods by grouping them into three categories, according to their clinical applications: (i) to replace computed tomography in magnetic resonance (MR) based treatment planning, (ii) facilitate cone-beam computed tomography based image-guided adaptive radiotherapy, and (iii) derive attenuation maps for the correction of positron emission tomography. Appropriate database searching was performed on journal articles published between January 2014 and December 2020. The DL methods' key characteristics were extracted from each eligible study, and a comprehensive comparison among network architectures and metrics was reported. A detailed review of each category was given, highlighting essential contributions, identifying specific challenges, and summarizing the achievements. Lastly, the statistics of all the cited works from various aspects were analyzed, revealing the popularity and future trends and the potential of DL-based sCT generation. The current status of DL-based sCT generation was evaluated, assessing the clinical readiness of the presented methods.
Collapse
Affiliation(s)
- Maria Francesca Spadea
- Department Experimental and Clinical Medicine, University "Magna Graecia" of Catanzaro, Catanzaro, 88100, Italy
| | - Matteo Maspero
- Division of Imaging & Oncology, Department of Radiotherapy, University Medical Center Utrecht, Heidelberglaan, Utrecht, The Netherlands.,Computational Imaging Group for MR Diagnostics & Therapy, Center for Image Sciences, University Medical Center Utrecht, Heidelberglaan, Utrecht, The Netherlands
| | - Paolo Zaffino
- Department Experimental and Clinical Medicine, University "Magna Graecia" of Catanzaro, Catanzaro, 88100, Italy
| | - Joao Seco
- Division of Biomedical Physics in Radiation Oncology, DKFZ German Cancer Research Center, Heidelberg, Germany.,Department of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
| |
Collapse
|
16
|
Koh H, Park TY, Chung YA, Lee JH, Kim H. Acoustic simulation for transcranial focused ultrasound using GAN-based synthetic CT. IEEE J Biomed Health Inform 2021; 26:161-171. [PMID: 34388098 DOI: 10.1109/jbhi.2021.3103387] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Transcranial focused ultrasound (tFUS) is a promising non-invasive technique for treating neurological and psychiatric disorders. One of the challenges for tFUS is the disruption of wave propagation through the skull. Consequently, despite the risks associated with exposure to ionizing radiation, computed tomography (CT) is required to estimate the acoustic transmission through the skull. This study aims to generate synthetic CT (sCT) from T1-weighted magnetic resonance imaging (MRI) and investigate its applicability to tFUS acoustic simulation. We trained a 3D conditional generative adversarial network (3D-cGAN) with 15 subjects. We then assessed image quality with 15 test subjects: mean absolute error (MAE) = 85.72± 9.50 HU (head) and 280.25±24.02 HU (skull), dice coefficient similarity (DSC) = 0.88±0.02 (skull). In terms of skull density ratio (SDR) and skull thickness (ST), no significant difference was found between sCT and real CT (rCT). When the acoustic simulation results of rCT and sCT were compared, the intracranial peak acoustic pressure ratio was found to be less than 4%, and the distance between focal points less than 1 mm.
Collapse
|
17
|
Irmak S, Zimmermann L, Georg D, Kuess P, Lechner W. Cone beam CT based validation of neural network generated synthetic CTs for radiotherapy in the head region. Med Phys 2021; 48:4560-4571. [PMID: 34028053 DOI: 10.1002/mp.14987] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2020] [Revised: 05/06/2021] [Accepted: 05/09/2021] [Indexed: 11/10/2022] Open
Abstract
PURPOSE In the past years, many different neural network-based conversion techniques for synthesizing computed tomographys (sCTs) from MR images have been published. While the model's performance can be checked during the training against the test set, test datasets can never represent the whole population. Conversion errors can still occur for special cases, for example, for unusual anatomical situations. Therefore, the performance of sCT conversion needs to be verified on a patient specific level, especially in the absence of a planning CT (pCT). In this study, the capability of cone-beam CTs (CBCTs) for the validation of sCTs generated by a neural network was investigated. METHODS 41 patients with tumors in the head region were selected. 20 of them were used for model training and 10 for validation. Different implementations of CycleGAN (with/without identity and feature loss) were used to generate sCTs. The pixel (MAE, RMSE, PSNR) and geometric error (DICE, Sensitivity, Specificity) values were reported to identify the best model. VMAT plans were created for the remaining 11 patients on the pCTs. These plans were re-calculated on sCTs and CBCTs. An automatic density overriding method ( C B C T RS ) and a population-based dose calculation method ( C B C T Pop ) were employed for CBCT-based dose calculation. The dose distributions were analysed using 3D global gamma analysis, applying a threshold of 10% with respect to the prescribed dose. Differences in DVH metrics for the PTV and the organs-at-risk were compared among the dose distributions based on pCTs, sCTs, and CBCTs. RESULTS The best model was the CycleGAN without identity and feature matching loss. Including the identity loss led to a metric decrease of 10% for DICE and a metric increase of 20-60 HU for MAE. Using the 2%/2 mm gamma criterion and pCT as reference, the mean gamma pass rates were 99.0 ± 0.4% for sCTs. Mean gamma pass rate values comparing pCT and CBCT were 99.0 ± 0.8% and 99.1 ± 0.8% for the C B C T RS and C B C T Pop , respectively. The mean gamma pass rates comparing sCT and CBCT resulted in 98.4 ± 1.6% and 99.2 ± 0.6% for C B C T RS and C B C T Pop , respectively. The differences between the gamma-pass-rates of the sCT and two CBCT-based methods were not significant. The majority of deviations of the investigated DVH metrices between sCTs and CBCTs were within 2%. CONCLUSION The dosimetric results demonstrate good agreement between sCT, CBCT, and pCT based calculations. A properly applied CBCT conversion method can serve as a tool for quality assurance procedures in an MR only radiotherapy workflow for head patients. Dosimetric deviations of DVH metrics between sCT and CBCTs of larger than 2% should be followed up. A systematic shift of approximately 1% should be taken into account when using the C B C T RS approach in an MR only workflow.
Collapse
Affiliation(s)
- Sinan Irmak
- Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Lukas Zimmermann
- Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria.,Faculty of Engineering, University of Applied Sciences, Wiener Neustadt, Austria.,Competence Center for Preclinical Imaging and Biomedical Engineering, University of Applied Sciences, Wiener Neustadt, Austria
| | - Dietmar Georg
- Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Peter Kuess
- Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Wolfgang Lechner
- Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| |
Collapse
|
18
|
Fu Y, Lei Y, Wang T, Curran WJ, Liu T, Yang X. A review of deep learning based methods for medical image multi-organ segmentation. Phys Med 2021; 85:107-122. [PMID: 33992856 PMCID: PMC8217246 DOI: 10.1016/j.ejmp.2021.05.003] [Citation(s) in RCA: 73] [Impact Index Per Article: 24.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/27/2020] [Revised: 03/12/2021] [Accepted: 05/03/2021] [Indexed: 12/12/2022] Open
Abstract
Deep learning has revolutionized image processing and achieved the-state-of-art performance in many medical image segmentation tasks. Many deep learning-based methods have been published to segment different parts of the body for different medical applications. It is necessary to summarize the current state of development for deep learning in the field of medical image segmentation. In this paper, we aim to provide a comprehensive review with a focus on multi-organ image segmentation, which is crucial for radiotherapy where the tumor and organs-at-risk need to be contoured for treatment planning. We grouped the surveyed methods into two broad categories which are 'pixel-wise classification' and 'end-to-end segmentation'. Each category was divided into subgroups according to their network design. For each type, we listed the surveyed works, highlighted important contributions and identified specific challenges. Following the detailed review, we discussed the achievements, shortcomings and future potentials of each category. To enable direct comparison, we listed the performance of the surveyed works that used thoracic and head-and-neck benchmark datasets.
Collapse
Affiliation(s)
- Yabo Fu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Yang Lei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Tonghe Wang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Walter J Curran
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Tian Liu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA.
| |
Collapse
|
19
|
Wang T, Lei Y, Harms J, Ghavidel B, Lin L, Beitler JJ, McDonald M, Curran WJ, Liu T, Zhou J, Yang X. Learning-Based Stopping Power Mapping on Dual-Energy CT for Proton Radiation Therapy. Int J Part Ther 2021; 7:46-60. [PMID: 33604415 PMCID: PMC7886267 DOI: 10.14338/ijpt-d-20-00020.1] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2020] [Accepted: 12/04/2020] [Indexed: 12/30/2022] Open
Abstract
Purpose Dual-energy computed tomography (DECT) has been used to derive relative stopping power (RSP) maps by obtaining the energy dependence of photon interactions. The DECT-derived RSP maps could potentially be compromised by image noise levels and the severity of artifacts when using physics-based mapping techniques. This work presents a noise-robust learning-based method to predict RSP maps from DECT for proton radiation therapy. Materials and Methods The proposed method uses a residual attention cycle-consistent generative adversarial network to bring DECT-to-RSP mapping close to a 1-to-1 mapping by introducing an inverse RSP-to-DECT mapping. To evaluate the proposed method, we retrospectively investigated 20 head-and-neck cancer patients with DECT scans in proton radiation therapy simulation. Ground truth RSP values were assigned by calculation based on chemical compositions and acted as learning targets in the training process for DECT datasets; they were evaluated against results from the proposed method using a leave-one-out cross-validation strategy. Results The predicted RSP maps showed an average normalized mean square error of 2.83% across the whole body volume and an average mean error less than 3% in all volumes of interest. With additional simulated noise added in DECT datasets, the proposed method still maintained a comparable performance, while the physics-based stoichiometric method suffered degraded inaccuracy from increased noise level. The average differences from ground truth in dose volume histogram metrics for clinical target volumes were less than 0.2 Gy for D95% and Dmax with no statistical significance. Maximum difference in dose volume histogram metrics of organs at risk was around 1 Gy on average. Conclusion These results strongly indicate the high accuracy of RSP maps predicted by our machine-learning–based method and show its potential feasibility for proton treatment planning and dose calculation.
Collapse
Affiliation(s)
- Tonghe Wang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Yang Lei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Joseph Harms
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Beth Ghavidel
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Liyong Lin
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Jonathan J Beitler
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Mark McDonald
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Walter J Curran
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Tian Liu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Jun Zhou
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA
| |
Collapse
|
20
|
Zimmermann L, Buschmann M, Herrmann H, Heilemann G, Kuess P, Goldner G, Nyholm T, Georg D, Nesvacil N. An MR-only acquisition and artificial intelligence based image-processing protocol for photon and proton therapy using a low field MR. Z Med Phys 2021; 31:78-88. [PMID: 33455822 DOI: 10.1016/j.zemedi.2020.10.004] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2020] [Revised: 09/14/2020] [Accepted: 10/27/2020] [Indexed: 10/22/2022]
Abstract
OBJECTIVE Recent developments on synthetically generated CTs (sCT), hybrid MRI linacs and MR-only simulations underlined the clinical feasibility and acceptance of MR guided radiation therapy. However, considering clinical application of open and low field MR with a limited field of view can result in truncation of the patient's anatomy which further affects the MR to sCT conversion. In this study an acquisition protocol and subsequent MR image stitching is proposed to overcome the limited field of view restriction of open MR scanners, for MR-only photon and proton therapy. MATERIAL AND METHODS 12 prostate cancer patients scanned with an open 0.35T scanner were included. To obtain the full body contour an enhanced imaging protocol including two repeated scans after bilateral table movement was introduced. All required structures (patient contour, target and organ at risk) were delineated on a post-processed combined transversal image set (stitched MRI). The postprocessed MR was converted into a sCT by a pretrained neural network generator. Inversely planned photon and proton plans (VMAT and SFUD) were designed using the sCT and recalculated for rigidly and deformably registered CT images and compared based on D2%, D50%, V70Gy for organs at risk and based on D2%, D50%, D98% for the CTV and PTV. The stitched MRI and the untruncated MRI were compared to the CT, and the maximum surface distance was calculated. The sCT was evaluated with respect to delineation accuracy by comparing on stitched MRI and sCT using the DICE coefficient for femoral bones and the whole body. RESULTS Maximum surface distance analysis revealed uncertainties in lateral direction of 1-3mm on average. DICE coefficient analysis confirms good performance of the sCT conversion, i.e. 92%, 93%, and 100% were obtained for femoral bone left and right and whole body. Dose comparison resulted in uncertainties below 1% between deformed CT and sCT and below 2% between rigidly registered CT and sCT in the CTV for photon and proton treatment plans. DISCUSSION A newly developed acquisition protocol for open MR scanners and subsequent Sct generation revealed good acceptance for photon and proton therapy. Moreover, this protocol tackles the restriction of the limited FOVs and expands the capacities towards MR guided proton therapy with horizontal beam lines.
Collapse
Affiliation(s)
- Lukas Zimmermann
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria.
| | - Martin Buschmann
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Harald Herrmann
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Gerd Heilemann
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Peter Kuess
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Gregor Goldner
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Tufve Nyholm
- Department of Radiation Sciences, Umeå University, Umeå, Sweden
| | - Dietmar Georg
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| | - Nicole Nesvacil
- Division of Medical Radiation Physics, Department of Radiation Oncology, Medical University of Vienna, Vienna, Austria
| |
Collapse
|
21
|
Wang T, Lei Y, Fu Y, Wynne JF, Curran WJ, Liu T, Yang X. A review on medical imaging synthesis using deep learning and its clinical applications. J Appl Clin Med Phys 2021; 22:11-36. [PMID: 33305538 PMCID: PMC7856512 DOI: 10.1002/acm2.13121] [Citation(s) in RCA: 102] [Impact Index Per Article: 34.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Revised: 11/12/2020] [Accepted: 11/21/2020] [Indexed: 02/06/2023] Open
Abstract
This paper reviewed the deep learning-based studies for medical imaging synthesis and its clinical application. Specifically, we summarized the recent developments of deep learning-based methods in inter- and intra-modality image synthesis by listing and highlighting the proposed methods, study designs, and reported performances with related clinical applications on representative studies. The challenges among the reviewed studies were then summarized with discussion.
Collapse
Affiliation(s)
- Tonghe Wang
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
- Winship Cancer InstituteEmory UniversityAtlantaGAUSA
| | - Yang Lei
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
| | - Yabo Fu
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
| | - Jacob F. Wynne
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
| | - Walter J. Curran
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
- Winship Cancer InstituteEmory UniversityAtlantaGAUSA
| | - Tian Liu
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
- Winship Cancer InstituteEmory UniversityAtlantaGAUSA
| | - Xiaofeng Yang
- Department of Radiation OncologyEmory UniversityAtlantaGAUSA
- Winship Cancer InstituteEmory UniversityAtlantaGAUSA
| |
Collapse
|
22
|
Wang T, Lei Y, Fu Y, Curran WJ, Liu T, Nye JA, Yang X. Machine learning in quantitative PET: A review of attenuation correction and low-count image reconstruction methods. Phys Med 2020; 76:294-306. [PMID: 32738777 PMCID: PMC7484241 DOI: 10.1016/j.ejmp.2020.07.028] [Citation(s) in RCA: 55] [Impact Index Per Article: 13.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/23/2020] [Revised: 07/13/2020] [Accepted: 07/21/2020] [Indexed: 02/08/2023] Open
Abstract
The rapid expansion of machine learning is offering a new wave of opportunities for nuclear medicine. This paper reviews applications of machine learning for the study of attenuation correction (AC) and low-count image reconstruction in quantitative positron emission tomography (PET). Specifically, we present the developments of machine learning methodology, ranging from random forest and dictionary learning to the latest convolutional neural network-based architectures. For application in PET attenuation correction, two general strategies are reviewed: 1) generating synthetic CT from MR or non-AC PET for the purposes of PET AC, and 2) direct conversion from non-AC PET to AC PET. For low-count PET reconstruction, recent deep learning-based studies and the potential advantages over conventional machine learning-based methods are presented and discussed. In each application, the proposed methods, study designs and performance of published studies are listed and compared with a brief discussion. Finally, the overall contributions and remaining challenges are summarized.
Collapse
Affiliation(s)
- Tonghe Wang
- Department of Radiation Oncology, Emory University, Atlanta, GA, USA; Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Yang Lei
- Department of Radiation Oncology, Emory University, Atlanta, GA, USA
| | - Yabo Fu
- Department of Radiation Oncology, Emory University, Atlanta, GA, USA
| | - Walter J Curran
- Department of Radiation Oncology, Emory University, Atlanta, GA, USA; Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Tian Liu
- Department of Radiation Oncology, Emory University, Atlanta, GA, USA; Winship Cancer Institute, Emory University, Atlanta, GA, USA
| | - Jonathon A Nye
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology, Emory University, Atlanta, GA, USA; Winship Cancer Institute, Emory University, Atlanta, GA, USA.
| |
Collapse
|
23
|
Kazemifar S, Barragán Montero AM, Souris K, Rivas ST, Timmerman R, Park YK, Jiang S, Geets X, Sterpin E, Owrangi A. Dosimetric evaluation of synthetic CT generated with GANs for MRI-only proton therapy treatment planning of brain tumors. J Appl Clin Med Phys 2020; 21:76-86. [PMID: 32216098 PMCID: PMC7286008 DOI: 10.1002/acm2.12856] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2019] [Revised: 02/10/2020] [Accepted: 02/14/2020] [Indexed: 12/23/2022] Open
Abstract
PURPOSE The purpose of this study was to address the dosimetric accuracy of synthetic computed tomography (sCT) images of patients with brain tumor generated using a modified generative adversarial network (GAN) method, for their use in magnetic resonance imaging (MRI)-only treatment planning for proton therapy. METHODS Dose volume histogram (DVH) analysis was performed on CT and sCT images of patients with brain tumor for plans generated for intensity-modulated proton therapy (IMPT). All plans were robustly optimized using a commercially available treatment planning system (RayStation, from RaySearch Laboratories) and standard robust parameters reported in the literature. The IMPT plan was then used to compute the dose on CT and sCT images for dosimetric comparison, using RayStation analytical (pencil beam) dose algorithm. We used a second, independent Monte Carlo dose calculation engine to recompute the dose on both CT and sCT images to ensure a proper analysis of the dosimetric accuracy of the sCT images. RESULTS The results extracted from RayStation showed excellent agreement for most DVH metrics computed on the CT and sCT for the nominal case, with a mean absolute difference below 0.5% (0.3 Gy) of the prescription dose for the clinical target volume (CTV) and below 2% (1.2 Gy) for the organs at risk (OARs) considered. This demonstrates a high dosimetric accuracy for the generated sCT images, especially in the target volume. The metrics obtained from the Monte Carlo doses mostly agreed with the values extracted from RayStation for the nominal and worst-case scenarios (mean difference below 3%). CONCLUSIONS This work demonstrated the feasibility of using sCT generated with a GAN-based deep learning method for MRI-only treatment planning of patients with brain tumor in intensity-modulated proton therapy.
Collapse
Affiliation(s)
- Samaneh Kazemifar
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Ana M Barragán Montero
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA.,Institut de Recherche Expérimentale et Clinique, Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Université catholique de Louvain, Brussels, Belgium
| | - Kevin Souris
- Institut de Recherche Expérimentale et Clinique, Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Université catholique de Louvain, Brussels, Belgium
| | - Sara T Rivas
- Institut de Recherche Expérimentale et Clinique, Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Université catholique de Louvain, Brussels, Belgium
| | - Robert Timmerman
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Yang K Park
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Steve Jiang
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | - Xavier Geets
- Institut de Recherche Expérimentale et Clinique, Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Université catholique de Louvain, Brussels, Belgium.,Department of Radiation Oncology, Cliniques universitaires Saint-Luc, Brussels, Belgium
| | - Edmond Sterpin
- Institut de Recherche Expérimentale et Clinique, Center of Molecular Imaging, Radiotherapy and Oncology (MIRO), Université catholique de Louvain, Brussels, Belgium.,Department of Oncology, Laboratory of Experimental Radiotherapy, KULeuven, Leuven, Belgium
| | - Amir Owrangi
- Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| |
Collapse
|