1
|
Sherwani MK, Gopalakrishnan S. A systematic literature review: deep learning techniques for synthetic medical image generation and their applications in radiotherapy. FRONTIERS IN RADIOLOGY 2024; 4:1385742. [PMID: 38601888 PMCID: PMC11004271 DOI: 10.3389/fradi.2024.1385742] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Accepted: 03/11/2024] [Indexed: 04/12/2024]
Abstract
The aim of this systematic review is to determine whether Deep Learning (DL) algorithms can provide a clinically feasible alternative to classic algorithms for synthetic Computer Tomography (sCT). The following categories are presented in this study: ∙ MR-based treatment planning and synthetic CT generation techniques. ∙ Generation of synthetic CT images based on Cone Beam CT images. ∙ Low-dose CT to High-dose CT generation. ∙ Attenuation correction for PET images. To perform appropriate database searches, we reviewed journal articles published between January 2018 and June 2023. Current methodology, study strategies, and results with relevant clinical applications were analyzed as we outlined the state-of-the-art of deep learning based approaches to inter-modality and intra-modality image synthesis. This was accomplished by contrasting the provided methodologies with traditional research approaches. The key contributions of each category were highlighted, specific challenges were identified, and accomplishments were summarized. As a final step, the statistics of all the cited works from various aspects were analyzed, which revealed that DL-based sCTs have achieved considerable popularity, while also showing the potential of this technology. In order to assess the clinical readiness of the presented methods, we examined the current status of DL-based sCT generation.
Collapse
Affiliation(s)
- Moiz Khan Sherwani
- Section for Evolutionary Hologenomics, Globe Institute, University of Copenhagen, Copenhagen, Denmark
| | | |
Collapse
|
2
|
Li X, Johnson JM, Strigel RM, Bancroft LCH, Hurley SA, Estakhraji SIZ, Kumar M, Fowler AM, McMillan AB. Attenuation correction and truncation completion for breast PET/MR imaging using deep learning. Phys Med Biol 2024; 69:045031. [PMID: 38252969 DOI: 10.1088/1361-6560/ad2126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Accepted: 01/22/2024] [Indexed: 01/24/2024]
Abstract
Objective. Simultaneous PET/MR scanners combine the high sensitivity of MR imaging with the functional imaging of PET. However, attenuation correction of breast PET/MR imaging is technically challenging. The purpose of this study is to establish a robust attenuation correction algorithm for breast PET/MR images that relies on deep learning (DL) to recreate the missing portions of the patient's anatomy (truncation completion), as well as to provide bone information for attenuation correction from only the PET data.Approach. Data acquired from 23 female subjects with invasive breast cancer scanned with18F-fluorodeoxyglucose PET/CT and PET/MR localized to the breast region were used for this study. Three DL models, U-Net with mean absolute error loss (DLMAE) model, U-Net with mean squared error loss (DLMSE) model, and U-Net with perceptual loss (DLPerceptual) model, were trained to predict synthetic CT images (sCT) for PET attenuation correction (AC) given non-attenuation corrected (NAC) PETPET/MRimages as inputs. The DL and Dixon-based sCT reconstructed PET images were compared against those reconstructed from CT images by calculating the percent error of the standardized uptake value (SUV) and conducting Wilcoxon signed rank statistical tests.Main results. sCT images from the DLMAEmodel, the DLMSEmodel, and the DLPerceptualmodel were similar in mean absolute error (MAE), peak-signal-to-noise ratio, and normalized cross-correlation. No significant difference in SUV was found between the PET images reconstructed using the DLMSEand DLPerceptualsCTs compared to the reference CT for AC in all tissue regions. All DL methods performed better than the Dixon-based method according to SUV analysis.Significance. A 3D U-Net with MSE or perceptual loss model can be implemented into a reconstruction workflow, and the derived sCT images allow successful truncation completion and attenuation correction for breast PET/MR images.
Collapse
Affiliation(s)
- Xue Li
- Department of Electrical and Computer Engineering, University of Wisconsin, Madison, WI, United States of America
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
| | - Jacob M Johnson
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
| | - Roberta M Strigel
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
- Department of Medical Physics, University of Wisconsin, Madison, WI, United States of America
- University of Wisconsin Carbone Cancer Center, Madison, WI, United States of America
| | - Leah C Henze Bancroft
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
| | - Samuel A Hurley
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
| | - S Iman Zare Estakhraji
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
| | - Manoj Kumar
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
- ICTR Graduate Program in Clinical Investigation, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
| | - Amy M Fowler
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
- Department of Medical Physics, University of Wisconsin, Madison, WI, United States of America
- University of Wisconsin Carbone Cancer Center, Madison, WI, United States of America
| | - Alan B McMillan
- Department of Electrical and Computer Engineering, University of Wisconsin, Madison, WI, United States of America
- Department of Radiology, University of Wisconsin School of Medicine and Public Health, Madison, WI, United States of America
- Department of Medical Physics, University of Wisconsin, Madison, WI, United States of America
- University of Wisconsin Carbone Cancer Center, Madison, WI, United States of America
| |
Collapse
|
3
|
Krokos G, Kotwal T, Malaih A, Barrington S, Jackson P, Hicks RJ, Marsden PK, Fischer BM. Evaluation of manual and automated approaches for segmentation and extraction of quantitative indices from [ 18F]FDG PET-CT images. Biomed Phys Eng Express 2024; 10:025007. [PMID: 38100790 PMCID: PMC10767880 DOI: 10.1088/2057-1976/ad160e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Revised: 11/28/2023] [Accepted: 12/15/2023] [Indexed: 12/17/2023]
Abstract
Utilisation of whole organ volumes to extract anatomical and functional information from computed tomography (CT) and positron emission tomography (PET) images may provide key information for the treatment and follow-up of cancer patients. However, manual organ segmentation, is laborious and time-consuming. In this study, a CT-based deep learning method and a multi-atlas method were evaluated for segmenting the liver and spleen on CT images to extract quantitative tracer information from Fluorine-18 fluorodeoxyglucose ([18F]FDG) PET images of 50 patients with advanced Hodgkin lymphoma (HL). Manual segmentation was used as the reference method. The two automatic methods were also compared with a manually defined volume of interest (VOI) within the organ, a technique commonly performed in clinical settings. Both automatic methods provided accurate CT segmentations, with the deep learning method outperforming the multi-atlas with a DICE coefficient of 0.93 ± 0.03 (mean ± standard deviation) in liver and 0.87 ± 0.17 in spleen compared to 0.87 ± 0.05 (liver) and 0.78 ± 0.11 (spleen) for the multi-atlas. Similarly, a mean relative error of -3.2% for the liver and -3.4% for the spleen across patients was found for the mean standardized uptake value (SUVmean) using the deep learning regions while the corresponding errors for the multi-atlas method were -4.7% and -9.2%, respectively. For the maximum SUV (SUVmax), both methods resulted in higher than 20% overestimation due to the extension of organ boundaries to include neighbouring, high-uptake regions. The conservative VOI method which did not extend into neighbouring tissues, provided a more accurate SUVmaxestimate. In conclusion, the automatic, and particularly the deep learning method could be used to rapidly extract information of the SUVmeanwithin the liver and spleen. However, activity from neighbouring organs and lesions can lead to high biases in SUVmaxand current practices of manually defining a volume of interest in the organ should be considered instead.
Collapse
Affiliation(s)
- Georgios Krokos
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Tejas Kotwal
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Afnan Malaih
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Sally Barrington
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | | | - Rodney J Hicks
- Department of Medicine, St Vincent’s Hospital Medical School, the University of Melbourne, Australia
| | - Paul K Marsden
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
| | - Barbara Malene Fischer
- School of Biomedical Engineering and Imaging Sciences, King’s College London, London, United Kingdom
- Dept. Clinical Physiology and Nuclear Medicine, Rigshospitalet, Copenhagen, Denmark
- Dept. of Clinical Medicine, University of Copenhagen, Denmark
| |
Collapse
|
4
|
Krokos G, MacKewn J, Dunn J, Marsden P. A review of PET attenuation correction methods for PET-MR. EJNMMI Phys 2023; 10:52. [PMID: 37695384 PMCID: PMC10495310 DOI: 10.1186/s40658-023-00569-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 08/07/2023] [Indexed: 09/12/2023] Open
Abstract
Despite being thirteen years since the installation of the first PET-MR system, the scanners constitute a very small proportion of the total hybrid PET systems installed. This is in stark contrast to the rapid expansion of the PET-CT scanner, which quickly established its importance in patient diagnosis within a similar timeframe. One of the main hurdles is the development of an accurate, reproducible and easy-to-use method for attenuation correction. Quantitative discrepancies in PET images between the manufacturer-provided MR methods and the more established CT- or transmission-based attenuation correction methods have led the scientific community in a continuous effort to develop a robust and accurate alternative. These can be divided into four broad categories: (i) MR-based, (ii) emission-based, (iii) atlas-based and the (iv) machine learning-based attenuation correction, which is rapidly gaining momentum. The first is based on segmenting the MR images in various tissues and allocating a predefined attenuation coefficient for each tissue. Emission-based attenuation correction methods aim in utilising the PET emission data by simultaneously reconstructing the radioactivity distribution and the attenuation image. Atlas-based attenuation correction methods aim to predict a CT or transmission image given an MR image of a new patient, by using databases containing CT or transmission images from the general population. Finally, in machine learning methods, a model that could predict the required image given the acquired MR or non-attenuation-corrected PET image is developed by exploiting the underlying features of the images. Deep learning methods are the dominant approach in this category. Compared to the more traditional machine learning, which uses structured data for building a model, deep learning makes direct use of the acquired images to identify underlying features. This up-to-date review goes through the literature of attenuation correction approaches in PET-MR after categorising them. The various approaches in each category are described and discussed. After exploring each category separately, a general overview is given of the current status and potential future approaches along with a comparison of the four outlined categories.
Collapse
Affiliation(s)
- Georgios Krokos
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK.
| | - Jane MacKewn
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK
| | - Joel Dunn
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK
| | - Paul Marsden
- School of Biomedical Engineering and Imaging Sciences, The PET Centre at St Thomas' Hospital London, King's College London, 1st Floor Lambeth Wing, Westminster Bridge Road, London, SE1 7EH, UK
| |
Collapse
|
5
|
Zhong L, Chen Z, Shu H, Zheng Y, Zhang Y, Wu Y, Feng Q, Li Y, Yang W. QACL: Quartet attention aware closed-loop learning for abdominal MR-to-CT synthesis via simultaneous registration. Med Image Anal 2023; 83:102692. [PMID: 36442293 DOI: 10.1016/j.media.2022.102692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Revised: 10/27/2022] [Accepted: 11/09/2022] [Indexed: 11/18/2022]
Abstract
Synthesis of computed tomography (CT) images from magnetic resonance (MR) images is an important task to overcome the lack of electron density information in MR-only radiotherapy treatment planning (RTP). Some innovative methods have been proposed for abdominal MR-to-CT synthesis. However, it is still challenging due to the large misalignment between preprocessed abdominal MR and CT images and the insufficient feature information learned by models. Although several studies have used the MR-to-CT synthesis to alleviate the difficulty of multi-modal registration, this misalignment remains unsolved when training the MR-to-CT synthesis model. In this paper, we propose an end-to-end quartet attention aware closed-loop learning (QACL) framework for MR-to-CT synthesis via simultaneous registration. Specifically, the proposed quartet attention generator and mono-modal registration network form a closed-loop to improve the performance of MR-to-CT synthesis via simultaneous registration. In particular, a quartet-attention mechanism is developed to enlarge the receptive fields in networks to extract the long-range and cross-dimension spatial dependencies. Experimental results on two independent abdominal datasets demonstrate that our QACL achieves impressive results with MAE of 55.30±10.59 HU, PSNR of 22.85±1.43 dB, and SSIM of 0.83±0.04 for synthesis, and with Dice of 0.799±0.129 for registration. The proposed QACL outperforms the state-of-the-art MR-to-CT synthesis and multi-modal registration methods.
Collapse
Affiliation(s)
- Liming Zhong
- School of Biomedical Engineering, Southern Medical University, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, 510515, China
| | - Zeli Chen
- School of Biomedical Engineering, Southern Medical University, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, 510515, China
| | - Hai Shu
- Department of Biostatistics, School of Global Public Health, New York University, New York, NY, 10003, United States
| | - Yikai Zheng
- School of Biomedical Engineering, Southern Medical University, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, 510515, China
| | - Yiwen Zhang
- School of Biomedical Engineering, Southern Medical University, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, 510515, China
| | - Yuankui Wu
- Department of Medical Imaging Center, Nanfang Hospital, Southern Medical University, Guangzhou, 510515, China
| | - Qianjin Feng
- School of Biomedical Engineering, Southern Medical University, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, 510515, China
| | - Yin Li
- Department of Information, The Sixth Affiliated Hospital, Sun Yat-sen University, Guangzhou, 510655, China.
| | - Wei Yang
- School of Biomedical Engineering, Southern Medical University, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, 510515, China.
| |
Collapse
|
6
|
Minoshima S, Cross D. Application of artificial intelligence in brain molecular imaging. Ann Nucl Med 2022; 36:103-110. [PMID: 35028878 DOI: 10.1007/s12149-021-01697-2] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Accepted: 11/15/2021] [Indexed: 12/22/2022]
Abstract
Initial development of artificial Intelligence (AI) and machine learning (ML) dates back to the mid-twentieth century. A growing awareness of the potential for AI, as well as increases in computational resources, research, and investment are rapidly advancing AI applications to medical imaging and, specifically, brain molecular imaging. AI/ML can improve imaging operations and decision making, and potentially perform tasks that are not readily possible by physicians, such as predicting disease prognosis, and identifying latent relationships from multi-modal clinical information. The number of applications of image-based AI algorithms, such as convolutional neural network (CNN), is increasing rapidly. The applications for brain molecular imaging (MI) include image denoising, PET and PET/MRI attenuation correction, image segmentation and lesion detection, parametric image formation, and the detection/diagnosis of Alzheimer's disease and other brain disorders. When effectively used, AI will likely improve the quality of patient care, instead of replacing radiologists. A regulatory framework is being developed to facilitate AI adaptation for medical imaging.
Collapse
Affiliation(s)
- Satoshi Minoshima
- Department of Radiology and Imaging Sciences, University of Utah, 30 North 1900 East #1A071, Salt Lake City, UT, 84132, USA.
| | - Donna Cross
- Department of Radiology and Imaging Sciences, University of Utah, 30 North 1900 East #1A071, Salt Lake City, UT, 84132, USA
| |
Collapse
|
7
|
Arabi H, Zaidi H. MRI-guided attenuation correction in torso PET/MRI: Assessment of segmentation-, atlas-, and deep learning-based approaches in the presence of outliers. Magn Reson Med 2021; 87:686-701. [PMID: 34480771 PMCID: PMC9292636 DOI: 10.1002/mrm.29003] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2021] [Revised: 08/14/2021] [Accepted: 08/21/2021] [Indexed: 12/22/2022]
Abstract
Purpose We compare the performance of three commonly used MRI‐guided attenuation correction approaches in torso PET/MRI, namely segmentation‐, atlas‐, and deep learning‐based algorithms. Methods Twenty‐five co‐registered torso 18F‐FDG PET/CT and PET/MR images were enrolled. PET attenuation maps were generated from in‐phase Dixon MRI using a three‐tissue class segmentation‐based approach (soft‐tissue, lung, and background air), voxel‐wise weighting atlas‐based approach, and a residual convolutional neural network. The bias in standardized uptake value (SUV) was calculated for each approach considering CT‐based attenuation corrected PET images as reference. In addition to the overall performance assessment of these approaches, the primary focus of this work was on recognizing the origins of potential outliers, notably body truncation, metal‐artifacts, abnormal anatomy, and small malignant lesions in the lungs. Results The deep learning approach outperformed both atlas‐ and segmentation‐based methods resulting in less than 4% SUV bias across 25 patients compared to the segmentation‐based method with up to 20% SUV bias in bony structures and the atlas‐based method with 9% bias in the lung. The deep learning‐based method exhibited superior performance. Yet, in case of sever truncation and metallic‐artifacts in the input MRI, this approach was outperformed by the atlas‐based method, exhibiting suboptimal performance in the affected regions. Conversely, for abnormal anatomies, such as a patient presenting with one lung or small malignant lesion in the lung, the deep learning algorithm exhibited promising performance compared to other methods. Conclusion The deep learning‐based method provides promising outcome for synthetic CT generation from MRI. However, metal‐artifact and body truncation should be specifically addressed.
Collapse
Affiliation(s)
- Hossein Arabi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland.,Geneva University Neurocenter, Geneva University, Geneva, Switzerland.,Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen, Groningen, Netherlands.,Department of Nuclear Medicine, University of Southern Denmark, Odense, Denmark
| |
Collapse
|