Feasibility of Synthetic Computed Tomography Images Generated from Magnetic Resonance Imaging Scans Using Various Deep Learning Methods in the Planning of Radiation Therapy for Prostate Cancer

Generative AI for synthetic data across multiple medical modalities: A systematic review of recent developments and challenges

This paper presents a comprehensive systematic review of generative models (GANs, VAEs, DMs, and LLMs) used to synthesize various medical data types, including imaging (dermoscopic, mammographic, ultrasound, CT, MRI, and X-ray), text, time-series, and tabular data (EHR). Unlike previous narrowly focused reviews, our study encompasses a broad array of medical data modalities and explores various generative models. Our aim is to offer insights into their current and future applications in medical research, particularly in the context of synthesis applications, generation techniques, and evaluation methods, as well as providing a GitHub repository as a dynamic resource for ongoing collaboration and innovation. Our search strategy queries databases such as Scopus, PubMed, and ArXiv, focusing on recent works from January 2021 to November 2023, excluding reviews and perspectives. This period emphasizes recent advancements beyond GANs, which have been extensively covered in previous reviews. The survey also emphasizes the aspect of conditional generation, which is not focused on in similar work. Key contributions include a broad, multi-modality scope that identifies cross-modality insights and opportunities unavailable in single-modality surveys. While core generative techniques are transferable, we find that synthesis methods often lack sufficient integration of patient-specific context, clinical knowledge, and modality-specific requirements tailored to the unique characteristics of medical data. Conditional models leveraging textual conditioning and multimodal synthesis remain underexplored but offer promising directions for innovation. Our findings are structured around three themes: (1) Synthesis applications, highlighting clinically valid synthesis applications and significant gaps in using synthetic data beyond augmentation, such as for validation and evaluation; (2) Generation techniques, identifying gaps in personalization and cross-modality innovation; and (3) Evaluation methods, revealing the absence of standardized benchmarks, the need for large-scale validation, and the importance of privacy-aware, clinically relevant evaluation frameworks. These findings emphasize the need for benchmarking and comparative studies to promote openness and collaboration.

An overview of utilizing artificial intelligence in localized prostate cancer imaging

INTRODUCTION

Prostate cancer (PCa) is a leading cause of cancer-related deaths among men, and accurate diagnosis is critical for effective management. Multiparametric MRI (mpMRI) has become an essential tool in PCa diagnosis due to its superior spatial resolution which enables detailed anatomical, functional information and its resultant ability to detect clinically significant PCa. However, challenges such as subjective interpretation methods and high inter-reader variability remain. In recent years, artificial intelligence (AI) has emerged as a promising solution to enhance the diagnostic performance of mpMRI by automating key tasks such as prostate segmentation, lesion detection, classification.

AREAS COVERED

This review provides a comprehensive overview of the current AI applications in prostate mpMRI, discussing advancements in automated image analysis and how AI-driven models are developed to improve detection and risk stratification. A literature search was conducted to examine both machine learning and deep learning techniques applied in this field, highlighting key studies and future directions.

EXPERT OPINION

While AI models have shown significant promise, their clinical integration remains limited due to the need for larger, multi-institutional validation studies. As AI continues to evolve, multimodal approaches combining imaging with clinical data are likely to play pivotal role in personalized PCa diagnosis, treatment planning.

Evaluation and comparison of synthetic computed tomography algorithms with 3T MRI for prostate radiotherapy: AI-based versus bulk density method

PURPOSE

Synthetic computed tomography (sCT)-algorithms, which generate computed tomography images from magnetic resonance imaging data, are becoming part of the clinical radiotherapy workflow. The aim of this retrospective study was to evaluate and compare commercial bulk-density-method (BM)-based and AI (artificial intelligence)-based-algorithms using 3T magnetic resonance imaging (MRI) with patient data as part of the local clinical commissioning process.

METHODS

44 prostate radiotherapy patients were subjected to MRI and treatment planning CT (TPCT) scans. From the MRI images, sCT images with two different sCT algorithms were generated. The sCT images were evaluated by visual inspection of artifacts. Both sCT methods were compared to TPCT, with Dice similarity score(DSC) of bone and body contours, DVH parameters for CTV, bladder and rectum, and gamma-analysis. Accuracy for treatment alignment using sCT images was also tested. Various limits were used to define whether the differences between sCT methods to TPCT were clinically relevant (DVH parameters <2%, gamma-analysis passing rates 90%, 95%, and 98%, and the DSC 0.98 for body and 0.7 for bone).

RESULTS

Our results show that, differences in CTV-dose coverage values were <2% in most of the patients with both sCT algorithms when compared to reference dose coverage. While AI-sCT had mean dose coverage difference <0,5% and BM-sCT <1%. Gamma-analysis showed that the AI-sCT mean passing rates were 95.4%, 98.6%, and 99.4% with 1mm1%, 2mm2%, and 3mm3% criteria, respectively. Similarly for BM-sCT the mean passing rates were 93.4%, 98.2%, and 99.2%. For the treatment alignment accuracy, the mean difference in magnitude of the translational shifts was 1.43 mm for BM-sCT and 1.57 mm for AI-sCT. Even though AI-sCT showed statistically better correspondence to TPCT, the differences were not clinically relevant with any of the limits. Visual evaluation showed artifacts in the AI-sCT especially in the bowel area and fiducial markers were not generated with either of the sCT algorithms.

CONCLUSIONS

In conclusion, sCT-algorithms were clinically usable on prostate treatments using 3T MR-only workflow. While AI-sCT showed better correspondence to TPCT than BM-sCT, it generated characteristic artifacts. As sCT algorithms perform well, we still recommend testing the sCT-algorithms with retrospective analyses from patient data prior to implementing sCT into the routine workflow to better understand the specific limitations and capabilities of these algorithms.

CycleSeg: Simultaneous synthetic CT generation and unsupervised segmentation for MR-only radiotherapy treatment planning of prostate cancer

BACKGROUND

MR-only radiotherapy treatment planning is an attractive alternative to conventional workflow, reducing scan time and ionizing radiation. It is crucial to derive the electron density map or synthetic CT (sCT) from MR data to perform dose calculations to enable MR-only treatment planning. Automatic segmentation of relevant organs in MR images can accelerate the process by preventing the time-consuming manual contouring step. However, the segmentation label is available only for CT data in many cases.

PURPOSE

We propose CycleSeg, a unified framework that generates sCT and corresponding segmentation from MR images without access to MR segmentation labels METHODS: CycleSeg utilizes the CycleGAN formulation to perform unpaired synthesis of sCT and image alignment. To enable MR (sCT) segmentation, CycleSeg incorporates unsupervised domain adaptation by using a pseudo-labeling approach with feature alignment in semantic segmentation space. In contrast to previous approaches that perform segmentation on MR data, CycleSeg could perform segmentation on both MR and sCT. Experiments were performed with data from prostate cancer patients, with 78/7/10 subjects in the training/validation/test sets, respectively.

RESULTS

CycleSeg showed the best sCT generation results, with the lowest mean absolute error of 102.2 and the lowest Fréchet inception distance of 13.0. CycleSeg also performed best on MR segmentation, with the highest average dice score of 81.0 and 81.1 for MR and sCT segmentation, respectively. Ablation experiments confirmed the contribution of the proposed components of CycleSeg.

CONCLUSION

CycleSeg effectively synthesized CT and performed segmentation on MR images of prostate cancer patients. Thus, CycleSeg has the potential to expedite MR-only radiotherapy treatment planning, reducing the prescribed scans and manual segmentation effort, and increasing throughput.

Channel-wise attention enhanced and structural similarity constrained cycleGAN for effective synthetic CT generation from head and neck MRI images

BACKGROUND

Magnetic resonance imaging (MRI) plays an increasingly important role in radiotherapy, enhancing the accuracy of target and organs at risk delineation, but the absence of electron density information limits its further clinical application. Therefore, the aim of this study is to develop and evaluate a novel unsupervised network (cycleSimulationGAN) for unpaired MR-to-CT synthesis.

METHODS

The proposed cycleSimulationGAN in this work integrates contour consistency loss function and channel-wise attention mechanism to synthesize high-quality CT-like images. Specially, the proposed cycleSimulationGAN constrains the structural similarity between the synthetic and input images for better structural retention characteristics. Additionally, we propose to equip a novel channel-wise attention mechanism based on the traditional generator of GAN to enhance the feature representation capability of deep network and extract more effective features. The mean absolute error (MAE) of Hounsfield Units (HU), peak signal-to-noise ratio (PSNR), root-mean-square error (RMSE) and structural similarity index (SSIM) were calculated between synthetic CT (sCT) and ground truth (GT) CT images to quantify the overall sCT performance.

RESULTS

One hundred and sixty nasopharyngeal carcinoma (NPC) patients who underwent volumetric-modulated arc radiotherapy (VMAT) were enrolled in this study. The generated sCT of our method were more consistent with the GT compared with other methods in terms of visual inspection. The average MAE, RMSE, PSNR, and SSIM calculated over twenty patients were 61.88 ± 1.42, 116.85 ± 3.42, 36.23 ± 0.52 and 0.985 ± 0.002 for the proposed method. The four image quality assessment metrics were significantly improved by our approach compared to conventional cycleGAN, the proposed cycleSimulationGAN produces significantly better synthetic results except for SSIM in bone.

CONCLUSIONS

We developed a novel cycleSimulationGAN model that can effectively create sCT images, making them comparable to GT images, which could potentially benefit the MRI-based treatment planning.

Machine Learning for Medical Image Translation: A Systematic Review

BACKGROUND

CT scans are often the first and only form of brain imaging that is performed to inform treatment plans for neurological patients due to its time- and cost-effective nature. However, MR images give a more detailed picture of tissue structure and characteristics and are more likely to pick up abnormalities and lesions. The purpose of this paper is to review studies which use deep learning methods to generate synthetic medical images of modalities such as MRI and CT.

METHODS

A literature search was performed in March 2023, and relevant articles were selected and analyzed. The year of publication, dataset size, input modality, synthesized modality, deep learning architecture, motivations, and evaluation methods were analyzed.

RESULTS

A total of 103 studies were included in this review, all of which were published since 2017. Of these, 74% of studies investigated MRI to CT synthesis, and the remaining studies investigated CT to MRI, Cross MRI, PET to CT, and MRI to PET. Additionally, 58% of studies were motivated by synthesizing CT scans from MRI to perform MRI-only radiation therapy. Other motivations included synthesizing scans to aid diagnosis and completing datasets by synthesizing missing scans.

CONCLUSIONS

Considerably more research has been carried out on MRI to CT synthesis, despite CT to MRI synthesis yielding specific benefits. A limitation on medical image synthesis is that medical datasets, especially paired datasets of different modalities, are lacking in size and availability; it is therefore recommended that a global consortium be developed to obtain and make available more datasets for use. Finally, it is recommended that work be carried out to establish all uses of the synthesis of medical scans in clinical practice and discover which evaluation methods are suitable for assessing the synthesized images for these needs.

Comprehensive dose evaluation of a Deep Learning based synthetic Computed Tomography algorithm for pelvic Magnetic Resonance-only radiotherapy

BACKGROUND AND PURPOSE

Magnetic Resonance (MR)-only radiotherapy enables the use of MR without the uncertainty of MR-Computed Tomography (CT) registration. This requires a synthetic CT (sCT) for dose calculations, which can be facilitated by a novel Zero Echo Time (ZTE) sequence where bones are visible and images are acquired in 65 seconds. This study evaluated the dose calculation accuracy for pelvic sites of a ZTE-based Deep Learning sCT algorithm developed by GE Healthcare.

MATERIALS AND METHODS

ZTE and CT images were acquired in 56 pelvic radiotherapy patients in the radiotherapy position. A 2D U-net convolutional neural network was trained using pairs of deformably registered CT and ZTE images from 36 patients. In the remaining 20 patients the dosimetric accuracy of the sCT was assessed using cylindrical dummy Planning Target Volumes (PTVs) positioned at four different central axial locations, as well as the clinical treatment plans (for prostate (n=10), rectum (n=4) and anus (n=6) cancers). The sCT was rigidly and deformably registered, the plan recalculated and the doses compared using mean differences and gamma analysis.

RESULTS

Mean dose differences to the PTV D98% were ≤ 0.5% for all dummy PTVs and clinical plans (rigid registration). Mean gamma pass rates at 1%/1 mm were 98.0 ± 0.4% (rigid) and 100.0 ± 0.0% (deformable), 96.5 ± 0.8% and 99.8 ± 0.1%, and 95.4 ± 0.6% and 99.4 ± 0.4% for the clinical prostate, rectum and anus plans respectively.

CONCLUSIONS

A ZTE-based sCT algorithm with high dose accuracy throughout the pelvis has been developed. This suggests the algorithm is sufficiently accurate for MR-only radiotherapy for all pelvic sites.

Deep Learning in MRI-guided Radiation Therapy: A Systematic Review

MRI-guided radiation therapy (MRgRT) offers a precise and adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed. MRI-guided radiation therapy offers a precise, adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed with emphasis placed on underlying methods. Studies are further categorized into the areas of segmentation, synthesis, radiomics, and real time MRI. Finally, clinical implications, current challenges, and future directions are discussed.

Synthetic computed tomography generation for abdominal adaptive radiotherapy using low-field magnetic resonance imaging

Background and Purpose

Magnetic Resonance guided Radiotherapy (MRgRT) still needs the acquisition of Computed Tomography (CT) images and co-registration between CT and Magnetic Resonance Imaging (MRI). The generation of synthetic CT (sCT) images from the MR data can overcome this limitation. In this study we aim to propose a Deep Learning (DL) based approach for sCT image generation for abdominal Radiotherapy using low field MR images.

Materials and methods

CT and MR images were collected from 76 patients treated on abdominal sites. U-Net and conditional Generative Adversarial Network (cGAN) architectures were used to generate sCT images. Additionally, sCT images composed of only six bulk densities were generated with the aim of having a Simplified sCT.Radiotherapy plans calculated using the generated images were compared to the original plan in terms of gamma pass rate and Dose Volume Histogram (DVH) parameters.

Results

sCT images were generated in 2 s and 2.5 s with U-Net and cGAN architectures respectively.Gamma pass rates for 2%/2mm and 3%/3mm criteria were 91% and 95% respectively. Dose differences within 1% for DVH parameters on the target volume and organs at risk were obtained.

Conclusion

U-Net and cGAN architectures are able to generate abdominal sCT images fast and accurately from low field MRI.

Generation and Evaluation of Synthetic Computed Tomography (CT) from Cone-Beam CT (CBCT) by Incorporating Feature-Driven Loss into Intensity-Based Loss Functions in Deep Convolutional Neural Network

Simple Summary

Despite numerous benefits of cone-beam computed tomography (CBCT), its applications to radiotherapy were limited mainly due to degraded image quality. Recently, enhancing the CBCT image quality by generating synthetic CT image by deep convolutional neural network (CNN) has become frequent. Most of the previous works, however, generated synthetic CT with simple, classical intensity-driven loss in network training, while not specifying a full-package of verifications. This work trained the network by combining feature- and intensity-driven losses and attempted to demonstrate clinical relevance of the synthetic CT images by assessing both image similarity and dose calculating accuracy throughout a commercial Monte-Carlo.

Abstract

Deep convolutional neural network (CNN) helped enhance image quality of cone-beam computed tomography (CBCT) by generating synthetic CT. Most of the previous works, however, trained network by intensity-based loss functions, possibly undermining to promote image feature similarity. The verifications were not sufficient to demonstrate clinical applicability, either. This work investigated the effect of variable loss functions combining feature- and intensity-driven losses in synthetic CT generation, followed by strengthening the verification of generated images in both image similarity and dosimetry accuracy. The proposed strategy highlighted the feature-driven quantification in (1) training the network by perceptual loss, besides L1 and structural similarity (SSIM) losses regarding anatomical similarity, and (2) evaluating image similarity by feature mapping ratio (FMR), besides conventional metrics. In addition, the synthetic CT images were assessed in terms of dose calculating accuracy by a commercial Monte-Carlo algorithm. The network was trained with 50 paired CBCT-CT scans acquired at the same CT simulator and treatment unit to constrain environmental factors any other than loss functions. For 10 independent cases, incorporating perceptual loss into L1 and SSIM losses outperformed the other combinations, which enhanced FMR of image similarity by 10%, and the dose calculating accuracy by 1–2% of gamma passing rate in 1%/1mm criterion.

Improving the clinical workflow of a MR-Linac by dosimetric evaluation of synthetic CT

Adaptive radiotherapy performed on the daily magnetic resonance imaging (MRI) is an option to improve the treatment quality. In the adapt-to-shape workflow of 1.5-T MR-Linac, the contours of structures are adjusted on the basis of patient daily MRI, and the adapted plan is recalculated on the MRI-based synthetic computed tomography (syCT) generated by bulk density assignment. Because dosimetric accuracy of this strategy is a priority and requires evaluation, this study aims to explore the usefulness of adding an assessment of dosimetric errors associated with recalculation on syCT to the clinical workflow. Sixty-one patients, with various tumor sites, treated using a 1.5-T MR-Linac were included in this study. In Monaco V5.4, the target and organs at risk (OARs) were contoured, and a reference CT plan that contains information about the outlined contours, their average electron density (ED), and the priority of ED assignment was generated. To evaluate the dosimetric error of syCT caused by the inherent approximation within bulk density assignment, the reference CT plan was recalculated on the syCT obtained from the reference CT by forcing all contoured structures to their mean ED defined on the reference plan. The dose–volume histogram (DVH) and dose distribution of the CT and syCT plan were compared. The causes of dosimetric discrepancies were investigated, and the reference plan was reworked to minimize errors if needed. For 54 patients, gamma analysis of the dose distribution on syCT and CT show a median pass rate of 99.7% and 98.5% with the criteria of 3%/3 mm and 2%/2 mm, respectively. DVH difference of targets and OARs remained less than 1.5% or 1 Gy. For the remaining patients, factors (i.e., inappropriate ED assignments) influenced the dosimetric agreement of the syCT vs. CT reference DVH by up to 21%. The causes of the errors were promptly identified, and the DVH dosimetry was realigned except for two lung treatments for which a significant discrepancy remained. The recalculation on the syCT obtained from the planning CT is a powerful tool to assess and decrease the minimal error committed during the adaptive plan on the MRI-based syCT.

Synthetic 18F-FDG PET Image Generation Using a Combination of Biomathematical Modeling and Machine Learning

No previous works have attempted to combine generative adversarial network (GAN) architectures and the biomathematical modeling of positron emission tomography (PET) radiotracer uptake in tumors to generate extra training samples. Here, we developed a novel computational model to produce synthetic 18F-fluorodeoxyglucose (18F-FDG) PET images of solid tumors in different stages of progression and angiogenesis. First, a comprehensive biomathematical model is employed for creating tumor-induced angiogenesis, intravascular and extravascular fluid flow, as well as modeling of the transport phenomena and reaction processes of 18F-FDG in a tumor microenvironment. Then, a deep convolutional GAN (DCGAN) model is employed for producing synthetic PET images using 170 input images of 18F-FDG uptake in each of 10 different tumor microvascular networks. The interstitial fluid parameters and spatiotemporal distribution of 18F-FDG uptake in tumor and healthy tissues have been compared against previously published numerical and experimental studies, indicating the accuracy of the model. The structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) of the generated PET sample and the experimental one are 0.72 and 28.53, respectively. Our results demonstrate that a combination of biomathematical modeling and GAN-based augmentation models provides a robust framework for the non-invasive and accurate generation of synthetic PET images of solid tumors in different stages.