Comparison of deep learning synthesis of synthetic CTs using clinical MRI inputs

A systematic literature review: deep learning techniques for synthetic medical image generation and their applications in radiotherapy

The aim of this systematic review is to determine whether Deep Learning (DL) algorithms can provide a clinically feasible alternative to classic algorithms for synthetic Computer Tomography (sCT). The following categories are presented in this study: ∙ MR-based treatment planning and synthetic CT generation techniques. ∙ Generation of synthetic CT images based on Cone Beam CT images. ∙ Low-dose CT to High-dose CT generation. ∙ Attenuation correction for PET images. To perform appropriate database searches, we reviewed journal articles published between January 2018 and June 2023. Current methodology, study strategies, and results with relevant clinical applications were analyzed as we outlined the state-of-the-art of deep learning based approaches to inter-modality and intra-modality image synthesis. This was accomplished by contrasting the provided methodologies with traditional research approaches. The key contributions of each category were highlighted, specific challenges were identified, and accomplishments were summarized. As a final step, the statistics of all the cited works from various aspects were analyzed, which revealed that DL-based sCTs have achieved considerable popularity, while also showing the potential of this technology. In order to assess the clinical readiness of the presented methods, we examined the current status of DL-based sCT generation.

Uncertainty-aware MR-based CT synthesis for robust proton therapy planning of brain tumour

BACKGROUND AND PURPOSE

Deep learning techniques excel in MR-based CT synthesis, but missing uncertainty prediction limits its clinical use in proton therapy. We developed an uncertainty-aware framework and evaluated its efficiency in robust proton planning.

MATERIALS AND METHODS

A conditional generative-adversarial network was trained on 64 brain tumour patients with paired MR-CT images to generate synthetic CTs (sCT) from combined T1-T2 MRs of three orthogonal planes. A Bayesian neural network predicts Laplacian distributions for all voxels with parameters (μ, b). A robust proton plan was optimized using three sCTs of μ and μ±b. The dosimetric differences between the plan from sCT (sPlan) and the recalculated plan (rPlan) on planning CT (pCT) were quantified for each patient. The uncertainty-aware robust plan was compared to conventional robust (global ± 3 %) and non-robust plans.

RESULTS

In 8-fold cross-validation, sCT-pCT image differences (Mean-Absolute-Error) were 80.84 ± 9.84HU (body), 35.78 ± 6.07HU (soft tissues) and 221.88 ± 31.69HU (bones), with Dice scores of 90.33 ± 2.43 %, 95.13 ± 0.80 %, and 85.53 ± 4.16 %, respectively. The uncertainty distribution positively correlated with absolute prediction error (Correlation Coefficient: 0.62 ± 0.01). The uncertainty-conditioned robust optimisation improved the rPlan-sPlan agreement, e.g., D95 absolute difference (CTV) was 1.10 ± 1.24 % compared to conventional (1.64 ± 2.71 %) and non-robust (2.08 ± 2.96 %) optimisation. This trend was consistent across all target and organs-at-risk indexes.

CONCLUSION

The enhanced framework incorporates 3D uncertainty prediction and generates high-quality sCTs from MR images. The framework also facilitates conditioned robust optimisation, bolstering proton plan robustness against network prediction errors. The innovative feature of uncertainty visualisation and robust analyses contribute to evaluating sCT clinical utility for individual patients.

Deep learning in MRI-guided radiation therapy: A systematic review

Recent advances in MRI-guided radiation therapy (MRgRT) and deep learning techniques encourage fully adaptive radiation therapy (ART), real-time MRI monitoring, and the MRI-only treatment planning workflow. Given the rapid growth and emergence of new state-of-the-art methods in these fields, we systematically review 197 studies written on or before December 31, 2022, and categorize the studies into the areas of image segmentation, image synthesis, radiomics, and real time MRI. Building from the underlying deep learning methods, we discuss their clinical importance and current challenges in facilitating small tumor segmentation, accurate x-ray attenuation information from MRI, tumor characterization and prognosis, and tumor motion tracking. In particular, we highlight the recent trends in deep learning such as the emergence of multi-modal, visual transformer, and diffusion models.

Deep-Learning-Aided Evaluation of Spondylolysis Imaged with Ultrashort Echo Time Magnetic Resonance Imaging

Isthmic spondylolysis results in fracture of pars interarticularis of the lumbar spine, found in as many as half of adolescent athletes with persistent low back pain. While computed tomography (CT) is the gold standard for the diagnosis of spondylolysis, the use of ionizing radiation near reproductive organs in young subjects is undesirable. While magnetic resonance imaging (MRI) is preferable, it has lowered sensitivity for detecting the condition. Recently, it has been shown that ultrashort echo time (UTE) MRI can provide markedly improved bone contrast compared to conventional MRI. To take UTE MRI further, we developed supervised deep learning tools to generate (1) CT-like images and (2) saliency maps of fracture probability from UTE MRI, using ex vivo preparation of cadaveric spines. We further compared quantitative metrics of the contrast-to-noise ratio (CNR), mean squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) between UTE MRI (inverted to make the appearance similar to CT) and CT and between CT-like images and CT. Qualitative results demonstrated the feasibility of successfully generating CT-like images from UTE MRI to provide easier interpretability for bone fractures thanks to improved image contrast and CNR. Quantitatively, the mean CNR of bone against defect-filled tissue was 35, 97, and 146 for UTE MRI, CT-like, and CT images, respectively, being significantly higher for CT-like than UTE MRI images. For the image similarity metrics using the CT image as the reference, CT-like images provided a significantly lower mean MSE (0.038 vs. 0.0528), higher mean PSNR (28.6 vs. 16.5), and higher SSIM (0.73 vs. 0.68) compared to UTE MRI images. Additionally, the saliency maps enabled quick detection of the location with probable pars fracture by providing visual cues to the reader. This proof-of-concept study is limited to the data from ex vivo samples, and additional work in human subjects with spondylolysis would be necessary to refine the models for clinical use. Nonetheless, this study shows that the utilization of UTE MRI and deep learning tools could be highly useful for the evaluation of isthmic spondylolysis.

Exploring contrast generalisation in deep learning-based brain MRI-to-CT synthesis

BACKGROUND

Synthetic computed tomography (sCT) has been proposed and increasingly clinically adopted to enable magnetic resonance imaging (MRI)-based radiotherapy. Deep learning (DL) has recently demonstrated the ability to generate accurate sCT from fixed MRI acquisitions. However, MRI protocols may change over time or differ between centres resulting in low-quality sCT due to poor model generalisation.

PURPOSE

investigating domain randomisation (DR) to increase the generalisation of a DL model for brain sCT generation.

METHODS

CT and corresponding T1-weighted MRI with/without contrast, T2-weighted, and FLAIR MRI from 95 patients undergoing RT were collected, considering FLAIR the unseen sequence where to investigate generalisation. A "Baseline" generative adversarial network was trained with/without the FLAIR sequence to test how a model performs without DR. Image similarity and accuracy of sCT-based dose plans were assessed against CT to select the best-performing DR approach against the Baseline.

RESULTS

The Baseline model had the poorest performance on FLAIR, with mean absolute error (MAE) = 106 ± 20.7 HU (mean ±σ). Performance on FLAIR significantly improved for the DR model with MAE = 99.0 ± 14.9 HU, but still inferior to the performance of the Baseline+FLAIR model (MAE = 72.6 ± 10.1 HU). Similarly, an improvement in γ-pass rate was obtained for DR vs Baseline.

CONCLUSION

DR improved image similarity and dose accuracy on the unseen sequence compared to training only on acquired MRI. DR makes the model more robust, reducing the need for re-training when applying a model on sequences unseen and unavailable for retraining.

CBCT-based synthetic CT generated using CycleGAN with HU correction for adaptive radiotherapy of nasopharyngeal carcinoma

This study aims to utilize a hybrid approach of phantom correction and deep learning for synthesized CT (sCT) images generation based on cone-beam CT (CBCT) images for nasopharyngeal carcinoma (NPC). 52 CBCT/CT paired images of NPC patients were used for model training (41), validation (11). Hounsfield Units (HU) of the CBCT images was calibrated by a commercially available CIRS phantom. Then the original CBCT and the corrected CBCT (CBCT_cor) were trained separately with the same cycle generative adversarial network (CycleGAN) to generate SCT1 and SCT2. The mean error and mean absolute error (MAE) were used to quantify the image quality. For validations, the contours and treatment plans in CT images were transferred to original CBCT, CBCT_cor, SCT1 and SCT2 for dosimetric comparison. Dose distribution, dosimetric parameters and 3D gamma passing rate were analyzed. Compared with rigidly registered CT (RCT), the MAE of CBCT, CBCT_cor, SCT1 and SCT2 were 346.11 ± 13.58 HU, 145.95 ± 17.64 HU, 105.62 ± 16.08 HU and 83.51 ± 7.71 HU, respectively. Moreover, the average dosimetric parameter differences for the CBCT_cor, SCT1 and SCT2 were 2.7% ± 1.4%, 1.2% ± 1.0% and 0.6% ± 0.6%, respectively. Using the dose distribution of RCT images as reference, the 3D gamma passing rate of the hybrid method was significantly better than the other methods. The effectiveness of CBCT-based sCT generated using CycleGAN with HU correction for adaptive radiotherapy of nasopharyngeal carcinoma was confirmed. The image quality and dose accuracy of SCT2 were outperform the simple CycleGAN method. This finding has great significance for the clinical application of adaptive radiotherapy for NPC.

Deep learning synthesis of cone-beam computed tomography from zero echo time magnetic resonance imaging

Cone-beam computed tomography (CBCT) produces high-resolution of hard tissue even in small voxel size, but the process is associated with radiation exposure and poor soft tissue imaging. Thus, we synthesized a CBCT image from the magnetic resonance imaging (MRI), using deep learning and to assess its clinical accuracy. We collected patients who underwent both CBCT and MRI simultaneously in our institution (Seoul). MRI data were registered with CBCT data, and both data were prepared into 512 slices of axial, sagittal, and coronal sections. A deep learning-based synthesis model was trained and the output data were evaluated by comparing the original and synthetic CBCT (syCBCT). According to expert evaluation, syCBCT images showed better performance in terms of artifacts and noise criteria but had poor resolution compared to the original CBCT images. In syCBCT, hard tissue showed better clarity with significantly different MAE and SSIM. This study result would be a basis for replacing CBCT with non-radiation imaging that would be helpful for patients planning to undergo both MRI and CBCT.

Deep Learning in MRI-guided Radiation Therapy: A Systematic Review

MRI-guided radiation therapy (MRgRT) offers a precise and adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed. MRI-guided radiation therapy offers a precise, adaptive approach to treatment planning. Deep learning applications which augment the capabilities of MRgRT are systematically reviewed with emphasis placed on underlying methods. Studies are further categorized into the areas of segmentation, synthesis, radiomics, and real time MRI. Finally, clinical implications, current challenges, and future directions are discussed.

Sex Disparities of the Sacroiliac Joint: Focus on Joint Anatomy and Imaging Appearance

The sacroiliac joint (SIJ) is an anatomically complex joint which, as a functional unit with the pelvis and spine, is of decisive biomechanical importance for the human body. It is also a commonly overlooked source of lower back pain. Like the entire bony pelvis, the SIJ exhibits major sexual dimorphisms; thus, the sex-dependent evaluation of this joint is becoming increasingly important in clinical practice, both anatomically with joint shape variations and biomechanical differences as well as in terms of image appearance. The influence of the SIJ shape, which differs in women and men, is crucial for the different biomechanical joint properties. These differences are important in the development of joint diseases at the SIJ, which shows a specific difference between the sexes. This article aims to provide an overview of sex disparities of the SIJ regarding different anatomical and imaging appearances to further understand the insights into the interplay of sex differences and SIJ disease.

Adoption of Hybrid MRI-Linac Systems for the Treatment of Brain Tumors: A Systematic Review of the Current Literature Regarding Clinical and Technical Features

BACKGROUND

Possible advantages of magnetic resonance (MR)-guided radiation therapy (MRgRT) for the treatment of brain tumors include improved definition of treatment volumes and organs at risk (OARs) that could allow margin reductions, resulting in limited dose to the OARs and/or dose escalation to target volumes. Recently, hybrid systems integrating a linear accelerator and an magnetic resonance imaging (MRI) scan (MRI-linacs, MRL) have been introduced, that could potentially lead to a fully MRI-based treatment workflow.

METHODS

We performed a systematic review of the published literature regarding the adoption of MRL for the treatment of primary or secondary brain tumors (last update November 3, 2022), retrieving a total of 2487 records; after a selection based on title and abstracts, the full text of 74 articles was analyzed, finally resulting in the 52 papers included in this review.

RESULTS AND DISCUSSION

Several solutions have been implemented to achieve a paradigm shift from CT-based radiotherapy to MRgRT, such as the management of geometric integrity and the definition of synthetic CT models that estimate electron density. Multiple sequences have been optimized to acquire images with adequate quality with on-board MR scanner in limited times. Various sophisticated algorithms have been developed to compensate the impact of magnetic field on dose distribution and calculate daily adaptive plans in a few minutes with satisfactory dosimetric parameters for the treatment of primary brain tumors and cerebral metastases. Dosimetric studies and preliminary clinical experiences demonstrated the feasibility of treating brain lesions with MRL.

CONCLUSIONS

The adoption of an MRI-only workflow is feasible and could offer several advantages for the treatment of brain tumors, including superior image quality for lesions and OARs and the possibility to adapt the treatment plan on the basis of daily MRI. The growing body of clinical data will clarify the potential benefit in terms of toxicity and response to treatment.

CT-like images in MRI improve specificity of erosion detection in patients with hand arthritis: a diagnostic accuracy study with CT as standard of reference

OBJECTIVE

To compare the diagnostic accuracy of susceptibility-weighted imaging (SWI), standard T1-weighted (T1w) images and high-resolution 3D-gradient echo sequences (volumetric interpolated breath-hold examination (VIBE)) for detection of erosions in patients with peripheral arthritis using CT as standard of reference.

MATERIALS AND METHODS

A total of 36 patients were included in the study. All patients underwent CT and MRI, including SWI, VIBE and T1w sequences of the clinically more affected hand. Two trained readers scored all imaging datasets separately for erosions in a blinded fashion. Specificity, sensitivity and diagnostic accuracy of MRI sequences were calculated on a per-patient level.

RESULTS

CT was positive for erosion in 16 patients and 77 bones (Rheumatoid Arthritis MRI Score >0), T1w in 28 patients, VIBE in 25 patients and SWI in 17 patients. All MRI sequences performed with comparably high sensitivities (T1w 100%, VIBE 94% and SWI 94%). SWI had the highest specificity of 90%, followed by VIBE (50%) and T1w (40%). Both T1w and VIBE produced significantly higher sum scores than CT (341 and 331 vs 148, p<0.0001), while the sum score for SWI did not differ from CT (119 vs 148; p=0.411).

CONCLUSION

Specificity for erosion detection remains a challenge for MRI when conventional and high-resolution sequences are used but can be improved by direct bone depiction with SWI. Both T1w and VIBE tend to overestimate erosions, when CT is used as the standard of reference.

Application of artificial intelligence in brain molecular imaging

Initial development of artificial Intelligence (AI) and machine learning (ML) dates back to the mid-twentieth century. A growing awareness of the potential for AI, as well as increases in computational resources, research, and investment are rapidly advancing AI applications to medical imaging and, specifically, brain molecular imaging. AI/ML can improve imaging operations and decision making, and potentially perform tasks that are not readily possible by physicians, such as predicting disease prognosis, and identifying latent relationships from multi-modal clinical information. The number of applications of image-based AI algorithms, such as convolutional neural network (CNN), is increasing rapidly. The applications for brain molecular imaging (MI) include image denoising, PET and PET/MRI attenuation correction, image segmentation and lesion detection, parametric image formation, and the detection/diagnosis of Alzheimer's disease and other brain disorders. When effectively used, AI will likely improve the quality of patient care, instead of replacing radiologists. A regulatory framework is being developed to facilitate AI adaptation for medical imaging.

Synthesis of pseudo-CT images from pelvic MRI images based on MD-CycleGAN model for radiotherapy

OBJECTIVE

A multi-discriminator-based cycle generative adversarial network (MD-CycleGAN) model was proposed to synthesize higher-quality pseudo-CT from MRI.

APPROACH

The MRI and CT images obtained at the simulation stage with cervical cancer were selected to train the model. The generator adopted the DenseNet as the main architecture. The local and global discriminators based on convolutional neural network jointly discriminated the authenticity of the input image data. In the testing phase, the model was verified by four-fold cross-validation method. In the prediction stage, the data were selected to evaluate the accuracy of the pseudo-CT in anatomy and dosimetry, and they were compared with the pseudo-CT synthesized by GAN with generator based on the architectures of ResNet, sU-Net, and FCN.

MAIN RESULTS

There are significant differences(P<0.05) in the four-fold-cross validation results on peak signal-to-noise ratio and structural similarity index metrics between the pseudo-CT obtained based on MD-CycleGAN and the ground truth CT (CTgt). The pseudo-CT synthesized by MD-CycleGAN had closer anatomical information to the CTgt with root mean square error of 47.83±2.92 HU and normalized mutual information value of 0.9014±0.0212 and mean absolute error value of 46.79±2.76 HU. The differences in dose distribution between the pseudo-CT obtained by MD-CycleGAN and the CTgt were minimal. The mean absolute dose errors of Dosemax, Dosemin and Dosemean based on the planning target volume were used to evaluate the dose uncertainty of the four pseudo-CT. The u-values of the Wilcoxon test were 55.407, 41.82 and 56.208, and the differences were statistically significant. The 2%/2 mm-based gamma pass rate (%) of the proposed method was 95.45±1.91, and the comparison methods (ResNet_GAN, sUnet_GAN and FCN_GAN) were 93.33±1.20, 89.64±1.63 and 87.31±1.94, respectively.

SIGNIFICANCE

The pseudo-CT obtained based on MD-CycleGAN have higher imaging quality and are closer to the CTgt in terms of anatomy and dosimetry than other GAN models.

A deep-learning method for generating synthetic kV-CT and improving tumor segmentation for helical tomotherapy of nasopharyngeal carcinoma

Objective:Megavoltage computed tomography (MV-CT) is used for setup verification and adaptive radiotherapy in tomotherapy. However, its low contrast and high noise lead to poor image quality. This study aimed to develop a deep-learning-based method to generate synthetic kilovoltage CT (skV-CT) and then evaluate its ability to improve image quality and tumor segmentation.Approach:The planning kV-CT and MV-CT images of 270 patients with nasopharyngeal carcinoma (NPC) treated on an Accuray TomoHD system were used. An improved cycle-consistent adversarial network which used residual blocks as its generator was adopted to learn the mapping between MV-CT and kV-CT and then generate skV-CT from MV-CT. A Catphan 700 phantom and 30 patients with NPC were used to evaluate image quality. The quantitative indices included contrast-to-noise ratio (CNR), uniformity and signal-to-noise ratio (SNR) for the phantom and the structural similarity index measure (SSIM), mean absolute error (MAE), and peak signal-to-noise ratio (PSNR) for patients. Next, we trained three models for segmentation of the clinical target volume (CTV): MV-CT, skV-CT, and MV-CT combined with skV-CT. The segmentation accuracy was compared with indices of the dice similarity coefficient (DSC) and mean distance agreement (MDA).Mainresults:Compared with MV-CT, skV-CT showed significant improvement in CNR (184.0%), image uniformity (34.7%), and SNR (199.0%) in the phantom study and improved SSIM (1.7%), MAE (24.7%), and PSNR (7.5%) in the patient study. For CTV segmentation with only MV-CT, only skV-CT, and MV-CT combined with skV-CT, the DSCs were 0.75 ± 0.04, 0.78 ± 0.04, and 0.79 ± 0.03, respectively, and the MDAs (in mm) were 3.69 ± 0.81, 3.14 ± 0.80, and 2.90 ± 0.62, respectively.Significance:The proposed method improved the image quality of MV-CT and thus tumor segmentation in helical tomotherapy. The method potentially can benefit adaptive radiotherapy.

Deep learning methods to generate synthetic CT from MRI in radiotherapy: A literature review

PURPOSE

In radiotherapy, MRI is used for target volume and organs-at-risk delineation for its superior soft-tissue contrast as compared to CT imaging. However, MRI does not provide the electron density of tissue necessary for dose calculation. Several methods of synthetic-CT (sCT) generation from MRI data have been developed for radiotherapy dose calculation. This work reviewed deep learning (DL) sCT generation methods and their associated image and dose evaluation, in the context of MRI-based dose calculation.

METHODS

We searched the PubMed and ScienceDirect electronic databases from January 2010 to March 2021. For each paper, several items were screened and compiled in figures and tables.

RESULTS

This review included 57 studies. The DL methods were either generator-only based (45% of the reviewed studies), or generative adversarial network (GAN) architecture and its variants (55% of the reviewed studies). The brain and pelvis were the most commonly investigated anatomical localizations (39% and 28% of the reviewed studies, respectively), and more rarely, the head-and-neck (H&N) (15%), abdomen (10%), liver (5%) or breast (3%). All the studies performed an image evaluation of sCTs with a diversity of metrics, with only 36 studies performing dosimetric evaluations of sCT.

CONCLUSIONS

The median mean absolute errors were around 76 HU for the brain and H&N sCTs and 40 HU for the pelvis sCTs. For the brain, the mean dose difference between the sCT and the reference CT was <2%. For the H&N and pelvis, the mean dose difference was below 1% in most of the studies. Recent GAN architectures have advantages compared to generator-only, but no superiority was found in term of image or dose sCT uncertainties. Key challenges of DL-based sCT generation methods from MRI in radiotherapy is the management of movement for abdominal and thoracic localizations, the standardization of sCT evaluation, and the investigation of multicenter impacts.

Deep learning based synthetic-CT generation in radiotherapy and PET: A review

Recently,deep learning (DL)-based methods for the generation of synthetic computed tomography (sCT) have received significant research attention as an alternative to classical ones. We present here a systematic review of these methods by grouping them into three categories, according to their clinical applications: (i) to replace computed tomography in magnetic resonance (MR) based treatment planning, (ii) facilitate cone-beam computed tomography based image-guided adaptive radiotherapy, and (iii) derive attenuation maps for the correction of positron emission tomography. Appropriate database searching was performed on journal articles published between January 2014 and December 2020. The DL methods' key characteristics were extracted from each eligible study, and a comprehensive comparison among network architectures and metrics was reported. A detailed review of each category was given, highlighting essential contributions, identifying specific challenges, and summarizing the achievements. Lastly, the statistics of all the cited works from various aspects were analyzed, revealing the popularity and future trends and the potential of DL-based sCT generation. The current status of DL-based sCT generation was evaluated, assessing the clinical readiness of the presented methods.

Prediction of an oxygen extraction fraction map by convolutional neural network: validation of input data among MR and PET images

Purpose

Oxygen extraction fraction (OEF) is a biomarker for the viability of brain tissue in ischemic stroke. However, acquisition of the OEF map using positron emission tomography (PET) with oxygen-15 gas is uncomfortable for patients because of the long fixation time, invasive arterial sampling, and radiation exposure. We aimed to predict the OEF map from magnetic resonance (MR) and PET images using a deep convolutional neural network (CNN) and to demonstrate which PET and MR images are optimal as inputs for the prediction of OEF maps.

Methods

Cerebral blood flow at rest (CBF) and during stress (sCBF), cerebral blood volume (CBV) maps acquired from oxygen-15 PET, and routine MR images (T1-, T2-, and T2*-weighted images) for 113 patients with steno-occlusive disease were learned with U-Net. MR and PET images acquired from the other 25 patients were used as test data. We compared the predicted OEF maps and intraclass correlation (ICC) with the real OEF values among combinations of MRI, CBF, CBV, and sCBF.

Results

Among the combinations of input images, OEF maps predicted by the model learned with MRI, CBF, CBV, and sCBF maps were the most similar to the real OEF maps (ICC: 0.597 ± 0.082). However, the contrast of predicted OEF maps was lower than that of real OEF maps.

Conclusion

These results suggest that the deep CNN learned useful features from CBF, sCBF, CBV, and MR images and predict qualitatively realistic OEF maps. These findings suggest that the deep CNN model can shorten the fixation time for ¹⁵O PET by skipping ¹⁵O₂ scans. Further training with a larger data set is required to predict accurate OEF maps quantitatively.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11548-021-02356-7.