Deep learning-based convolutional neural network for intramodality brain MRI synthesis

Medical image translation with deep learning: Advances, datasets and perspectives

Traditional medical image generation often lacks patient-specific clinical information, limiting its clinical utility despite enhancing downstream task performance. In contrast, medical image translation precisely converts images from one modality to another, preserving both anatomical structures and cross-modal features, thus enabling efficient and accurate modality transfer and offering unique advantages for model development and clinical practice. This paper reviews the latest advancements in deep learning(DL)-based medical image translation. Initially, it elaborates on the diverse tasks and practical applications of medical image translation. Subsequently, it provides an overview of fundamental models, including convolutional neural networks (CNNs), transformers, and state space models (SSMs). Additionally, it delves into generative models such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), Autoregressive Models (ARs), diffusion Models, and flow Models. Evaluation metrics for assessing translation quality are discussed, emphasizing their importance. Commonly used datasets in this field are also analyzed, highlighting their unique characteristics and applications. Looking ahead, the paper identifies future trends, challenges, and proposes research directions and solutions in medical image translation. It aims to serve as a valuable reference and inspiration for researchers, driving continued progress and innovation in this area.

Coupling of state space modules and attention mechanisms: An input-aware multi-contrast MRI synthesis method

BACKGROUND

Medical imaging plays a pivotal role in the real-time monitoring of patients during the diagnostic and therapeutic processes. However, in clinical scenarios, the acquisition of multi-modal imaging protocols is often impeded by a number of factors, including time and economic costs, the cooperation willingness of patients, imaging quality, and even safety concerns.

PURPOSE

We proposed a learning-based medical image synthesis method to simplify the acquisition of multi-contrast MRI.

METHODS

We redesigned the basic structure of the Mamba block and explored different integration patterns between Mamba layers and Transformer layers to make it more suitable for medical image synthesis tasks. Experiments were conducted on the IXI (a total of 575 samples, training set: 450 samples; validation set: 25 samples; test set: 100 samples) and BRATS (a total of 494 samples, training set: 350 samples; validation set: 44 samples; test set: 100 samples) datasets to assess the synthesis performance of our proposed method in comparison to some state-of-the-art models on the task of multi-contrast MRI synthesis.

RESULTS

Our proposed model outperformed other state-of-the-art models in some multi-contrast MRI synthesis tasks. In the synthesis task from T1 to PD, our proposed method achieved the peak signal-to-noise ratio (PSNR) of 33.70 dB (95% CI, 33.61, 33.79) and the structural similarity index (SSIM) of 0.966 (95% CI, 0.964, 0.968). In the synthesis task from T2 to PD, the model achieved a PSNR of 33.90 dB (95% CI, 33.82, 33.98) and SSMI of 0.971 (95% CI, 0.969, 0.973). In the synthesis task from FLAIR to T2, the model achieved PSNR of 30.43 dB (95% CI, 30.29, 30.57) and SSIM of 0.938 (95% CI, 0.935, 0.941).

CONCLUSIONS

Our proposed method could effectively model not only the high-dimensional, nonlinear mapping relationships between the magnetic signals of the hydrogen nucleus in tissues and the proton density signals in tissues, but also of the recovery process of suppressed liquid signals in FLAIR. The model proposed in our work employed distinct mechanisms in the synthesis of images belonging to normal and lesion samples, which demonstrated that our model had a profound comprehension of the input data. We also proved that in a hierarchical network, only the deeper self-attention layers were responsible for directing more attention on lesion areas.

Generating synthetic brain PET images of synaptic density based on MR T1 images using deep learning

PURPOSE

Synaptic vesicle glycoprotein 2 A (SV2A) in human brains is an important biomarker of synaptic loss associated with several neurological disorders. However, SV2A tracers, such as [11C]UCB-J, are less available in practice due to constrains such as cost, radiation exposure and onsite cyclotron. We therefore aim to generate synthetic [11C]UCB-J PET images based on MRI in this study.

METHODS

We implemented a convolution-based 3D encoder-decoder to predict [11C]UCB-J SV2A PET images. A total of 160 participants who underwent both MRI and [11C]UCB-J PET imaging, including individuals with schizophrenia, cannabis use disorder, Alzheimer's disease, were used in this study. The model was trained on pairs of T1-weighted MRI and [11C]UCB-J distribution volume ratio images, and tested through a 10-fold cross-validation process. The image translation accuracy was evaluated based on the mean squared error, structural similarity index, percentage bias and Pearson's correlation coefficient between the ground truth and the predicted images. Additionally, we assessed the prediction accuracy of selected regions of interest (ROIs) crucial for brain disorders to evaluate our results.

RESULTS

The generated SV2A PET images are visually similar to the ground truth in terms of contrast and tracer distribution, quantitatively with low bias (< 2%) and high similarity (> 0.9). Across all diagnostic categories and ROIs, including the hippocampus, frontal, occipital, parietal, and temporal regions, the synthetic SV2A PET images exhibit an average bias of less than 5% compared to the ground truth. The model also demonstrates a capacity for noise reduction, producing images of higher quality compared to the low-dose scans.

CONCLUSION

We conclude that it is feasible to generate robust SV2A PET images with promising accuracy from MRI via a data-driven approach.

Validation of ten federated learning strategies for multi-contrast image-to-image MRI data synthesis from heterogeneous sources

Deep learning (DL)-based image synthesis has recently gained enormous interest in medical imaging, allowing for generating multi-contrast data and therefore, the recovery of missing samples from interrupted or artefact-distorted acquisitions. However, the accuracy of DL models heavily relies on the representativeness of the training datasets naturally characterized by their distributions, experimental setups or preprocessing schemes. These complicate generalizing DL models across multi-site heterogeneous data sets while maintaining the confidentiality of the data. One of the possible solutions is to employ federated learning (FL), which enables the collaborative training of a DL model in a decentralized manner, demanding the involved sites to share only the characteristics of the models without transferring their sensitive medical data. The paper presents a DL-based magnetic resonance (MR) data translation in a FL way. We introduce a new aggregation strategy called FedBAdam that couples two state-of-the-art methods with complementary strengths by incorporating momentum in the aggregation scheme and skipping the batch normalization layers. The work comprehensively validates 10 FL-based strategies for an image-to-image multi-contrast MR translation, considering healthy and tumorous brain scans from five different institutions. Our study has revealed that the FedBAdam shows superior results in terms of mean squared error and structural similarity index over personalized methods, like the FedMRI, and standard FL-based aggregation techniques, such as the FedAvg or FedProx, considering multi-site multi-vendor heterogeneous environment. The FedBAdam has prevented the overfitting of the model and gradually reached the optimal model parameters, exhibiting no oscillations.

Leptomeningeal metastatic disease: new frontiers and future directions

Leptomeningeal metastatic disease (LMD), encompassing entities of 'meningeal carcinomatosis', neoplastic meningitis' and 'leukaemic/lymphomatous meningitis', arises secondary to the metastatic dissemination of cancer cells from extracranial and certain intracranial malignancies into the leptomeninges and cerebrospinal fluid. The clinical burden of LMD has been increasing secondary to more sensitive diagnostics, aggressive local therapies for discrete brain metastases, and improved management of extracranial disease with targeted and immunotherapeutic agents, resulting in improved survival. However, owing to drug delivery challenges and the unique microenvironment of LMD, novel therapies against systemic disease have not yet translated into improved outcomes for these patients. Underdiagnosis and misdiagnosis are common, response assessment remains challenging, and the prognosis associated with this disease of whole neuroaxis remains extremely poor. The dearth of effective therapies is further challenged by the difficulties in studying this dynamic disease state. In this Review, a multidisciplinary group of experts describe the emerging evidence and areas of active investigation in LMD and provide directed recommendations for future research. Drawing upon paradigm-changing advances in mechanistic science, computational approaches, and trial design, the authors discuss domain-specific and cross-disciplinary strategies for optimizing the clinical and translational research landscape for LMD. Advances in diagnostics, multi-agent intrathecal therapies, cell-based therapies, immunotherapies, proton craniospinal irradiation and ongoing clinical trials offer hope for improving outcomes for patients with LMD.

Deep learning for the harmonization of structural MRI scans: a survey

Medical imaging datasets for research are frequently collected from multiple imaging centers using different scanners, protocols, and settings. These variations affect data consistency and compatibility across different sources. Image harmonization is a critical step to mitigate the effects of factors like inherent differences between various vendors, hardware upgrades, protocol changes, and scanner calibration drift, as well as to ensure consistent data for medical image processing techniques. Given the critical importance and widespread relevance of this issue, a vast array of image harmonization methodologies have emerged, with deep learning-based approaches driving substantial advancements in recent times. The goal of this review paper is to examine the latest deep learning techniques employed for image harmonization by analyzing cutting-edge architectural approaches in the field of medical image harmonization, evaluating both their strengths and limitations. This paper begins by providing a comprehensive fundamental overview of image harmonization strategies, covering three critical aspects: established imaging datasets, commonly used evaluation metrics, and characteristics of different scanners. Subsequently, this paper analyzes recent structural MRI (Magnetic Resonance Imaging) harmonization techniques based on network architecture, network learning algorithm, network supervision strategy, and network output. The underlying architectures include U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based generative models, transformer-based approaches, as well as custom-designed network architectures. This paper investigates the effectiveness of Disentangled Representation Learning (DRL) as a pivotal learning algorithm in harmonization. Lastly, the review highlights the primary limitations in harmonization techniques, specifically the lack of comprehensive quantitative comparisons across different methods. The overall aim of this review is to serve as a guide for researchers and practitioners to select appropriate architectures based on their specific conditions and requirements. It also aims to foster discussions around ongoing challenges in the field and shed light on promising future research directions with the potential for significant advancements.

Deep Learning for MRI Segmentation and Molecular Subtyping in Glioblastoma: Critical Aspects from an Emerging Field

Deep learning (DL) has been applied to glioblastoma (GBM) magnetic resonance imaging (MRI) assessment for tumor segmentation and inference of molecular, diagnostic, and prognostic information. We comprehensively overviewed the currently available DL applications, critically examining the limitations that hinder their broader adoption in clinical practice and molecular research. Technical limitations to the routine application of DL include the qualitative heterogeneity of MRI, related to different machinery and protocols, and the absence of informative sequences, possibly compensated by artificial image synthesis. Moreover, taking advantage from the available benchmarks of MRI, algorithms should be trained on large amounts of data. Additionally, the segmentation of postoperative imaging should be further addressed to limit the inaccuracies previously observed for this task. Indeed, molecular information has been promisingly integrated in the most recent DL tools, providing useful prognostic and therapeutic information. Finally, ethical concerns should be carefully addressed and standardized to allow for data protection. DL has provided reliable results for GBM assessment concerning MRI analysis and segmentation, but the routine clinical application is still limited. The current limitations could be prospectively addressed, giving particular attention to data collection, introducing new technical advancements, and carefully regulating ethical issues.

A deep-learning-based scatter correction with water equivalent path length map for digital radiography

We proposed a new deep learning (DL) model for accurate scatter correction in digital radiography. The proposed network featured a pixel-wise water equivalent path length (WEPL) map of subjects with diverse sizes and 3D inner structures. The proposed U-Net model comprises two concatenated modules: one for generating a WEPL map and the other for predicting scatter using the WEPL map as auxiliary information. First, 3D CT images were used as numerical phantoms for training and validation, generating observed and scattered images by Monte Carlo simulation, and WEPL maps using Siddon's algorithm. Then, we optimised the model without overfitting. Next, we validated the proposed model's performance by comparing it with other DL models. The proposed model obtained scatter-corrected images with a peak signal-to-noise ratio of 44.24 ± 2.89 dB and a structural similarity index measure of 0.9987 ± 0.0004, which were higher than other DL models. Finally, scatter fractions (SFs) were compared with other DL models using an actual phantom to confirm practicality. Among DL models, the proposed model showed the smallest deviation from measured SF values. Furthermore, using an actual radiograph containing an acrylic object, the contrast-to-noise ratio (CNR) of the proposed model and the anti-scatter grid were compared. The CNR of the images corrected using the proposed model are 16% and 82% higher than those of the raw and grid-applied images, respectively. The advantage of the proposed method is that no actual radiography system is required for collecting training dataset, as the dataset is created from CT images using Monte Carlo simulation.

ReeGAN: MRI image edge-preserving synthesis based on GANs trained with misaligned data

As a crucial medical examination technique, different modalities of magnetic resonance imaging (MRI) complement each other, offering multi-angle and multi-dimensional insights into the body's internal information. Therefore, research on MRI cross-modality conversion is of great significance, and many innovative techniques have been explored. However, most methods are trained on well-aligned data, and the impact of misaligned data has not received sufficient attention. Additionally, many methods focus on transforming the entire image and ignore crucial edge information. To address these challenges, we propose a generative adversarial network based on multi-feature fusion, which effectively preserves edge information while training on noisy data. Notably, we consider images with limited range random transformations as noisy labels and use an additional small auxiliary registration network to help the generator adapt to the noise distribution. Moreover, we inject auxiliary edge information to improve the quality of synthesized target modality images. Our goal is to find the best solution for cross-modality conversion. Comprehensive experiments and ablation studies demonstrate the effectiveness of the proposed method.

Application of MR images in radiotherapy planning for brain tumor based on deep learning

PURPOSE

Explore the function and dose calculation accuracy of MRI images in radiotherapy planning through deep learning methods.

METHODS

131 brain tumor patients undergoing radiotherapy with previous MR and CT images were recruited for this study. A new series of MRI from the aligned MR was firstly registered to CT images strictly using MIM software and then resampled. A deep learning method (U-NET) was used to establish a MRI-to-CT conversion model, for which 105 patient images were used as the training set and 26 patient images were used as the tuning set. Data from additional 8 patients were collected as the test set, and the accuracy of the model was evaluated from a dosimetric standpoint.

RESULTS

Comparing the synthetic CT images with the original CT images, the difference in dosimetric parameters D98, D95, D2 and Dmean of PTV in 8 patients was less than 0.5%. The gamma passed rates of PTV and whole body volume were: 1%/1 mm: 93.96%±6.75%, 2%/2 mm: 99.87%±0.30%, 3%/3 mm: 100.00%±0.00%; and 1%/1 mm: 99.14%±0.80%, 2%/2 mm: 99.92%±0.08%, 3%/3 mm: 99.99%±0.01%.

CONCLUSION

MR images can be used both in delineation and treatment efficacy evaluation and in dose calculation. Using the deep learning way to convert MR image to CT image is a viable method and can be further used in dose calculation.

Deep learning based synthesis of MRI, CT and PET: Review and analysis

Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis.

SuperCUT, an unsupervised multimodal image registration with deep learning for biomedical microscopy

Numerous imaging techniques are available for observing and interrogating biological samples, and several of them can be used consecutively to enable correlative analysis of different image modalities with varying resolutions and the inclusion of structural or molecular information. Achieving accurate registration of multimodal images is essential for the correlative analysis process, but it remains a challenging computer vision task with no widely accepted solution. Moreover, supervised registration methods require annotated data produced by experts, which is limited. To address this challenge, we propose a general unsupervised pipeline for multimodal image registration using deep learning. We provide a comprehensive evaluation of the proposed pipeline versus the current state-of-the-art image registration and style transfer methods on four types of biological problems utilizing different microscopy modalities. We found that style transfer of modality domains paired with fully unsupervised training leads to comparable image registration accuracy to supervised methods and, most importantly, does not require human intervention.

GANcMRI: Cardiac magnetic resonance video generation and physiologic guidance using latent space prompting

Generative artificial intelligence can be applied to medical imaging on tasks such as privacy-preserving image generation and superresolution and denoising of existing images. Few prior approaches have used cardiac magnetic resonance imaging (cMRI) as a modality given the complexity of videos (the addition of the temporal dimension) as well as the limited scale of publicly available datasets. We introduce GANcMRI, a generative adversarial network that can synthesize cMRI videos with physiological guidance based on latent space prompting. GANcMRI uses a StyleGAN framework to learn the latent space from individual video frames and leverages the timedependent trajectory between end-systolic and end-diastolic frames in the latent space to predict progression and generate motion over time. We proposed various methods for modeling latent time-dependent trajectories and found that our Frame-to-frame approach generates the best motion and video quality. GANcMRI generated high-quality cMRI image frames that are indistinguishable by cardiologists, however, artifacts in video generation allow cardiologists to still recognize the difference between real and generated videos. The generated cMRI videos can be prompted to apply physiologybased adjustments which produces clinically relevant phenotypes recognizable by cardiologists. GANcMRI has many potential applications such as data augmentation, education, anomaly detection, and preoperative planning.

Contrast-enhanced MRI synthesis using dense-dilated residual convolutions based 3D network toward elimination of gadolinium in neuro-oncology

Recent studies have raised broad safety and health concerns about using of gadolinium contrast agents during magnetic resonance imaging (MRI) to enhance identification of active tumors. In this paper, we developed a deep learning-based method for three-dimensional (3D) contrast-enhanced T1-weighted (T1) image synthesis from contrast-free image(s). The MR images of 1251 patients with glioma from the RSNA-ASNR-MICCAI BraTS Challenge 2021 dataset were used in this study. A 3D dense-dilated residual U-Net (DD-Res U-Net) was developed for contrast-enhanced T1 image synthesis from contrast-free image(s). The model was trained on a randomly split training set (n = 800) using a customized loss function and validated on a validation set (n = 200) to improve its generalizability. The generated images were quantitatively assessed against the ground-truth on a test set (n = 251) using the mean absolute error (MAE), mean-squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), normalized mutual information (NMI), and Hausdorff distance (HDD) metrics. We also performed a qualitative visual similarity assessment between the synthetic and ground-truth images. The effectiveness of the proposed model was compared with a 3D U-Net baseline model and existing deep learning-based methods in the literature. Our proposed DD-Res U-Net model achieved promising performance for contrast-enhanced T1 synthesis in both quantitative metrics and perceptual evaluation on the test set (n = 251). Analysis of results on the whole brain region showed a PSNR (in dB) of 29.882 ± 5.924, a SSIM of 0.901 ± 0.071, a MAE of 0.018 ± 0.013, a MSE of 0.002 ± 0.002, a HDD of 2.329 ± 9.623, and a NMI of 1.352 ± 0.091 when using only T1 as input; and a PSNR (in dB) of 30.284 ± 4.934, a SSIM of 0.915 ± 0.063, a MAE of 0.017 ± 0.013, a MSE of 0.001 ± 0.002, a HDD of 1.323 ± 3.551, and a NMI of 1.364 ± 0.089 when combining T1 with other MRI sequences. Compared to the U-Net baseline model, our model revealed superior performance. Our model demonstrated excellent capability in generating synthetic contrast-enhanced T1 images from contrast-free MR image(s) of the whole brain region when using multiple contrast-free images as input. Without incorporating tumor mask information during network training, its performance was inferior in the tumor regions compared to the whole brain which requires further improvements to replace the gadolinium administration in neuro-oncology.

[Deep learning method for magnetic resonance imaging fluid-attenuated inversion recovery image synthesis]

Magnetic resonance imaging(MRI) can obtain multi-modal images with different contrast, which provides rich information for clinical diagnosis. However, some contrast images are not scanned or the quality of the acquired images cannot meet the diagnostic requirements due to the difficulty of patient's cooperation or the limitation of scanning conditions. Image synthesis techniques have become a method to compensate for such image deficiencies. In recent years, deep learning has been widely used in the field of MRI synthesis. In this paper, a synthesis network based on multi-modal fusion is proposed, which firstly uses a feature encoder to encode the features of multiple unimodal images separately, and then fuses the features of different modal images through a feature fusion module, and finally generates the target modal image. The similarity measure between the target image and the predicted image in the network is improved by introducing a dynamic weighted combined loss function based on the spatial domain and K-space domain. After experimental validation and quantitative comparison, the multi-modal fusion deep learning network proposed in this paper can effectively synthesize high-quality MRI fluid-attenuated inversion recovery (FLAIR) images. In summary, the method proposed in this paper can reduce MRI scanning time of the patient, as well as solve the clinical problem of missing FLAIR images or image quality that is difficult to meet diagnostic requirements.

Deep learning-based algorithm for postoperative glioblastoma MRI segmentation: a promising new tool for tumor burden assessment

OBJECTIVE

Clinical and surgical decisions for glioblastoma patients depend on a tumor imaging-based evaluation. Artificial Intelligence (AI) can be applied to magnetic resonance imaging (MRI) assessment to support clinical practice, surgery planning and prognostic predictions. In a real-world context, the current obstacles for AI are low-quality imaging and postoperative reliability. The aim of this study is to train an automatic algorithm for glioblastoma segmentation on a clinical MRI dataset and to obtain reliable results both pre- and post-operatively.

METHODS

The dataset used for this study comprises 237 (71 preoperative and 166 postoperative) MRIs from 71 patients affected by a histologically confirmed Grade IV Glioma. The implemented U-Net architecture was trained by transfer learning to perform the segmentation task on postoperative MRIs. The training was carried out first on BraTS2021 dataset for preoperative segmentation. Performance is evaluated using DICE score (DS) and Hausdorff 95% (H95).

RESULTS

In preoperative scenario, overall DS is 91.09 (± 0.60) and H95 is 8.35 (± 1.12), considering tumor core, enhancing tumor and whole tumor (ET and edema). In postoperative context, overall DS is 72.31 (± 2.88) and H95 is 23.43 (± 7.24), considering resection cavity (RC), gross tumor volume (GTV) and whole tumor (WT). Remarkably, the RC segmentation obtained a mean DS of 63.52 (± 8.90) in postoperative MRIs.

CONCLUSIONS

The performances achieved by the algorithm are consistent with previous literature for both pre-operative and post-operative glioblastoma's MRI evaluation. Through the proposed algorithm, it is possible to reduce the impact of low-quality images and missing sequences.

Synthesis of diffusion-weighted MRI scalar maps from FLAIR volumes using generative adversarial networks

Introduction

Acquisition and pre-processing pipelines for diffusion-weighted imaging (DWI) volumes are resource- and time-consuming. Generating synthetic DWI scalar maps from commonly acquired brain MRI sequences such as fluid-attenuated inversion recovery (FLAIR) could be useful for supplementing datasets. In this work we design and compare GAN-based image translation models for generating DWI scalar maps from FLAIR MRI for the first time.

Methods

We evaluate a pix2pix model, two modified CycleGANs using paired and unpaired data, and a convolutional autoencoder in synthesizing DWI fractional anisotropy (FA) and mean diffusivity (MD) from whole FLAIR volumes. In total, 420 FLAIR and DWI volumes (11,957 images) from multi-center dementia and vascular disease cohorts were used for training/testing. Generated images were evaluated using two groups of metrics: (1) human perception metrics including peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), (2) structural metrics including a newly proposed histogram similarity (Hist-KL) metric and mean squared error (MSE).

Results

Pix2pix demonstrated the best performance both quantitatively and qualitatively with mean PSNR, SSIM, and MSE metrics of 23.41 dB, 0.8, 0.004, respectively for MD generation, and 24.05 dB, 0.78, 0.004, respectively for FA generation. The new histogram similarity metric demonstrated sensitivity to differences in fine details between generated and real images with mean pix2pix MD and FA Hist-KL metrics of 11.73 and 3.74, respectively. Detailed analysis of clinically relevant regions of white matter (WM) and gray matter (GM) in the pix2pix images also showed strong significant (p < 0.001) correlations between real and synthetic FA values in both tissue types (R = 0.714 for GM, R = 0.877 for WM).

Discussion/conclusion

Our results show that pix2pix's FA and MD models had significantly better structural similarity of tissue structures and fine details than other models, including WM tracts and CSF spaces, between real and generated images. Regional analysis of synthetic volumes showed that synthetic DWI images can not only be used to supplement clinical datasets, but demonstrates potential utility in bypassing or correcting registration in data pre-processing.

Deep-learning-based generation of synthetic 6-minute MRI from 2-minute MRI for use in head and neck cancer radiotherapy

Background

Quick magnetic resonance imaging (MRI) scans with low contrast-to-noise ratio are typically acquired for daily MRI-guided radiotherapy setup. However, for patients with head and neck (HN) cancer, these images are often insufficient for discriminating target volumes and organs at risk (OARs). In this study, we investigated a deep learning (DL) approach to generate high-quality synthetic images from low-quality images.

Methods

We used 108 unique HN image sets of paired 2-minute T2-weighted scans (2mMRI) and 6-minute T2-weighted scans (6mMRI). 90 image sets (~20,000 slices) were used to train a 2-dimensional generative adversarial DL model that utilized 2mMRI as input and 6mMRI as output. Eighteen image sets were used to test model performance. Similarity metrics, including the mean squared error (MSE), structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR) were calculated between normalized synthetic 6mMRI and ground-truth 6mMRI for all test cases. In addition, a previously trained OAR DL auto-segmentation model was used to segment the right parotid gland, left parotid gland, and mandible on all test case images. Dice similarity coefficients (DSC) were calculated between 2mMRI and either ground-truth 6mMRI or synthetic 6mMRI for each OAR; two one-sided t-tests were applied between the ground-truth and synthetic 6mMRI to determine equivalence. Finally, a visual Turing test using paired ground-truth and synthetic 6mMRI was performed using three clinician observers; the percentage of images that were correctly identified was compared to random chance using proportion equivalence tests.

Results

The median similarity metrics across the whole images were 0.19, 0.93, and 33.14 for MSE, SSIM, and PSNR, respectively. The median of DSCs comparing ground-truth vs. synthetic 6mMRI auto-segmented OARs were 0.86 vs. 0.85, 0.84 vs. 0.84, and 0.82 vs. 0.85 for the right parotid gland, left parotid gland, and mandible, respectively (equivalence p<0.05 for all OARs). The percent of images correctly identified was equivalent to chance (p<0.05 for all observers).

Conclusions

Using 2mMRI inputs, we demonstrate that DL-generated synthetic 6mMRI outputs have high similarity to ground-truth 6mMRI, but further improvements can be made. Our study facilitates the clinical incorporation of synthetic MRI in MRI-guided radiotherapy.

A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis

Magnetic resonance (MR) imaging plays an important role in medical diagnosis and treatment; different modalities of MR images can provide rich and complementary information to improve the accuracy of diagnosis. However, due to the limitations of scanning time and medical conditions, certain modalities of MR may be unavailable or of low quality in clinical practice. In this study, we propose a new multimodal MR image synthesis network to generate missing MR images. The proposed model comprises three stages: feature extraction, feature fusion, and image generation. During feature extraction, 2D and 3D self-supervised pretext tasks are introduced to pre-train the backbone for better representations of each modality. Then, a channel attention mechanism is used when fusing features so that the network can adaptively weigh different fusion operations to learn common representations of all modalities. Finally, a generative adversarial network is considered as the basic framework to generate images, in which a feature-level edge information loss is combined with the pixel-wise loss to ensure consistency between the synthesized and real images in terms of anatomical characteristics. 2D and 3D self-supervised pre-training can have better performance on feature extraction to retain more details in the synthetic images. Moreover, the proposed multimodal attention feature fusion block (MAFFB) in the well-designed layer-wise fusion strategy can model both common and unique information in all modalities, consistent with the clinical analysis. We also perform an interpretability analysis to confirm the rationality and effectiveness of our method. The experimental results demonstrate that our method can be applied in both single-modal and multimodal synthesis with high robustness and outperforms other state-of-the-art approaches objectively and subjectively.