Denoising diffusion probabilistic models for 3D medical image generation

Generation of synthetic PET/MR fusion images from MR images using a combination of generative adversarial networks and conditional denoising diffusion probabilistic models based on simultaneous 18F-FDG PET/MR image data of pyogenic spondylodiscitis

BACKGROUND CONTEXT

Cross-modality image generation from magnetic resonance (MR) to positron emission tomography (PET) using the generative model can be expected to have complementary effects by addressing the limitations and maximizing the advantages inherent in each modality.

PURPOSE

This study aims to generate synthetic PET/MR fusion images from MR images using a combination of generative adversarial networks (GANs) and conditional denoising diffusion probabilistic models (cDDPMs) based on simultaneous 18F-fluorodeoxyglucose (18F-FDG) PET/MR image data.

STUDY DESIGN

Retrospective study with prospectively collected clinical and radiological data.

PATIENT SAMPLE

This study included 94 patients (60 men and 34 women) with thoraco-lumbar pyogenic spondylodiscitis (PSD) from February 2017 to January 2020 in a single tertiary institution.

OUTCOME MEASURES

Quantitative and qualitative image similarity were analyzed between the real and synthetic PET/ T2-weighted fat saturation MR (T2FS) fusion images on the test data set.

METHODS

We used paired spinal sagittal T2FS and PET/T2FS fusion images of simultaneous 18F-FDG PET/MR imaging examination in patients with PSD, which were employed to generate synthetic PET/T2FS fusion images from T2FS images using a combination of Pix2Pix (U-Net generator + Least Squares GANs discriminator) and cDDPMs algorithms. In the analyses of image similarity between the real and synthetic PET/T2FS fusion images, we adopted the values of mean peak signal to noise ratio (PSNR), mean structural similarity measurement (SSIM), mean absolute error (MAE), and mean squared error (MSE) for quantitative analysis, while the discrimination accuracy by three spine surgeons was applied for qualitative analysis.

RESULTS

Total of 2,082 pairs of T2FS and PET/T2FS fusion images were obtained from 172 examinations on 94 patients, which were randomly assigned to training, validation, and test data sets in 8:1:1 ratio (1664, 209, and 209 pairs). The quantitative analysis revealed PSNR of 30.634 ± 3.437, SSIM of 0.910 ± 0.067, MAE of 0.017 ± 0.008, and MSE of 0.001 ± 0.001, respectively. The values of PSNR, MAE, and MSE significantly decreased as FDG uptake increased in real PET/T2FS fusion image, with no significant correlation on SSIM. In the qualitative analysis, the overall discrimination accuracy between real and synthetic PET/T2FS fusion images was 47.4%.

CONCLUSIONS

The combination of Pix2Pix and cDDPMs demonstrated the potential for cross-modal image generation from MR to PET images, with reliable quantitative and qualitative image similarities.

Generative models improve fairness of medical classifiers under distribution shifts

Domain generalization is a ubiquitous challenge for machine learning in healthcare. Model performance in real-world conditions might be lower than expected because of discrepancies between the data encountered during deployment and development. Underrepresentation of some groups or conditions during model development is a common cause of this phenomenon. This challenge is often not readily addressed by targeted data acquisition and 'labeling' by expert clinicians, which can be prohibitively expensive or practically impossible because of the rarity of conditions or the available clinical expertise. We hypothesize that advances in generative artificial intelligence can help mitigate this unmet need in a steerable fashion, enriching our training dataset with synthetic examples that address shortfalls of underrepresented conditions or subgroups. We show that diffusion models can automatically learn realistic augmentations from data in a label-efficient manner. We demonstrate that learned augmentations make models more robust and statistically fair in-distribution and out of distribution. To evaluate the generality of our approach, we studied three distinct medical imaging contexts of varying difficulty: (1) histopathology, (2) chest X-ray and (3) dermatology images. Complementing real samples with synthetic ones improved the robustness of models in all three medical tasks and increased fairness by improving the accuracy of clinical diagnosis within underrepresented groups, especially out of distribution.

Lesion-conditioning of synthetic MRI-derived subtraction-MIPs of the breast using a latent diffusion model

The purpose of this feasibility study is to investigate if latent diffusion models (LDMs) are capable to generate contrast enhanced (CE) MRI-derived subtraction maximum intensity projections (MIPs) of the breast, which are conditioned by lesions. We trained an LDM with n = 2832 CE-MIPs of breast MRI examinations of n = 1966 patients (median age: 50 years) acquired between the years 2015 and 2020. The LDM was subsequently conditioned with n = 756 segmented lesions from n = 407 examinations, indicating their location and BI-RADS scores. By applying the LDM, synthetic images were generated from the segmentations of an independent validation dataset. Lesions, anatomical correctness, and realistic impression of synthetic and real MIP images were further assessed in a multi-rater study with five independent raters, each evaluating n = 204 MIPs (50% real/50% synthetic images). The detection of synthetic MIPs by the raters was akin to random guessing with an AUC of 0.58. Interrater reliability of the lesion assessment was high both for real (Kendall's W = 0.77) and synthetic images (W = 0.85). A higher AUC was observed for the detection of suspicious lesions (BI-RADS ≥ 4) in synthetic MIPs (0.88 vs. 0.77; p = 0.051). Our results show that LDMs can generate lesion-conditioned MRI-derived CE subtraction MIPs of the breast, however, they also indicate that the LDM tended to generate rather typical or 'textbook representations' of lesions.

Improved outcome models with denoising diffusion

PURPOSE

Radiotherapy outcome modelling often suffers from class imbalance in the modelled endpoints. One of the main options to address this issue is by introducing new synthetically generated datapoints, using generative models, such as Denoising Diffusion Probabilistic Models (DDPM). In this study, we implemented DDPM to improve performance of a tumor local control model, trained on imbalanced dataset, and compare this approach with other common techniques.

METHODS

A dataset of 535 NSCLC patients treated with SBRT (50 Gy/5 fractions) was used to train a deep learning outcome model for tumor local control prediction. The dataset included complete treatment planning data (planning CT images, 3D planning dose distribution and patient demographics) with sparsely distributed endpoints (6-7 % experiencing local failure). Consequently, we trained a novel conditional 3D DDPM model to generate synthetic treatment planning data. Synthetically generated treatment planning datapoints were used to supplement the real training dataset and the improvement in the model's performance was studied. Obtained results were also compared to other common techniques for class imbalanced training, such as Oversampling, Undersampling, Augmentation, Class Weights, SMOTE and ADASYN.

RESULTS

Synthetic DDPM-generated data were visually trustworthy, with Fréchet inception distance (FID) below 50. Extending the training dataset with the synthetic data improved the model's performance by more than 10%, while other techniques exhibited only about 4% improvement.

CONCLUSIONS

DDPM introduces a novel approach to class-imbalanced outcome modelling problems. The model generates realistic synthetic radiotherapy planning data, with a strong potential to increase performance and robustness of outcome models.

Denoising diffusion probabilistic models for generation of realistic fully-annotated microscopy image datasets

Recent advances in computer vision have led to significant progress in the generation of realistic image data, with denoising diffusion probabilistic models proving to be a particularly effective method. In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets through an unsupervised and intuitive approach, using rough sketches of desired structures as the starting point. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches and enables the segmentation of diverse datasets without the need for human annotations. We demonstrate that segmentation models trained with a small set of synthetic image data reach accuracy levels comparable to those of generalist models trained with a large and diverse collection of manually annotated image data, thereby offering a streamlined and specialized application of segmentation models.

Deep learning based synthesis of MRI, CT and PET: Review and analysis

Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis.

Synthesizing 3D Multi-Contrast Brain Tumor MRIs Using Tumor Mask Conditioning

Data scarcity and data imbalance are two major challenges in training deep learning models on medical images, such as brain tumor MRI data. The recent advancements in generative artificial intelligence have opened new possibilities for synthetically generating MRI data, including brain tumor MRI scans. This approach can be a potential solution to mitigate the data scarcity problem and enhance training data availability. This work focused on adapting the 2D latent diffusion models to generate 3D multi-contrast brain tumor MRI data with a tumor mask as the condition. The framework comprises two components: a 3D autoencoder model for perceptual compression and a conditional 3D Diffusion Probabilistic Model (DPM) for generating high-quality and diverse multi-contrast brain tumor MRI samples, guided by a conditional tumor mask. Unlike existing works that focused on generating either 2D multi-contrast or 3D single-contrast MRI samples, our models generate multi-contrast 3D MRI samples. We also integrated a conditional module within the UNet backbone of the DPM to capture the semantic class-dependent data distribution driven by the provided tumor mask to generate MRI brain tumor samples based on a specific brain tumor mask. We trained our models using two brain tumor datasets: The Cancer Genome Atlas (TCGA) public dataset and an internal dataset from the University of Texas Southwestern Medical Center (UTSW). The models were able to generate high-quality 3D multi-contrast brain tumor MRI samples with the tumor location aligned by the input condition mask. The quality of the generated images was evaluated using the Fréchet Inception Distance (FID) score. This work has the potential to mitigate the scarcity of brain tumor data and improve the performance of deep learning models involving brain tumor MRI data.

Addiction-related brain networks identification via Graph Diffusion Reconstruction Network

Functional magnetic resonance imaging (fMRI) provides insights into complex patterns of brain functional changes, making it a valuable tool for exploring addiction-related brain connectivity. However, effectively extracting addiction-related brain connectivity from fMRI data remains challenging due to the intricate and non-linear nature of brain connections. Therefore, this paper proposed the Graph Diffusion Reconstruction Network (GDRN), a novel framework designed to capture addiction-related brain connectivity from fMRI data acquired from addicted rats. The proposed GDRN incorporates a diffusion reconstruction module that effectively maintains the unity of data distribution by reconstructing the training samples, thereby enhancing the model's ability to reconstruct nicotine addiction-related brain networks. Experimental evaluations conducted on a nicotine addiction rat dataset demonstrate that the proposed GDRN effectively explores nicotine addiction-related brain connectivity. The findings suggest that the GDRN holds promise for uncovering and understanding the complex neural mechanisms underlying addiction using fMRI data.

A Critical Assessment of Generative Models for Synthetic Data Augmentation on Limited Pneumonia X-ray Data

In medical imaging, deep learning models serve as invaluable tools for expediting diagnoses and aiding specialized medical professionals in making clinical decisions. However, effectively training deep learning models typically necessitates substantial quantities of high-quality data, a resource often lacking in numerous medical imaging scenarios. One way to overcome this deficiency is to artificially generate such images. Therefore, in this comparative study we train five generative models to artificially increase the amount of available data in such a scenario. This synthetic data approach is evaluated on a a downstream classification task, predicting four causes for pneumonia as well as healthy cases on 1082 chest X-ray images. Quantitative and medical assessments show that a Generative Adversarial Network (GAN)-based approach significantly outperforms more recent diffusion-based approaches on this limited dataset with better image quality and pathological plausibility. We show that better image quality surprisingly does not translate to improved classification performance by evaluating five different classification models and varying the amount of additional training data. Class-specific metrics like precision, recall, and F1-score show a substantial improvement by using synthetic images, emphasizing the data rebalancing effect of less frequent classes. However, overall performance does not improve for most models and configurations, except for a DreamBooth approach which shows a +0.52 improvement in overall accuracy. The large variance of performance impact in this study suggests a careful consideration of utilizing generative models for limited data scenarios, especially with an unexpected negative correlation between image quality and downstream classification improvement.

Dual convolutional neural network for aberration pre-correction and image quality enhancement in integral imaging display

This paper proposes a method that utilizes a dual neural network model to address the challenges posed by aberration in the integral imaging microlens array (MLA) and the degradation of 3D image quality. The approach involves a cascaded dual convolutional neural network (CNN) model designed to handle aberration pre-correction and image quality restoration tasks. By training these models end-to-end, the MLA aberration is corrected effectively and the image quality of integral imaging is enhanced. The feasibility of the proposed method is validated through simulations and optical experiments, using an optimized, high-quality pre-corrected element image array (EIA) as the image source for 3D display. The proposed method achieves high-quality integral imaging 3D display by alleviating the contradiction between MLA aberration and 3D image resolution reduction caused by system noise without introducing additional complexity to the display system.

The Objective Dementia Severity Scale Based on MRI with Contrastive Learning: A Whole Brain Neuroimaging Perspective

In the clinical treatment of Alzheimer's disease, one of the most important tasks is evaluating its severity for diagnosis and therapy. However, traditional testing methods are deficient, such as their susceptibility to subjective factors, incomplete evaluation, low accuracy, or insufficient granularity, resulting in unreliable evaluation scores. To address these issues, we propose an objective dementia severity scale based on MRI (ODSS-MRI) using contrastive learning to automatically evaluate the neurological function of patients. The approach utilizes a deep learning framework and a contrastive learning strategy to mine relevant information from structural magnetic resonance images to obtain the patient's neurological function level score. Given that the model is driven by the patient's whole brain imaging data, but without any possible biased manual intervention or instruction from the physician or patient, it provides a comprehensive and objective evaluation of the patient's neurological function. We conducted experiments on the Alzheimer's disease Neuroimaging Initiative (ADNI) dataset, and the results showed that the proposed ODSS-MRI was correlated with the stages of AD 88.55% better than all existing methods. This demonstrates its efficacy to describe the neurological function changes of patients during AD progression. It also outperformed traditional psychiatric rating scales in discriminating different stages of AD, which is indicative of its superiority for neurological function evaluation.

A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis

Although generative adversarial networks (GANs) can produce large datasets, their limited diversity and fidelity have been recently addressed by denoising diffusion probabilistic models, which have demonstrated superiority in natural image synthesis. In this study, we introduce Medfusion, a conditional latent DDPM designed for medical image generation, and evaluate its performance against GANs, which currently represent the state-of-the-art. Medfusion was trained and compared with StyleGAN-3 using fundoscopy images from the AIROGS dataset, radiographs from the CheXpert dataset, and histopathology images from the CRCDX dataset. Based on previous studies, Progressively Growing GAN (ProGAN) and Conditional GAN (cGAN) were used as additional baselines on the CheXpert and CRCDX datasets, respectively. Medfusion exceeded GANs in terms of diversity (recall), achieving better scores of 0.40 compared to 0.19 in the AIROGS dataset, 0.41 compared to 0.02 (cGAN) and 0.24 (StyleGAN-3) in the CRMDX dataset, and 0.32 compared to 0.17 (ProGAN) and 0.08 (StyleGAN-3) in the CheXpert dataset. Furthermore, Medfusion exhibited equal or higher fidelity (precision) across all three datasets. Our study shows that Medfusion constitutes a promising alternative to GAN-based models for generating high-quality medical images, leading to improved diversity and less artifacts in the generated images.

Latest Developments in Adapting Deep Learning for Assessing TAVR Procedures and Outcomes

Aortic valve defects are among the most prevalent clinical conditions. A severely damaged or non-functioning aortic valve is commonly replaced with a bioprosthetic heart valve (BHV) via the transcatheter aortic valve replacement (TAVR) procedure. Accurate pre-operative planning is crucial for a successful TAVR outcome. Assessment of computational fluid dynamics (CFD), finite element analysis (FEA), and fluid-solid interaction (FSI) analysis offer a solution that has been increasingly utilized to evaluate BHV mechanics and dynamics. However, the high computational costs and the complex operation of computational modeling hinder its application. Recent advancements in the deep learning (DL) domain can offer a real-time surrogate that can render hemodynamic parameters in a few seconds, thus guiding clinicians to select the optimal treatment option. Herein, we provide a comprehensive review of classical computational modeling approaches, medical imaging, and DL approaches for planning and outcome assessment of TAVR. Particularly, we focus on DL approaches in previous studies, highlighting the utilized datasets, deployed DL models, and achieved results. We emphasize the critical challenges and recommend several future directions for innovative researchers to tackle. Finally, an end-to-end smart DL framework is outlined for real-time assessment and recommendation of the best BHV design for TAVR. Ultimately, deploying such a framework in future studies will support clinicians in minimizing risks during TAVR therapy planning and will help in improving patient care.