MedFusionGAN: multimodal medical image fusion using an unsupervised deep generative adversarial network

FFSWOAFuse: Multi-modal medical image fusion via fermatean fuzzy set and whale optimization algorithm

Multi-modal medical image fusion (MMIF) plays a crucial role in obtaining valuable and significant information from different medical modalities. This process has generated a single resultant image that is suitable for better clinical assessments and surgical planning. In this study, we proposed a new approach to medical image fusion via fermatean fuzzy set (FFS) and whale optimization algorithm (WOA). In the first phase, a gaussian filter was used to achieve a decomposed base and detailed layers individually. The base layers were transformed into fermatean fuzzy images (FFIs) using an optimized value (λ), obtained by using fermatean fuzzy entropy (FFE). In the second phase, the similarity and texture-based fusion rules for decomposed blocks of two FFIs enhance the textural and contrast details. In the third phase, the whale optimization algorithm (WOA) was employed to generate optimal weights for merging the detailed layers, preserving the significant edge details. In the final stage, the quality fused image was reconstructed by incorporating the fused base and detailed layer components. This manuscript compares fifteen state-of-the-art methods and evaluates the performance of proposed work using ten performance metrics. In both a visual and quantitative sense, the outstanding fusion results demonstrate that the presented model can adequately retain better color and high contrast with significant edge features than the other fusion methods.

Generative AI for synthetic data across multiple medical modalities: A systematic review of recent developments and challenges

This paper presents a comprehensive systematic review of generative models (GANs, VAEs, DMs, and LLMs) used to synthesize various medical data types, including imaging (dermoscopic, mammographic, ultrasound, CT, MRI, and X-ray), text, time-series, and tabular data (EHR). Unlike previous narrowly focused reviews, our study encompasses a broad array of medical data modalities and explores various generative models. Our aim is to offer insights into their current and future applications in medical research, particularly in the context of synthesis applications, generation techniques, and evaluation methods, as well as providing a GitHub repository as a dynamic resource for ongoing collaboration and innovation. Our search strategy queries databases such as Scopus, PubMed, and ArXiv, focusing on recent works from January 2021 to November 2023, excluding reviews and perspectives. This period emphasizes recent advancements beyond GANs, which have been extensively covered in previous reviews. The survey also emphasizes the aspect of conditional generation, which is not focused on in similar work. Key contributions include a broad, multi-modality scope that identifies cross-modality insights and opportunities unavailable in single-modality surveys. While core generative techniques are transferable, we find that synthesis methods often lack sufficient integration of patient-specific context, clinical knowledge, and modality-specific requirements tailored to the unique characteristics of medical data. Conditional models leveraging textual conditioning and multimodal synthesis remain underexplored but offer promising directions for innovation. Our findings are structured around three themes: (1) Synthesis applications, highlighting clinically valid synthesis applications and significant gaps in using synthetic data beyond augmentation, such as for validation and evaluation; (2) Generation techniques, identifying gaps in personalization and cross-modality innovation; and (3) Evaluation methods, revealing the absence of standardized benchmarks, the need for large-scale validation, and the importance of privacy-aware, clinically relevant evaluation frameworks. These findings emphasize the need for benchmarking and comparative studies to promote openness and collaboration.

Multimodal medical image fusion combining saliency perception and generative adversarial network

Multimodal medical image fusion is crucial for enhancing diagnostic accuracy by integrating complementary information from different imaging modalities. Current fusion techniques face challenges in effectively combining heterogeneous features while preserving critical diagnostic information. This paper presents a Temporal Decomposition Network (TDN), a novel deep learning architecture that optimizes multimodal medical image fusion through feature-level temporal analysis and adversarial learning mechanisms. The TDN architecture incorporates two key components: a salient perception model for discriminative feature extraction and a generative adversarial network for temporal feature matching. The salient perception model identifies and classifies distinct pixel distributions across different imaging modalities, while the adversarial component facilitates accurate feature mapping and fusion. This approach enables precise temporal Decomposition of heterogeneous features and robust quality assessment of fused regions. Experimental validation on diverse medical image datasets, encompassing multiple modalities and image dimensions, demonstrates the TDN's superior performance. Compared to state-of-the-art methods, the framework achieves an 11.378% improvement in fusion accuracy and a 12.441% enhancement in precision. These results indicate significant potential for clinical applications, particularly in radiological diagnosis, surgical planning, and medical image analysis, where multimodal visualization is critical for accurate interpretation and decision-making.

MRI super-resolution reconstruction using efficient diffusion probabilistic model with residual shifting

Objective

MRI offers superior soft-tissue contrast yet suffers from long acquisition times that can induce patient discomfort and motion artifacts. Super-resolution (SR) methods reconstruct high-resolution (HR) images from low-resolution (LR) scans, but diffusion models typically require numerous sampling steps, hindering real-time use. Here, we introduce a residual error-shifting strategy that reduce sampling steps without compromising anatomical fidelity, thereby improving MRI SR for clinical deployment.

Approach

We propose a novel diffusion-based SR framework called Res-SRDiff, which integrates residual error shifting into the forward diffusion process. This approach enables efficient HR image reconstruction by aligning the degraded HR image distribution with the LR image distribution. Our model was evaluated on two MRI datasets: ultra-high-field brain T1 MP2RAGE maps and T2-weighted prostate images. We compared Res-SRDiff against established methods, including Bicubic, Pix2pix, CycleGAN, and a conventional denoising diffusion probabilistic model with vision transformer backbone (TM-DDPM), using quantitative metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), gradient magnitude similarity deviation (GMSD), and learned perceptual image patch similarity (LPIPS).

Main results

Res-SRDiff significantly outperformed all comparative methods in terms of PSNR, SSIM, and GMSD across both datasets, with statistically significant improvements ( p -values ≪ 0.05). The model achieved high-fidelity image restoration with only four sampling steps, drastically reducing computational time to under one second per slice, which is substantially faster than conventional TM-DDPM with around 20 seconds per slice. Qualitative analyses further demonstrated that Res-SRDiff effectively preserved fine anatomical details and lesion morphology in both brain and pelvic MRI images.

Significance

Our findings show that Res-SRDiff is an efficient and accurate MRI SR method, markedly improving computational efficiency and image quality. By integrating residual error shifting into the diffusion process, it allows for rapid and robust HR image reconstruction, enhancing clinical MRI workflows and advancing medical imaging research. The source at: https://github.com/mosaf/Res-SRDiff.

Advancing MRI Reconstruction: A Systematic Review of Deep Learning and Compressed Sensing Integration

Magnetic resonance imaging (MRI) is a non-invasive imaging modality and provides comprehensive anatomical and functional insights into the human body. However, its long acquisition times can lead to patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, strategies such as parallel imaging have been applied, which utilize multiple receiver coils to speed up the data acquisition process. Additionally, compressed sensing (CS) is a method that facilitates image reconstruction from sparse data, significantly reducing image acquisition time by minimizing the amount of data collection needed. Recently, deep learning (DL) has emerged as a powerful tool for improving MRI reconstruction. It has been integrated with parallel imaging and CS principles to achieve faster and more accurate MRI reconstructions. This review comprehensively examines DL-based techniques for MRI reconstruction. We categorize and discuss various DL-based methods, including end-to-end approaches, unrolled optimization, and federated learning, highlighting their potential benefits. Our systematic review highlights significant contributions and underscores the potential of DL in MRI reconstruction. Additionally, we summarize key results and trends in DL-based MRI reconstruction, including quantitative metrics, the dataset, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based MRI reconstruction in advancing medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based MRI reconstruction publications and public datasets-https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Perfusion parameter map generation from TOF-MRA in stroke using generative adversarial networks

PURPOSE

To generate perfusion parameter maps from Time-of-flight magnetic resonance angiography (TOF-MRA) images using artificial intelligence to provide an alternative to traditional perfusion imaging techniques.

MATERIALS AND METHODS

This retrospective study included a total of 272 patients with cerebrovascular diseases; 200 with acute stroke (from 2010 to 2018), and 72 with steno-occlusive disease (from 2011 to 2014). For each patient the TOF MRA image and the corresponding Dynamic susceptibility contrast magnetic resonance imaging (DSC-MRI) were retrieved from the datasets. The authors propose an adapted generative adversarial network (GAN) architecture, 3D pix2pix GAN, that generates common perfusion maps (CBF, CBV, MTT, TTP, Tmax) from TOF-MRA images. The performance was evaluated by the structural similarity index measure (SSIM). For a subset of 20 patients from the acute stroke dataset, the Dice coefficient was calculated to measure the overlap between the generated and real hypoperfused lesions with a time-to-maximum (Tmax) > 6 s.

RESULTS

The GAN model exhibited high visual overlap and performance for all perfusion maps in both datasets: acute stroke (mean SSIM 0.88-0.92, mean PSNR 28.48-30.89, mean MAE 0.02-0.04 and mean NRMSE 0.14-0.37) and steno-occlusive disease patients (mean SSIM 0.83-0.98, mean PSNR 23.62-38.21, mean MAE 0.01-0.05 and mean NRMSE 0.03-0.15). For the overlap analysis for lesions with Tmax>6 s, the median Dice coefficient was 0.49.

CONCLUSION

Our AI model can successfully generate perfusion parameter maps from TOF-MRA images, paving the way for a non-invasive alternative for assessing cerebral hemodynamics in cerebrovascular disease patients. This method could impact the stratification of patients with cerebrovascular diseases. Our results warrant more extensive refinement and validation of the method.

Fast MRI Reconstruction Using Deep Learning-based Compressed Sensing: A Systematic Review

Magnetic resonance imaging (MRI) has revolutionized medical imaging, providing a non-invasive and highly detailed look into the human body. However, the long acquisition times of MRI present challenges, causing patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, researchers are exploring various techniques to reduce acquisition time and improve the overall efficiency of MRI. One such technique is compressed sensing (CS), which reduces data acquisition by leveraging image sparsity in transformed spaces. In recent years, deep learning (DL) has been integrated with CS-MRI, leading to a new framework that has seen remarkable growth. DL-based CS-MRI approaches are proving to be highly effective in accelerating MR imaging without compromising image quality. This review comprehensively examines DL-based CS-MRI techniques, focusing on their role in increasing MR imaging speed. We provide a detailed analysis of each category of DL-based CS-MRI including end-to-end, unroll optimization, self-supervised, and federated learning. Our systematic review highlights significant contributions and underscores the exciting potential of DL in CS-MRI. Additionally, our systematic review efficiently summarizes key results and trends in DL-based CS-MRI including quantitative metrics, the dataset used, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based CS-MRI in the advancement of medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based CS-MRI publications and publicly available datasets - https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Using Sparse Parts in Fused Information to Enhance Performance in Latent Low-Rank Representation-Based Fusion of Visible and Infrared Images

Latent Low-Rank Representation (LatLRR) has emerged as a prominent approach for fusing visible and infrared images. In this approach, images are decomposed into three fundamental components: the base part, salient part, and sparse part. The aim is to blend the base and salient features to reconstruct images accurately. However, existing methods often focus more on combining the base and salient parts, neglecting the importance of the sparse component, whereas we advocate for the comprehensive inclusion of all three parts generated from LatLRR image decomposition into the image fusion process, a novel proposition introduced in this study. Moreover, the effective integration of Convolutional Neural Network (CNN) technology with LatLRR remains challenging, particularly after the inclusion of sparse parts. This study utilizes fusion strategies involving weighted average, summation, VGG19, and ResNet50 in various combinations to analyze the fusion performance following the introduction of sparse parts. The research findings show a significant enhancement in fusion performance achieved through the inclusion of sparse parts in the fusion process. The suggested fusion strategy involves employing deep learning techniques for fusing both base parts and sparse parts while utilizing a summation strategy for the fusion of salient parts. The findings improve the performance of LatLRR-based methods and offer valuable insights for enhancement, leading to advancements in the field of image fusion.