Multi-Contrast Super-Resolution MRI Through a Progressive Network

MMR-Mamba: Multi-modal MRI reconstruction with Mamba and spatial-frequency information fusion

Multi-modal MRI offers valuable complementary information for diagnosis and treatment; however, its clinical utility is limited by prolonged scanning time. To accelerate the acquisition process, a practical approach is to reconstruct images of the target modality, which requires longer scanning time, from under-sampled k-space data using the fully-sampled reference modality with shorter scanning time as guidance. The primary challenge of this task lies in comprehensively and efficiently integrating complementary information from different modalities to achieve high-quality reconstruction. Existing methods struggle with this challenge: (1) convolution-based models fail to capture long-range dependencies; (2) transformer-based models, while excelling in global feature modeling, suffer from quadratic computational complexity. To address this dilemma, we propose MMR-Mamba, a novel framework that thoroughly and efficiently integrates multi-modal features for MRI reconstruction, leveraging Mamba's capability to capture long-range dependencies with linear computational complexity while exploiting global properties of the Fourier domain. Specifically, we first design a Target modality-guided Cross Mamba (TCM) module in the spatial domain, which maximally restores the target modality information by selectively incorporating relevant information from the reference modality. Then, we introduce a Selective Frequency Fusion (SFF) module to efficiently integrate global information in the Fourier domain and recover high-frequency signals for the reconstruction of structural details. Furthermore, we devise an Adaptive Spatial-Frequency Fusion (ASFF) module, which mutually enhances the spatial and frequency domains by supplementing less informative channels from one domain with corresponding channels from the other. Extensive experiments on the BraTS and fastMRI knee datasets demonstrate the superiority of our MMR-Mamba over state-of-the-art reconstruction methods. The code is publicly available at https://github.com/zoujing925/MMR-Mamba.

Partition-based k-space synthesis for multi-contrast parallel imaging

PURPOSE

Multi-contrast magnetic resonance imaging is a significant and essential medical imaging technique. However, multi-contrast imaging has longer acquisition time and is easy to cause motion artifacts. In particular, the acquisition time for a T2-weighted image is prolonged due to its longer repetition time (TR). On the contrary, T1-weighted image has a shorter TR. Therefore, utilizing complementary information across T1 and T2-weighted image is a way to decrease the overall imaging time. Previous T1-assisted T2 reconstruction methods have mostly focused on image domain using whole-based image fusion approaches. The image domain reconstruction method has the defects of high computational complexity and limited flexibility. To address this issue, we propose a novel multi-contrast imaging method called partition-based k-space synthesis (PKS) which can achieve better reconstruction quality of T2-weighted image by feature fusion.

METHODS

Concretely, we first decompose fully-sampled T1 k-space data and under-sampled T2 k-space data into two sub-data, separately. Then two new objects are constructed by combining the two sub-T1/T2 data. After that, the two new objects as the whole data to realize the reconstruction of T2-weighted image.

RESULTS

Experimental results showed that the developed PKS scheme can achieve comparable or better results than using traditional k-space parallel imaging (SAKE) that processes each contrast independently. At the same time, our method showed good adaptability and robustness under different contrast-assisted and T1-T2 ratios. Efficient target modal image reconstruction under various conditions were realized and had excellent performance in restoring image quality and preserving details.

CONCLUSIONS

This work proposed a PKS multi-contrast method to assist in target mode image reconstruction. We have conducted extensive experiments on different multi-contrast, diverse ratios of T1 to T2 and different sampling masks to demonstrate the generalization and robustness of our proposed model.

Dataset augmentation with multiple contrasts images in super-resolution processing of T1-weighted brain magnetic resonance images

This study investigated the effectiveness of augmenting datasets for super-resolution processing of brain Magnetic Resonance Images (MRI) T1-weighted images (T1WIs) using deep learning. By incorporating images with different contrasts from the same subject, this study sought to improve network performance and assess its impact on image quality metrics, such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). This retrospective study included 240 patients who underwent brain MRI. Two types of datasets were created: the Pure-Dataset group comprising T1WIs and the Mixed-Dataset group comprising T1WIs, T2-weighted images, and fluid-attenuated inversion recovery images. A U-Net-based network and an Enhanced Deep Super-Resolution network (EDSR) were trained on these datasets. Objective image quality analysis was performed using PSNR and SSIM. Statistical analyses, including paired t test and Pearson's correlation coefficient, were conducted to evaluate the results. Augmenting datasets with images of different contrasts significantly improved training accuracy as the dataset size increased. PSNR values ranged 29.84-30.26 dB for U-Net trained on mixed datasets, and SSIM values ranged 0.9858-0.9868. Similarly, PSNR values ranged 32.34-32.64 dB for EDSR trained on mixed datasets, and SSIM values ranged 0.9941-0.9945. Significant differences in PSNR and SSIM were observed between models trained on pure and mixed datasets. Pearson's correlation coefficient indicated a strong positive correlation between dataset size and image quality metrics. Using diverse image data obtained from the same subject can improve the performance of deep-learning models in medical image super-resolution tasks.

Advancing MRI Reconstruction: A Systematic Review of Deep Learning and Compressed Sensing Integration

Magnetic resonance imaging (MRI) is a non-invasive imaging modality and provides comprehensive anatomical and functional insights into the human body. However, its long acquisition times can lead to patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, strategies such as parallel imaging have been applied, which utilize multiple receiver coils to speed up the data acquisition process. Additionally, compressed sensing (CS) is a method that facilitates image reconstruction from sparse data, significantly reducing image acquisition time by minimizing the amount of data collection needed. Recently, deep learning (DL) has emerged as a powerful tool for improving MRI reconstruction. It has been integrated with parallel imaging and CS principles to achieve faster and more accurate MRI reconstructions. This review comprehensively examines DL-based techniques for MRI reconstruction. We categorize and discuss various DL-based methods, including end-to-end approaches, unrolled optimization, and federated learning, highlighting their potential benefits. Our systematic review highlights significant contributions and underscores the potential of DL in MRI reconstruction. Additionally, we summarize key results and trends in DL-based MRI reconstruction, including quantitative metrics, the dataset, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based MRI reconstruction in advancing medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based MRI reconstruction publications and public datasets-https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Let UNet Play an Adversarial Game: Investigating the Effect of Adversarial Training in Enhancing Low-Resolution MRI

Adversarial training has attracted much attention in enhancing the visual realism of images, but its efficacy in clinical imaging has not yet been explored. This work investigated adversarial training in a clinical context, by training 206 networks on the OASIS-1 dataset for improving low-resolution and low signal-to-noise ratio (SNR) magnetic resonance images. Each network corresponded to a different combination of perceptual and adversarial loss weights and distinct learning rate values. For each perceptual loss weighting, we identified its corresponding adversarial loss weighting that minimized structural disparity. Each optimally weighted adversarial loss yielded an average SSIM reduction of 1.5%. We further introduced a set of new metrics to assess other clinically relevant image features: Gradient Error (GE) to measure structural disparities; Sharpness to compute edge clarity; and Edge-Contrast Error (ECE) to quantify any distortion of the pixel distribution around edges. Including adversarial loss increased structural enhancement in visual inspection, which correlated with statistically consistent GE reductions (p-value << 0.05). This also resulted in increased Sharpness; however, the level of statistical significance was dependent on the perceptual loss weighting. Additionally, adversarial loss yielded ECE reductions for smaller perceptual loss weightings, while showing non-significant increases (p-value >> 0.05) when these weightings were higher, demonstrating that the increased Sharpness does not adversely distort the pixel distribution around the edges in the image. These studies clearly suggest that adversarial training significantly improves the performance of an MRI enhancement pipeline, and highlights the need for systematic studies of hyperparameter optimization and investigation of alternative image quality metrics.

FDoSR-Net: Frequency-Domain Informed Auto-Encoder Network for Arbitrary-Scale 3D Whole-Heart MRI Super-Resolution

This work aims to develop a three-dimensional (3D) super-resolution (SR) network that would perform arbitrary-scale 3D whole-heart (WH) magnetic resonance imaging (MRI) super-resolution, while maintaining fine image details. One-hundred-twenty 3D WH MR volumes, acquired using four different sequences, are used in this study for training, validation, and testing. The proposed method utilizes a frequency-domain regularization in training to maintain fine image detail along with a 3D autoencoder framework. It is also trained in manner to enable it to perform arbitrary factor SR. The proposed method is compared against multiple super-resolution algorithms including two state-of-the-art deep learning methods referred to here as ACNS and TFC as well as nearest neighbor interpolation. The proposed method was evaluated quantitatively and compared against the competing methods with the mean result of the proposed method and the improvements provided by the proposed method (reported by mean percentage between the proposed method and all other competing methods) were recorded. The metrics of interest used for the quantitative comparison are peak signal-to-noise ratio (PSNR, mean = 34.10, mean percentage of improvement = 4.5%), structural similarity index measure (SSIM, mean = 0.94, mean percentage of improvement = 2.2%), mean squared error (MSE, mean = 0.00094, mean percentage of improvement = 48.2%), and root mean squared error (RMSE, mean = 0.024, mean percentage of improvement = 31.0%). Moreover, qualitative comparison was performed using multiple visual comparisons. The quantitative results achieved demonstrate that the proposed method regularly outperforms all other comparison methods. The visual comparisons demonstrate that the proposed method outperforms current state-of-the-art methods in preserving fine image details, as well as its ability to do so for multiple SR factors.

NExpR: Neural Explicit Representation for fast arbitrary-scale medical image super-resolution

Medical images often require rescaling to various spatial resolutions to ensure interpretations at different levels. Conventional deep learning-based image super-resolution (SR) enhances the fixed-scale resolution. Implicit neural representation (INR) is a promising way of achieving arbitrary-scale image SR. However, existing INR-based methods require the repeated execution of the neural network (NN), which is slow and inefficient. In this paper, we present Neural Explicit Representation (NExpR) for fast arbitrary-scale medical image SR. Our algorithm represents an image with an explicit analytical function, whose input is the low-resolution image and output is the parameterization of the analytical function. After obtaining the analytical representation through a single NN inference, SR images of arbitrary scales can be derived by evaluating the explicit functions at desired coordinates. Because of the analytical explicit representation, NExpR is significantly faster than INR-based methods. In addition to speed, our method achieves on-par or better image quality than other strong competitors. Extensive experiments on Magnetic Resonance Imaging (MRI) datasets, including ProstateX, fastMRI, and our in-house clinical prostate dataset, as well as the Computerized Tomography (CT) dataset, specifically the Medical Segmentation Decathlon (MSD) liver dataset, demonstrate the superiority of our method. Our method reduces the rescaling time from the order of 1 ms to the order of 0.01 ms, achieving an over 100× speedup without losing the image quality. Code is available at https://github.com/Calvin-Pang/NExpR.

Bridging MRI Cross-Modality Synthesis and Multi-Contrast Super-Resolution by Fine-Grained Difference Learning

In multi-modal magnetic resonance imaging (MRI), the tasks of imputing or reconstructing the target modality share a common obstacle: the accurate modeling of fine-grained inter-modal differences, which has been sparingly addressed in current literature. These differences stem from two sources: 1) spatial misalignment remaining after coarse registration and 2) structural distinction arising from modality-specific signal manifestations. This paper integrates the previously separate research trajectories of cross-modality synthesis (CMS) and multi-contrast super-resolution (MCSR) to address this pervasive challenge within a unified framework. Connected through generalized down-sampling ratios, this unification not only emphasizes their common goal in reducing structural differences, but also identifies the key task distinguishing MCSR from CMS: modeling the structural distinctions using the limited information from the misaligned target input. Specifically, we propose a composite network architecture with several key components: a label correction module to align the coordinates of multi-modal training pairs, a CMS module serving as the base model, an SR branch to handle target inputs, and a difference projection discriminator for structural distinction-centered adversarial training. When training the SR branch as the generator, the adversarial learning is enhanced with distinction-aware incremental modulation to ensure better-controlled generation. Moreover, the SR branch integrates deformable convolutions to address cross-modal spatial misalignment at the feature level. Experiments conducted on three public datasets demonstrate that our approach effectively balances structural accuracy and realism, exhibiting overall superiority in comprehensive evaluations for both tasks over current state-of-the-art approaches. The code is available at https://github.com/papshare/FGDL.

Multi-contrast image super-resolution with deformable attention and neighborhood-based feature aggregation (DANCE): Applications in anatomic and metabolic MRI

Multi-contrast magnetic resonance imaging (MRI) reflects information about human tissues from different perspectives and has wide clinical applications. By utilizing the auxiliary information from reference images (Refs) in the easy-to-obtain modality, multi-contrast MRI super-resolution (SR) methods can synthesize high-resolution (HR) images from their low-resolution (LR) counterparts in the hard-to-obtain modality. In this study, we systematically discussed the potential impacts caused by cross-modal misalignments between LRs and Refs and, based on this discussion, proposed a novel deep-learning-based method with Deformable Attention and Neighborhood-based feature aggregation to be Computationally Efficient (DANCE) and insensitive to misalignments. Our method has been evaluated in two public MRI datasets, i.e., IXI and FastMRI, and an in-house MR metabolic imaging dataset with amide proton transfer weighted (APTW) images. Experimental results reveal that our method consistently outperforms baselines in various scenarios, with significant superiority observed in the misaligned group of IXI dataset and the prospective study of the clinical dataset. The robustness study proves that our method is insensitive to misalignments, maintaining an average PSNR of 30.67 dB when faced with a maximum range of ±9°and ±9 pixels of rotation and translation on Refs. Given our method's desirable comprehensive performance, good robustness, and moderate computational complexity, it possesses substantial potential for clinical applications.

ACGRHA-Net: Accelerated multi-contrast MR imaging with adjacency complementary graph assisted residual hybrid attention network

Multi-contrast magnetic resonance (MR) imaging is an advanced technology used in medical diagnosis, but the long acquisition process can lead to patient discomfort and limit its broader application. Shortening acquisition time by undersampling k-space data introduces noticeable aliasing artifacts. To address this, we propose a method that reconstructs multi-contrast MR images from zero-filled data by utilizing a fully-sampled auxiliary contrast MR image as a prior to learn an adjacency complementary graph. This graph is then combined with a residual hybrid attention network, forming the adjacency complementary graph assisted residual hybrid attention network (ACGRHA-Net) for multi-contrast MR image reconstruction. Specifically, the optimal structural similarity is represented by a graph learned from the fully sampled auxiliary image, where the node features and adjacency matrices are designed to precisely capture structural information among different contrast images. This structural similarity enables effective fusion with the target image, improving the detail reconstruction. Additionally, a residual hybrid attention module is designed in parallel with the graph convolution network, allowing it to effectively capture key features and adaptively emphasize these important features in target contrast MR images. This strategy prioritizes crucial information while preserving shallow features, thereby achieving comprehensive feature fusion at deeper levels to enhance multi-contrast MR image reconstruction. Extensive experiments on the different datasets, using various sampling patterns and accelerated factors demonstrate that the proposed method outperforms the current state-of-the-art reconstruction methods.

FEFA: Frequency Enhanced Multi-Modal MRI Reconstruction With Deep Feature Alignment

Integrating complementary information from multiple magnetic resonance imaging (MRI) modalities is often necessary to make accurate and reliable diagnostic decisions. However, the different acquisition speeds of these modalities mean that obtaining information can be time consuming and require significant effort. Reference-based MRI reconstruction aims to accelerate slower, under-sampled imaging modalities, such as T2-modality, by utilizing redundant information from faster, fully sampled modalities, such as T1-modality. Unfortunately, spatial misalignment between different modalities often negatively impacts the final results. To address this issue, we propose FEFA, which consists of cascading FEFA blocks. The FEFA block first aligns and fuses the two modalities at the feature level. The combined features are then filtered in the frequency domain to enhance the important features while simultaneously suppressing the less essential ones, thereby ensuring accurate reconstruction. Furthermore, we emphasize the advantages of combining the reconstruction results from multiple cascaded blocks, which also contributes to stabilizing the training process. Compared to existing registration-then-reconstruction and cross-attention-based approaches, our method is end-to-end trainable without requiring additional supervision, extensive parameters, or heavy computation. Experiments on the public fastMRI, IXI and in-house datasets demonstrate that our approach is effective across various under-sampling patterns and ratios.

Multicontrast MRI Super-Resolution via Transformer-Empowered Multiscale Contextual Matching and Aggregation

Magnetic resonance imaging (MRI) possesses the unique versatility to acquire images under a diverse array of distinct tissue contrasts, which makes multicontrast super-resolution (SR) techniques possible and needful. Compared with single-contrast MRI SR, multicontrast SR is expected to produce higher quality images by exploiting a variety of complementary information embedded in different imaging contrasts. However, existing approaches still have two shortcomings: 1) most of them are convolution-based methods and, hence, weak in capturing long-range dependencies, which are essential for MR images with complicated anatomical patterns and 2) they ignore to make full use of the multicontrast features at different scales and lack effective modules to match and aggregate these features for faithful SR. To address these issues, we develop a novel multicontrast MRI SR network via transformer-empowered multiscale feature matching and aggregation, dubbed McMRSR ++ . First, we tame transformers to model long-range dependencies in both reference and target images at different scales. Then, a novel multiscale feature matching and aggregation method is proposed to transfer corresponding contexts from reference features at different scales to the target features and interactively aggregate them. Furthermore, a texture-preserving branch and a contrastive constraint are incorporated into our framework for enhancing the textural details in the SR images. Experimental results on both public and clinical in vivo datasets show that McMRSR ++ outperforms state-of-the-art methods under peak signal to noise ratio (PSNR), structure similarity index measure (SSIM), and root mean square error (RMSE) metrics significantly. Visual results demonstrate the superiority of our method in restoring structures, demonstrating its great potential to improve scan efficiency in clinical practice.

Joint Under-Sampling Pattern and Dual-Domain Reconstruction for Accelerating Multi-Contrast MRI

Multi-Contrast Magnetic Resonance Imaging (MCMRI) utilizes the short-time reference image to facilitate the reconstruction of the long-time target one, providing a new solution for fast MRI. Although various methods have been proposed, they still have certain limitations. 1) existing methods featuring the preset under-sampling patterns give rise to redundancy between multi-contrast images and limit their model performance; 2) most methods focus on the information in the image domain, prior knowledge in the k-space domain has not been fully explored; and 3) most networks are manually designed and lack certain physical interpretability. To address these issues, we propose a joint optimization of the under-sampling pattern and a deep-unfolding dual-domain network for accelerating MCMRI. Firstly, to reduce the redundant information and sample more contrast-specific information, we propose a new framework to learn the optimal under-sampling pattern for MCMRI. Secondly, a dual-domain model is established to reconstruct the target image in both the image domain and the k-space frequency domain. The model in the image domain introduces a spatial transformation to explicitly model the inconsistent and unaligned structures of MCMRI. The model in the k-space learns prior knowledge from the frequency domain, enabling the model to capture more global information from the input images. Thirdly, we employ the proximal gradient algorithm to optimize the proposed model and then unfold the iterative results into a deep-unfolding network, called MC-DuDoN. We evaluate the proposed MC-DuDoN on MCMRI super-resolution and reconstruction tasks. Experimental results give credence to the superiority of the current model. In particular, since our approach explicitly models the inconsistent structures, it shows robustness on spatially misaligned MCMRI. In the reconstruction task, compared with conventional masks, the learned mask restores more realistic images, even under an ultra-high acceleration ratio ( ×30 ). Code is available at https://github.com/lpcccc-cv/MC-DuDoNet.

A Single-Frame and Multi-Frame Cascaded Image Super-Resolution Method

The objective of image super-resolution is to reconstruct a high-resolution (HR) image with the prior knowledge from one or several low-resolution (LR) images. However, in the real world, due to the limited complementary information, the performance of both single-frame and multi-frame super-resolution reconstruction degrades rapidly as the magnification increases. In this paper, we propose a novel two-step image super resolution method concatenating multi-frame super-resolution (MFSR) with single-frame super-resolution (SFSR), to progressively upsample images to the desired resolution. The proposed method consisting of an L0-norm constrained reconstruction scheme and an enhanced residual back-projection network, integrating the flexibility of the variational model-based method and the feature learning capacity of the deep learning-based method. To verify the effectiveness of the proposed algorithm, extensive experiments with both simulated and real world sequences were implemented. The experimental results show that the proposed method yields superior performance in both objective and perceptual quality measurements. The average PSNRs of the cascade model in set5 and set14 are 33.413 dB and 29.658 dB respectively, which are 0.76 dB and 0.621 dB more than the baseline method. In addition, the experiment indicates that this cascade model can be robustly applied to different SFSR and MFSR methods.

Latent diffusion model-based MRI superresolution enhances mild cognitive impairment prognostication and Alzheimer's disease classification

INTRODUCTION

Timely diagnosis and prognostication of Alzheimer's disease (AD) and mild cognitive impairment (MCI) are pivotal for effective intervention. Artificial intelligence (AI) in neuroradiology may aid in such appropriate diagnosis and prognostication. This study aimed to evaluate the potential of novel diffusion model-based AI for enhancing AD and MCI diagnosis through superresolution (SR) of brain magnetic resonance (MR) images.

METHODS

1.5T brain MR scans of patients with AD or MCI and healthy controls (NC) from Alzheimer's Disease Neuroimaging Initiative 1 (ADNI1) were superresolved to 3T using a novel diffusion model-based generative AI (d3T*) and a convolutional neural network-based model (c3T*). Comparisons of image quality to actual 1.5T and 3T MRI were conducted based on signal-to-noise ratio (SNR), naturalness image quality evaluator (NIQE), and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE). Voxel-based volumetric analysis was then conducted to study whether 3T* images offered more accurate volumetry than 1.5T images. Binary and multiclass classifications of AD, MCI, and NC were conducted to evaluate whether 3T* images offered superior AD classification performance compared to actual 1.5T MRI. Moreover, CNN-based classifiers were used to predict conversion of MCI to AD, to evaluate the prognostication performance of 3T* images. The classification performances were evaluated using accuracy, sensitivity, specificity, F1 score, Matthews correlation coefficient (MCC), and area under the receiver-operating curves (AUROC).

RESULTS

Analysis of variance (ANOVA) detected significant differences in image quality among the 1.5T, c3T*, d3T*, and 3T groups across all metrics. Both c3T* and d3T* showed superior image quality compared to 1.5T MRI in NIQE and BRISQUE with statistical significance. While the hippocampal volumes measured in 3T* and 3T images were not significantly different, the hippocampal volume measured in 1.5T images showed significant difference. 3T*-based AD classifications showed superior performance across all performance metrics compared to 1.5T-based AD classification. Classification performance between d3T* and actual 3T was not significantly different. 3T* images offered superior accuracy in predicting the conversion of MCI to AD than 1.5T images did.

CONCLUSIONS

The diffusion model-based MRI SR enhances the resolution of brain MR images, significantly improving diagnostic and prognostic accuracy for AD and MCI. Superresolved 3T* images closely matched actual 3T MRIs in quality and volumetric accuracy, and notably improved the prediction performance of conversion from MCI to AD.

Learning to deep learning: statistics and a paradigm test in selecting a UNet architecture to enhance MRI

OBJECTIVE

This study aims to assess the statistical significance of training parameters in 240 dense UNets (DUNets) used for enhancing low Signal-to-Noise Ratio (SNR) and undersampled MRI in various acquisition protocols. The objective is to determine the validity of differences between different DUNet configurations and their impact on image quality metrics.

MATERIALS AND METHODS

To achieve this, we trained all DUNets using the same learning rate and number of epochs, with variations in 5 acquisition protocols, 24 loss function weightings, and 2 ground truths. We calculated evaluation metrics for two metric regions of interest (ROI). We employed both Analysis of Variance (ANOVA) and Mixed Effects Model (MEM) to assess the statistical significance of the independent parameters, aiming to compare their efficacy in revealing differences and interactions among fixed parameters.

RESULTS

ANOVA analysis showed that, except for the acquisition protocol, fixed variables were statistically insignificant. In contrast, MEM analysis revealed that all fixed parameters and their interactions held statistical significance. This emphasizes the need for advanced statistical analysis in comparative studies, where MEM can uncover finer distinctions often overlooked by ANOVA.

DISCUSSION

These findings highlight the importance of utilizing appropriate statistical analysis when comparing different deep learning models. Additionally, the surprising effectiveness of the UNet architecture in enhancing various acquisition protocols underscores the potential for developing improved methods for characterizing and training deep learning models. This study serves as a stepping stone toward enhancing the transparency and comparability of deep learning techniques for medical imaging applications.

Multi-Modal Modality-Masked Diffusion Network for Brain MRI Synthesis With Random Modality Missing

Synthesis of unavailable imaging modalities from available ones can generate modality-specific complementary information and enable multi-modality based medical images diagnosis or treatment. Existing generative methods for medical image synthesis are usually based on cross-modal translation between acquired and missing modalities. These methods are usually dedicated to specific missing modality and perform synthesis in one shot, which cannot deal with varying number of missing modalities flexibly and construct the mapping across modalities effectively. To address the above issues, in this paper, we propose a unified Multi-modal Modality-masked Diffusion Network (M2DN), tackling multi-modal synthesis from the perspective of "progressive whole-modality inpainting", instead of "cross-modal translation". Specifically, our M2DN considers the missing modalities as random noise and takes all the modalities as a unity in each reverse diffusion step. The proposed joint synthesis scheme performs synthesis for the missing modalities and self-reconstruction for the available ones, which not only enables synthesis for arbitrary missing scenarios, but also facilitates the construction of common latent space and enhances the model representation ability. Besides, we introduce a modality-mask scheme to encode availability status of each incoming modality explicitly in a binary mask, which is adopted as condition for the diffusion model to further enhance the synthesis performance of our M2DN for arbitrary missing scenarios. We carry out experiments on two public brain MRI datasets for synthesis and downstream segmentation tasks. Experimental results demonstrate that our M2DN outperforms the state-of-the-art models significantly and shows great generalizability for arbitrary missing modalities.

Self-supervised anatomical continuity enhancement network for 7T SWI synthesis from 3T SWI

Synthesizing 7T Susceptibility Weighted Imaging (SWI) from 3T SWI could offer significant clinical benefits by combining the high sensitivity of 7T SWI for neurological disorders with the widespread availability of 3T SWI in diagnostic routines. Although methods exist for synthesizing 7T Magnetic Resonance Imaging (MRI), they primarily focus on traditional MRI modalities like T1-weighted imaging, rather than SWI. SWI poses unique challenges, including limited data availability and the invisibility of certain tissues in individual 3T SWI slices. To address these challenges, we propose a Self-supervised Anatomical Continuity Enhancement (SACE) network to synthesize 7T SWI from 3T SWI using plentiful 3T SWI data and limited 3T-7T paired data. The SACE employs two specifically designed pretext tasks to utilize low-level representations from abundant 3T SWI data for assisting 7T SWI synthesis in a downstream task with limited paired data. One pretext task emphasizes input-specific morphology by balancing the elimination of redundant patterns with the preservation of essential morphology, preventing the blurring of synthetic 7T SWI images. The other task improves the synthesis of tissues that are invisible in a single 3T SWI slice by aligning adjacent slices with the current slice and predicting their difference fields. The downstream task innovatively combines clinical knowledge with brain substructure diagrams to selectively enhance clinically relevant features. When evaluated on a dataset comprising 97 cases (5495 slices), the proposed method achieved a Peak Signal-to-Noise Ratio (PSNR) of 23.05 dB and a Structural Similarity Index (SSIM) of 0.688. Due to the absence of specific methods for 7T SWI, our method was compared with existing enhancement techniques for general 7T MRI synthesis, outperforming these techniques in the context of 7T SWI synthesis. Clinical evaluations have shown that our synthetic 7T SWI is clinically effective, demonstrating its potential as a clinical tool.

Deep Learning With Physics-Embedded Neural Network for Full Waveform Ultrasonic Brain Imaging

The convenience, safety, and affordability of ultrasound imaging make it a vital non-invasive diagnostic technique for examining soft tissues. However, significant differences in acoustic impedance between the skull and soft tissues hinder the successful application of traditional ultrasound for brain imaging. In this study, we propose a physics-embedded neural network with deep learning based full waveform inversion (PEN-FWI), which can achieve reliable quantitative imaging of brain tissues. The network consists of two fundamental components: forward convolutional neural network (FCNN) and inversion sub-neural network (ISNN). The FCNN explores the nonlinear mapping relationship between the brain model and the wavefield, replacing the tedious wavefield calculation process based on the finite difference method. The ISNN implements the mapping from the wavefield to the model. PEN-FWI includes three iterative steps, each embedding the F CNN into the ISNN, ultimately achieving tomography from wavefield to brain models. Simulation and laboratory tests indicate that PEN-FWI can produce high-quality imaging of the skull and soft tissues, even starting from a homogeneous water model. PEN-FWI can achieve excellent imaging of clot models with constant uniform distribution of velocity, randomly Gaussian distribution of velocity, and irregularly shaped randomly distributed velocity. Robust differentiation can also be achieved for brain slices of various tissues and skulls, resulting in high-quality imaging. The imaging time for a horizontal cross-sectional imag e of the brain is only 1.13 seconds. This algorithm can effectively promote ultrasound-based brain tomography and provide feasible solutions in other fields.

Dual-space high-frequency learning for transformer-based MRI super-resolution

BACKGROUND AND OBJECTIVE

Magnetic resonance imaging (MRI) can provide rich and detailed high-contrast information of soft tissues, while the scanning of MRI is time-consuming. To accelerate MR imaging, a variety of Transformer-based single image super-resolution methods are proposed in recent years, achieving promising results thanks to their superior capability of capturing long-range dependencies. Nevertheless, most existing works prioritize the design of transformer attention blocks to capture global information. The local high-frequency details, which are pivotal to faithful MRI restoration, are unfortunately neglected.

METHODS

In this work, we propose a high-frequency enhanced learning scheme to effectively improve the awareness of high frequency information in current Transformer-based MRI single image super-resolution methods. Specifically, we present two entirely plug-and-play modules designed to equip Transformer-based networks with the ability to recover high-frequency details from dual spaces: 1) in the feature space, we design a high-frequency block (Hi-Fe block) paralleled with Transformer-based attention layers to extract rich high-frequency features; while 2) in the image intensity space, we tailor a high-frequency amplification module (HFA) to further refine the high-frequency details. By fully exploiting the merits of the two modules, our framework can recover abundant and diverse high-frequency information, rendering faithful MRI super-resolved results with fine details.

RESULTS

We integrated our modules with six Transformer-based models and conducted experiments across three datasets. The results indicate that our plug-and-play modules can enhance the super-resolution performance of all foundational models to varying degrees, surpassing the capabilities of existing state-of-the-art single image super-resolution networks.

CONCLUSION

Comprehensive comparison of super-resolution images and high-frequency maps from various methods, clearly demonstrating that our module possesses the capability to restore high-frequency information, showing huge potential in clinical practice for accelerated MRI reconstruction.

Inter-scanner super-resolution of 3D cine MRI using a transfer-learning network for MRgRT

Objective. Deep-learning networks for super-resolution (SR) reconstruction enhance the spatial-resolution of 3D magnetic resonance imaging (MRI) for MR-guided radiotherapy (MRgRT). However, variations between MRI scanners and patients impact the quality of SR for real-time 3D low-resolution (LR) cine MRI. In this study, we present a personalized super-resolution (psSR) network that incorporates transfer-learning to overcome the challenges in inter-scanner SR of 3D cine MRI.Approach: Development of the proposed psSR network comprises two-stages: (1) a cohort-specific SR (csSR) network using clinical patient datasets, and (2) a psSR network using transfer-learning to target datasets. The csSR network was developed by training on breath-hold and respiratory-gated high-resolution (HR) 3D MRIs and their k-space down-sampled LR MRIs from 53 thoracoabdominal patients scanned at 1.5 T. The psSR network was developed through transfer-learning to retrain the csSR network using a single breath-hold HR MRI and a corresponding 3D cine MRI from 5 healthy volunteers scanned at 0.55 T. Image quality was evaluated using the peak-signal-noise-ratio (PSNR) and the structure-similarity-index-measure (SSIM). The clinical feasibility was assessed by liver contouring on the psSR MRI using an auto-segmentation network and quantified using the dice-similarity-coefficient (DSC).Results. Mean PSNR and SSIM values of psSR MRIs were increased by 57.2% (13.8-21.7) and 94.7% (0.38-0.74) compared to cine MRIs, with the reference 0.55 T breath-hold HR MRI. In the contour evaluation, DSC was increased by 15% (0.79-0.91). Average time consumed for transfer-learning was 90 s, psSR was 4.51 ms per volume, and auto-segmentation was 210 ms, respectively.Significance. The proposed psSR reconstruction substantially increased image and segmentation quality of cine MRI in an average of 215 ms across the scanners and patients with less than 2 min of prerequisite transfer-learning. This approach would be effective in overcoming cohort- and scanner-dependency of deep-learning for MRgRT.

3D-MRI super-resolution reconstruction using multi-modality based on multi-resolution CNN

BACKGROUND AND OBJECTIVE

High-resolution (HR) MR images provide rich structural detail to assist physicians in clinical diagnosis and treatment plan. However, it is arduous to acquire HR MRI due to equipment limitations, scanning time or patient comfort. Instead, HR MRI could be obtained through a number of computer assisted post-processing methods that have proven to be effective and reliable. This paper aims to develop a convolutional neural network (CNN) based super-resolution reconstruction framework for low-resolution (LR) T2w images.

METHOD

In this paper, we propose a novel multi-modal HR MRI generation framework based on deep learning techniques. Specifically, we construct a CNN based on multi-resolution analysis to learn an end-to-end mapping between LR T2w and HR T2w, where HR T1w is fed into the network to offer detailed a priori information to help generate HR T2w. Furthermore, a low-frequency filtering module is introduced to filter out the interference from HR-T1w during high-frequency information extraction. Based on the idea of multi-resolution analysis, detailed features extracted from HR T1w and LR T2w are fused at two scales in the network and then HR T2w is reconstructed by upsampling and dense connectivity module.

RESULTS

Extensive quantitative and qualitative evaluations demonstrate that the proposed method enhances the recovered HR T2w details and outperforms other state-of-the-art methods. In addition, the experimental results also suggest that our network has a lightweight structure and favorable generalization performance.

CONCLUSION

The results show that the proposed method is capable of reconstructing HR T2w with higher accuracy. Meanwhile, the super-resolution reconstruction results on other dataset illustrate the excellent generalization ability of the method.

STADNet: Spatial-Temporal Attention-Guided Dual-Path Network for cardiac cine MRI super-resolution

Cardiac cine magnetic resonance imaging (MRI) is a commonly used clinical tool for evaluating cardiac function and morphology. However, its diagnostic accuracy may be compromised by the low spatial resolution. Current methods for cine MRI super-resolution reconstruction still have limitations. They typically rely on 3D convolutional neural networks or recurrent neural networks, which may not effectively capture long-range or non-local features due to their limited receptive fields. Optical flow estimators are also commonly used to align neighboring frames, which may cause information loss and inaccurate motion estimation. Additionally, pre-warping strategies may involve interpolation, leading to potential loss of texture details and complicated anatomical structures. To overcome these challenges, we propose a novel Spatial-Temporal Attention-Guided Dual-Path Network (STADNet) for cardiac cine MRI super-resolution. We utilize transformers to model long-range dependencies in cardiac cine MR images and design a cross-frame attention module in the location-aware spatial path, which enhances the spatial details of the current frame by using complementary information from neighboring frames. We also introduce a recurrent flow-enhanced attention module in the motion-aware temporal path that exploits the correlation between cine MRI frames and extracts the motion information of the heart. Experimental results demonstrate that STADNet outperforms SOTA approaches and has significant potential for clinical practice.

Fast MRI Reconstruction Using Deep Learning-based Compressed Sensing: A Systematic Review

Magnetic resonance imaging (MRI) has revolutionized medical imaging, providing a non-invasive and highly detailed look into the human body. However, the long acquisition times of MRI present challenges, causing patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, researchers are exploring various techniques to reduce acquisition time and improve the overall efficiency of MRI. One such technique is compressed sensing (CS), which reduces data acquisition by leveraging image sparsity in transformed spaces. In recent years, deep learning (DL) has been integrated with CS-MRI, leading to a new framework that has seen remarkable growth. DL-based CS-MRI approaches are proving to be highly effective in accelerating MR imaging without compromising image quality. This review comprehensively examines DL-based CS-MRI techniques, focusing on their role in increasing MR imaging speed. We provide a detailed analysis of each category of DL-based CS-MRI including end-to-end, unroll optimization, self-supervised, and federated learning. Our systematic review highlights significant contributions and underscores the exciting potential of DL in CS-MRI. Additionally, our systematic review efficiently summarizes key results and trends in DL-based CS-MRI including quantitative metrics, the dataset used, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based CS-MRI in the advancement of medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based CS-MRI publications and publicly available datasets - https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Dual contrast attention-guided multi-frequency fusion for multi-contrast MRI super-resolution

Objective. Multi-contrast magnetic resonance (MR) imaging super-resolution (SR) reconstruction is an effective solution for acquiring high-resolution MR images. It utilizes anatomical information from auxiliary contrast images to improve the quality of the target contrast images. However, existing studies have simply explored the relationships between auxiliary contrast and target contrast images but did not fully consider different anatomical information contained in multi-contrast images, resulting in texture details and artifacts unrelated to the target contrast images.Approach. To address these issues, we propose a dual contrast attention-guided multi-frequency fusion (DCAMF) network to reconstruct SR MR images from low-resolution MR images, which adaptively captures relevant anatomical information and processes the texture details and low-frequency information from multi-contrast images in parallel. Specifically, after the feature extraction, a feature selection module based on a dual contrast attention mechanism is proposed to focus on the texture details of the auxiliary contrast images and the low-frequency features of the target contrast images. Then, based on the characteristics of the selected features, a high- and low-frequency fusion decoder is constructed to fuse these features. In addition, a texture-enhancing module is embedded in the high-frequency fusion decoder, to highlight and refine the texture details of the auxiliary contrast and target contrast images. Finally, the high- and low-frequency fusion process is constrained by integrating a deeply-supervised mechanism into the DCAMF network.Main results. The experimental results show that the DCAMF outperforms other state-of-the-art methods. The peak signal-to-noise ratio and structural similarity of DCAMF are 39.02 dB and 0.9771 on the IXI dataset and 37.59 dB and 0.9770 on the BraTS2018 dataset, respectively. The image recovery is further validated in segmentation tasks.Significance. Our proposed SR model can enhance the quality of MR images. The results of the SR study provide a reliable basis for clinical diagnosis and subsequent image-guided treatment.

BTSC-TNAS: A neural architecture search-based transformer for brain tumor segmentation and classification

Glioblastoma (GBM), isolated brain metastasis (SBM), and primary central nervous system lymphoma (PCNSL) possess a high level of similarity in histomorphology and clinical manifestations on multimodal MRI. Such similarities have led to challenges in the clinical diagnosis of these three malignant tumors. However, many existing models solely focus on either the task of segmentation or classification, which limits the application of computer-aided diagnosis in clinical diagnosis and treatment. To solve this problem, we propose a multi-task learning transformer with neural architecture search (NAS) for brain tumor segmentation and classification (BTSC-TNAS). In the segmentation stage, we use a nested transformer U-shape network (NTU-NAS) with NAS to directly predict brain tumor masks from multi-modal MRI images. In the tumor classification stage, we use the multiscale features obtained from the encoder of NTU-NAS as the input features of the classification network (MSC-NET), which are integrated and corrected by the classification feature correction enhancement (CFCE) block to improve the accuracy of classification. The proposed BTSC-TNAS achieves an average Dice coefficient of 80.86% and 87.12% for the segmentation of tumor region and the maximum abnormal region in clinical data respectively. The model achieves a classification accuracy of 0.941. The experiments performed on the BraTS 2019 dataset show that the proposed BTSC-TNAS has excellent generalizability and can provide support for some challenging tasks in the diagnosis and treatment of brain tumors.

MGDUN: An interpretable network for multi-contrast MRI image super-resolution reconstruction

Magnetic resonance imaging (MRI) Super-Resolution (SR) aims to obtain high resolution (HR) images with more detailed information for precise diagnosis and quantitative image analysis. Deep unfolding networks outperform general MRI SR reconstruction methods by providing better performance and improved interpretability, which enhance the trustworthiness required in clinical practice. Additionally, current SR reconstruction techniques often rely on a single contrast or a simple multi-contrast fusion mechanism, ignoring the complex relationships between different contrasts. To address these issues, in this paper, we propose a Model-Guided multi-contrast interpretable Deep Unfolding Network (MGDUN) for medical image SR reconstruction, which explicitly incorporates the well-studied multi-contrast MRI observation model into an unfolding iterative network. Specifically, we manually design an objective function for MGDUN that can be iteratively computed by the half-quadratic splitting algorithm. The iterative MGDUN algorithm is unfolded into a special model-guided deep unfolding network that explicitly takes into account both the multi-contrast relationship matrix and the MRI observation matrix during the end-to-end optimization process. Extensive experimental results on the multi-contrast IXI dataset and the BraTs 2019 dataset demonstrate the superiority of our proposed model, with PSNR reaching 37.3366 and 35.9690 respectively. Our proposed MGDUN provides a promising solution for multi-contrast MR image super-resolution reconstruction. Code is available at https://github.com/yggame/MGDUN.

Physics-informed deep learning for T2-deblurred superresolution turbo spin echo MRI

PURPOSE

Deep learning superresolution (SR) is a promising approach to reduce MRI scan time without requiring custom sequences or iterative reconstruction. Previous deep learning SR approaches have generated low-resolution training images by simple k-space truncation, but this does not properly model in-plane turbo spin echo (TSE) MRI resolution degradation, which has variable T2 relaxation effects in different k-space regions. To fill this gap, we developed a T2 -deblurred deep learning SR method for the SR of 3D-TSE images.

METHODS

A SR generative adversarial network was trained using physically realistic resolution degradation (asymmetric T2 weighting of raw high-resolution k-space data). For comparison, we trained the same network structure on previous degradation models without TSE physics modeling. We tested all models for both retrospective and prospective SR with 3 × 3 acceleration factor (in the two phase-encoding directions) of genetically engineered mouse embryo model TSE-MR images.

RESULTS

The proposed method can produce high-quality 3 × 3 SR images for a typical 500-slice volume with 6-7 mouse embryos. Because 3 × 3 SR was performed, the image acquisition time can be reduced from 15 h to 1.7 h. Compared to previous SR methods without TSE modeling, the proposed method achieved the best quantitative imaging metrics for both retrospective and prospective evaluations and achieved the best imaging-quality expert scores for prospective evaluation.

CONCLUSION

The proposed T2 -deblurring method improved accuracy and image quality of deep learning-based SR of TSE MRI. This method has the potential to accelerate TSE image acquisition by a factor of up to 9.

Predicting Antiseizure Medication Treatment in Children with Rare Tuberous Sclerosis Complex-Related Epilepsy Using Deep Learning

BACKGROUND AND PURPOSE

Tuberous sclerosis complex disease is a rare, multisystem genetic disease, but appropriate drug treatment allows many pediatric patients to have positive outcomes. The purpose of this study was to predict the effectiveness of antiseizure medication treatment in children with tuberous sclerosis complex-related epilepsy.

MATERIALS AND METHODS

We conducted a retrospective study involving 300 children with tuberous sclerosis complex-related epilepsy. The study included the analysis of clinical data and T2WI and FLAIR images. The clinical data consisted of sex, age of onset, age at imaging, infantile spasms, and antiseizure medication numbers. To forecast antiseizure medication treatment, we developed a multitechnique deep learning method called WAE-Net. This method used multicontrast MR imaging and clinical data. The T2WI and FLAIR images were combined as FLAIR3 to enhance the contrast between tuberous sclerosis complex lesions and normal brain tissues. We trained a clinical data-based model using a fully connected network with the above-mentioned variables. After that, a weighted-average ensemble network built from the ResNet3D architecture was created as the final model.

RESULTS

The experiments had shown that age of onset, age at imaging, infantile spasms, and antiseizure medication numbers were significantly different between the 2 drug-treatment outcomes (P < .05). The hybrid technique of FLAIR3 could accurately localize tuberous sclerosis complex lesions, and the proposed method achieved the best performance (area under the curve = 0.908 and accuracy of 0.847) in the testing cohort among the compared methods.

CONCLUSIONS

The proposed method could predict antiseizure medication treatment of children with rare tuberous sclerosis complex-related epilepsy and could be a strong baseline for future studies.

Global attention-enabled texture enhancement network for MR image reconstruction

PURPOSE

Although recent convolutional neural network (CNN) methodologies have shown promising results in fast MR imaging, there is still a desire to explore how they can be used to learn the frequency characteristics of multicontrast images and reconstruct texture details.

METHODS

A global attention-enabled texture enhancement network (GATE-Net) with a frequency-dependent feature extraction module (FDFEM) and convolution-based global attention module (GAM) is proposed to address the highly under-sampling MR image reconstruction problem. First, FDFEM enables GATE-Net to effectively extract high-frequency features from shareable information of multicontrast images to improve the texture details of reconstructed images. Second, GAM with less computation complexity has the receptive field of the entire image, which can fully explore useful shareable information of multi-contrast images and suppress less beneficial shareable information.

RESULTS

The ablation studies are conducted to evaluate the effectiveness of the proposed FDFEM and GAM. Experimental results under various acceleration rates and datasets consistently demonstrate the superiority of GATE-Net, in terms of peak signal-to-noise ratio, structural similarity and normalized mean square error.

CONCLUSION

A global attention-enabled texture enhancement network is proposed. it can be applied to multicontrast MR image reconstruction tasks with different acceleration rates and datasets and achieves superior performance in comparison with state-of-the-art methods.

Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain

Magnetic resonance (MR) imaging plays an important role in clinical and brain exploration. However, limited by factors such as imaging hardware, scanning time, and cost, it is challenging to acquire high-resolution MR images clinically. In this article, fine perceptive generative adversarial networks (FP-GANs) are proposed to produce super-resolution (SR) MR images from the low-resolution counterparts. By adopting the divide-and-conquer scheme, FP-GANs are designed to deal with the low-frequency (LF) and high-frequency (HF) components of MR images separately and parallelly. Specifically, FP-GANs first decompose an MR image into LF global approximation and HF anatomical texture subbands in the wavelet domain. Then, each subband generative adversarial network (GAN) simultaneously concentrates on super-resolving the corresponding subband image. In generator, multiple residual-in-residual dense blocks are introduced for better feature extraction. In addition, the texture-enhancing module is designed to trade off the weight between global topology and detailed textures. Finally, the reconstruction of the whole image is considered by integrating inverse discrete wavelet transformation in FP-GANs. Comprehensive experiments on the MultiRes_7T and ADNI datasets demonstrate that the proposed model achieves finer structure recovery and outperforms the competing methods quantitatively and qualitatively. Moreover, FP-GANs further show the value by applying the SR results in classification tasks.

Multiscale feature fusion network for 3D head MRI image registration

BACKGROUND

Image registration technology has become an important medical image preprocessing step with the wide application of computer-aided diagnosis technology in various medical image analysis tasks.

PURPOSE

We propose a multiscale feature fusion registration based on deep learning to achieve the accurate registration and fusion of head magnetic resonance imaging (MRI) and solve the problem that general registration methods cannot handle the complex spatial information and position information of head MRI.

METHODS

Our proposed multiscale feature fusion registration network consists of three sequentially trained modules. The first is an affine registration module that implements affine transformation; the second is to realize non-rigid transformation, a deformable registration module composed of top-down and bottom-up feature fusion subnetworks in parallel; and the third is a deformable registration module that also realizes non-rigid transformation and is composed of two feature fusion subnetworks in series. The network decomposes the deformation field of large displacement into multiple deformation fields of small displacement by multiscale registration and registration, which reduces the difficulty of registration. Moreover, multiscale information in head MRI is learned in a targeted manner, which improves the registration accuracy, by connecting the two feature fusion subnetworks.

RESULTS

We used 29 3D head MRIs for training and seven volumes for testing and calculated the values of the registration evaluation metrics for the new algorithm to register anterior and posterior lateral pterygoid muscles. The Dice similarity coefficient was 0.745 ± 0.021, the Hausdorff distance was 3.441 ± 0.935 mm, the Average surface distance was 0.738 ± 0.098 mm, and the Standard deviation of the Jacobian matrix was 0.425 ± 0.043. Our new algorithm achieved a higher registration accuracy compared with state-of-the-art registration methods.

CONCLUSIONS

Our proposed multiscale feature fusion registration network can realize end-to-end deformable registration of 3D head MRI, which can effectively cope with the characteristics of large deformation displacement and the rich details of head images and provide reliable technical support for the diagnosis and analysis of head diseases.

MHAN: Multi-Stage Hybrid Attention Network for MRI reconstruction and super-resolution

High-quality magnetic resonance imaging (MRI) affords clear body tissue structure for reliable diagnosing. However, there is a principal problem of the trade-off between acquisition speed and image quality. Image reconstruction and super-resolution are crucial techniques to solve these problems. In the main field of MR image restoration, most researchers mainly focus on only one of these aspects, namely reconstruction or super-resolution. In this paper, we propose an efficient model called Multi-Stage Hybrid Attention Network (MHAN) that performs the multi-task of recovering high-resolution (HR) MR images from low-resolution (LR) under-sampled measurements. Our model is highlighted by three major modules: (i) an Amplified Spatial Attention Block (ASAB) capable of enhancing the differences in spatial information, (ii) a Self-Attention Block with a Data-Consistency Layer (DC-SAB), which can improve the accuracy of the extracted feature information, (iii) an Adaptive Local Residual Attention Block (ALRAB) that focuses on both spatial and channel information. MHAN employs an encoder-decoder architecture to deeply extract contextual information and a pipeline to provide spatial accuracy. Compared with the recent multi-task model T2Net, our MHAN improves by 2.759 dB in PSNR and 0.026 in SSIM with scaling factor ×2 and acceleration factor 4× on T2 modality.

High-Resolution 3D MRI With Deep Generative Networks via Novel Slice-Profile Transformation Super-Resolution

High-resolution magnetic resonance imaging (MRI) sequences, such as 3D turbo or fast spin-echo (TSE/FSE) imaging, are clinically desirable but suffer from long scanning time-related blurring when reformatted into preferred orientations. Instead, multi-slice two-dimensional (2D) TSE imaging is commonly used because of its high in-plane resolution but is limited clinically by poor through-plane resolution due to elongated voxels and the inability to generate multi-planar reformations due to staircase artifacts. Therefore, multiple 2D TSE scans are acquired in various orthogonal imaging planes, increasing the overall MRI scan time. In this study, we propose a novel slice-profile transformation super-resolution (SPTSR) framework with deep generative learning for through-plane super-resolution (SR) of multi-slice 2D TSE imaging. The deep generative networks were trained by synthesized low-resolution training input via slice-profile downsampling (SP-DS), and the trained networks inferred on the slice profile convolved (SP-conv) testing input for 5.5x through-plane SR. The network output was further slice-profile deconvolved (SP-deconv) to achieve an isotropic super-resolution. Compared to SMORE SR method and the networks trained by conventional downsampling, our SPTSR framework demonstrated the best overall image quality from 50 testing cases, evaluated by two abdominal radiologists. The quantitative analysis cross-validated the expert reader study results. 3D simulation experiments confirmed the quantitative improvement of the proposed SPTSR and the effectiveness of the SP-deconv step, compared to 3D ground-truths. Ablation studies were conducted on the individual contributions of SP-DS and SP-conv, networks structure, training dataset size, and different slice profiles.

Recognizing Pediatric Tuberous Sclerosis Complex Based on Multi-Contrast MRI and Deep Weighted Fusion Network

Multi-contrast magnetic resonance imaging (MRI) is wildly applied to identify tuberous sclerosis complex (TSC) children in a clinic. In this work, a deep convolutional neural network with multi-contrast MRI is proposed to diagnose pediatric TSC. Firstly, by combining T2W and FLAIR images, a new synthesis modality named FLAIR3 was created to enhance the contrast between TSC lesions and normal brain tissues. After that, a deep weighted fusion network (DWF-net) using a late fusion strategy is proposed to diagnose TSC children. In experiments, a total of 680 children were enrolled, including 331 healthy children and 349 TSC children. The experimental results indicate that FLAIR3 successfully enhances the visibility of TSC lesions and improves the classification performance. Additionally, the proposed DWF-net delivers a superior classification performance compared to previous methods, achieving an AUC of 0.998 and an accuracy of 0.985. The proposed method has the potential to be a reliable computer-aided diagnostic tool for assisting radiologists in diagnosing TSC children.

Accelerating image reconstruction for multi-contrast MRI based on Y-Net3

BACKGROUND

As one of the significant preoperative imaging modalities in medical diagnosis, Magnetic resonance imaging (MRI) takes a long scanning time due to its special imaging principle.

OBJECTIVE

We propose an innovative MRI reconstruction strategy and data consistency method based on deep learning to reconstruct high-quality brain MRIs from down-sampled data and accelerate the MR imaging process.

METHODS

Sixteen healthy subjects undergoing T1-weighted spin-echo (SE) and T2-weighted fast spin-echo (FSE) sequences by a 1.5T MRI scanner were recruited. A Y-Net3+ network was used to facilitate the high-quality MRI reconstruction through context information. In addition, the existing data consistency fidelity method was improved. The difference between the reconstructed K-space and the original K-space was shorten by the linear regression algorithm. Therefore, the redundant artifacts derived from under-sampling were avoided. The Structural Similarity (SSIM) and Peak Signal to Noise Ratio (PSNR) were applied to quantitatively evaluate image reconstruction performance of different down-sampling patterns.

RESULTS

Compared with the classical Y-Net, Y-Net3+ network improved SSIM and PSNR of MRI images from 0.9164±0.0178 and 33.2216±3.2919 to 0.9387±0.0363 and 35.1785±3.3105, respectively, under compressed sensing reconstruction with acceleration factor of 4. The improved network increases signal-to-noise ratio and adds more image texture information in the reconstructed images. Furthermore, in the process of data consistency, linear regression analysis was used to reduce the difference between the reconstructed K-space and the original K-space, so that the SSIM and PSNR were increased to 0.9808±0.0081 and 40.9254±1.1911, respectively.

CONCLUSIONS

The improved Y-Net combined with data consistency fidelity method elucidates its potential in reconstructing high-quality T2-weighted images from the down-sampled data by fully exploring the T1-weighted information. With the advantage of avoiding down-sampled artifacts, the improved network exhibits remarkable clinical promise for fast MRI applications.

ESPRESO: An algorithm to estimate the slice profile of a single magnetic resonance image

To reduce scan time, magnetic resonance (MR) images are often acquired using 2D multi-slice protocols with thick slices that may also have gaps between them. The resulting image volumes have lower resolution in the through-plane direction than in the in-plane direction, and the through-plane resolution is in part characterized by the protocol's slice profile which acts as a through-plane point spread function (PSF). Although super-resolution (SR) has been shown to improve the visualization and down-stream processing of 2D multi-slice MR acquisitions, previous algorithms are usually unaware of the true slice profile, which may lead to sub-optimal SR performance. In this work, we present an algorithm to estimate the slice profile of a 2D multi-slice acquisition given only its own image volume without any external training data. We assume that an anatomical image is isotropic in the sense that, after accounting for a correctly estimated slice profile, the image patches along different orientations have the same probability distribution. Our proposed algorithm uses a modified generative adversarial network (GAN) where the generator network estimates the slice profile to reduce the resolution of the in-plane direction, and the discriminator network determines whether a direction is generated or real low resolution. The proposed algorithm, ESPRESO, which stands for "estimating the slice profile for resolution enhancement of a single image only", was tested with a state-of-the-art internally supervised SR algorithm. Specifically, ESPRESO is used to create training data for this SR algorithm, and results show improvements when ESPRESO is used over commonly-used PSFs.

Region-focused multi-view transformer-based generative adversarial network for cardiac cine MRI reconstruction

Cardiac cine magnetic resonance imaging (MRI) reconstruction is challenging due to spatial and temporal resolution trade-offs. Temporal correlation in cardiac cine MRI is informative and vital for understanding cardiac dynamic motion. Exploiting the temporal correlations in cine reconstruction is crucial to resolve aliasing artifacts and maintaining the cardiac motion patterns. However, existing methods have the following shortcomings: (1) they simultaneously compute pairwise correlations along spatial and temporal dimensions to establish dependencies, ignoring that learning spatial contextual information first will benefit the temporal modeling. (2) most studies neglect to focus on reconstructing the local cardiac regions, resulting in insufficient reconstruction accuracy due to a relatively large field of view. To address these problems, we propose a region-focused multi-view transformer-based generative adversarial network for cardiac cine MRI reconstruction. The proposed transformer divides consecutive cardiac frames into multiple views for cross-view feature extraction, establishing long-distance dependencies among features and effectively learning the spatio-temporal information. We further design a cross-view attention for spatio-temporal information fusion, ensuring the interaction of different spatio-temporal information in each view and capturing more temporal correlations of the cardiac motion. In addition, we introduce a cardiac region detection loss for improving the reconstruction quality of the cardiac region. Experimental results demonstrated that our method outperforms state-of-the-art methods. Especially with an acceleration factor as high as 10×, our method can reconstruct images with better accuracy and perceptual quality.

SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry

Every year, millions of brain magnetic resonance imaging (MRI) scans are acquired in hospitals across the world. These have the potential to revolutionize our understanding of many neurological diseases, but their morphometric analysis has not yet been possible due to their anisotropic resolution. We present an artificial intelligence technique, "SynthSR," that takes clinical brain MRI scans with any MR contrast (T1, T2, etc.), orientation (axial/coronal/sagittal), and resolution and turns them into high-resolution T1 scans that are usable by virtually all existing human neuroimaging tools. We present results on segmentation, registration, and atlasing of >10,000 scans of controls and patients with brain tumors, strokes, and Alzheimer's disease. SynthSR yields morphometric results that are very highly correlated with what one would have obtained with high-resolution T1 scans. SynthSR allows sample sizes that have the potential to overcome the power limitations of prospective research studies and shed new light on the healthy and diseased human brain.

An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation

High Resolution (HR) medical images provide rich anatomical structure details to facilitate early and accurate diagnosis. In magnetic resonance imaging (MRI), restricted by hardware capacity, scan time, and patient cooperation ability, isotropic 3-dimensional (3D) HR image acquisition typically requests long scan time and, results in small spatial coverage and low signal-to-noise ratio (SNR). Recent studies showed that, with deep convolutional neural networks, isotropic HR MR images could be recovered from low-resolution (LR) input via single image super-resolution (SISR) algorithms. However, most existing SISR methods tend to approach scale-specific projection between LR and HR images, thus these methods can only deal with fixed up-sampling rates. In this paper, we propose ArSSR, an Arbitrary Scale Super-Resolution approach for recovering 3D HR MR images. In the ArSSR model, the LR image and the HR image are represented using the same implicit neural voxel function with different sampling rates. Due to the continuity of the learned implicit function, a single ArSSR model is able to achieve arbitrary and infinite up-sampling rate reconstructions of HR images from any input LR image. Then the SR task is converted to approach the implicit voxel function via deep neural networks from a set of paired HR and LR training examples. The ArSSR model consists of an encoder network and a decoder network. Specifically, the convolutional encoder network is to extract feature maps from the LR input images and the fully-connected decoder network is to approximate the implicit voxel function. Experimental results on three datasets show that the ArSSR model can achieve state-of-the-art SR performance for 3D HR MR image reconstruction while using a single trained model to achieve arbitrary up-sampling scales.

Simultaneous superresolution reconstruction and distortion correction for single-shot EPI DWI using deep learning

PURPOSE

Single-shot (SS) EPI is widely used for clinical DWI. This study aims to develop an end-to-end deep learning-based method with a novel loss function in an improved network structure to simultaneously increase the resolution and correct distortions for SS-EPI DWI.

THEORY AND METHODS

Point-spread-function (PSF)-encoded EPI can provide high-resolution, distortion-free DWI images. A distorted image from SS-EPI can be described as the convolution between a PSF function with a distortion-free image. The deconvolution process to recover the distortion-free image can be achieved with a convolution neural network, which also learns the mapping function between low-resolution SS-EPI and high-resolution reference PSF-EPI to achieve superresolution. To suppress the oversmoothing effect, we proposed a modified generative adversarial network structure, in which a dense net with gradient map guidance and a multilevel fusion block was used as the generator. A fractional anisotropy loss was proposed to utilize the diffusion anisotropy information among diffusion directions. In vivo brain DWI data were used to test the proposed method.

RESULTS

The results show that distortion-corrected high-resolution DWI images with restored structural details can be obtained from low-resolution SS-EPI images by taking advantage of the high-resolution anatomical images. Additionally, the proposed network can improve the quantitative accuracy of diffusion metrics compared with previously reported networks.

CONCLUSION

Using high-resolution, distortion-free EPI-DWI images as references, a deep learning-based method to simultaneously increase the perceived resolution and correct distortions for low-resolution SS-EPI was proposed. The results show that DWI image quality and diffusion metrics can be improved.

Multi-modal brain MRI images enhancement based on framelet and local weights super-resolution

Magnetic resonance (MR) image enhancement technology can reconstruct high-resolution image from a low-resolution image, which is of great significance for clinical application and scientific research. T1 weighting and T2 weighting are the two common magnetic resonance imaging modes, each of which has its own advantages, but the imaging time of T2 is much longer than that of T1. Related studies have shown that they have very similar anatomical structures in brain images, which can be utilized to enhance the resolution of low-resolution T2 images by using the edge information of high-resolution T1 images that can be rapidly imaged, so as to shorten the imaging time needed for T2 images. In order to overcome the inflexibility of traditional methods using fixed weights for interpolation and the inaccuracy of using gradient threshold to determine edge regions, we propose a new model based on previous studies on multi-contrast MR image enhancement. Our model uses framelet decomposition to finely separate the edge structure of the T2 brain image, and uses the local regression weights calculated from T1 image to construct a global interpolation matrix, so that our model can not only guide the edge reconstruction more accurately where the weights are shared, but also carry out collaborative global optimization for the remaining pixels and their interpolated weights. Experimental results on a set of simulated MR data and two sets of real MR images show that the enhanced images obtained by the proposed method are superior to the compared methods in terms of visual sharpness or qualitative indicators.

Accurate and lightweight MRI super-resolution via multi-scale bidirectional fusion attention network

High-resolution magnetic resonance (MR) imaging has attracted much attention due to its contribution to clinical diagnoses and treatment. However, because of the interference of noise and the limitation of imaging equipment, it is expensive to generate a satisfactory image. Super-resolution (SR) is a technique that enhances an imaging system's resolution, which is effective and cost-efficient for MR imaging. In recent years, deep learning-based SR methods have made remarkable progress on natural images but not on medical images. Most existing medical images SR algorithms focus on the spatial information of a single image but ignore the temporal correlation between medical images sequence. We proposed two novel architectures for single medical image and sequential medical images, respectively. The multi-scale back-projection network (MSBPN) is constructed of several different scale back-projection units which consist of iterative up- and down-sampling layers. The multi-scale machine extracts different scale spatial information and strengthens the information fusion for a single image. Based on MSBPN, we proposed an accurate and lightweight Multi-Scale Bidirectional Fusion Attention Network(MSBFAN) that combines temporal information iteratively. That supplementary temporal information is extracted from the adjacent image sequence of the target image. The MSBFAN can effectively learn both the spatio-temporal dependencies and the iterative refinement process with only a lightweight number of parameters. Experimental results demonstrate that our MSBPN and MSBFAN are outperforming current SR methods in terms of reconstruction accuracy and parameter quantity of the model.

Semi-Supervised Learning of MRI Synthesis Without Fully-Sampled Ground Truths

Learning-based translation between MRI contrasts involves supervised deep models trained using high-quality source- and target-contrast images derived from fully-sampled acquisitions, which might be difficult to collect under limitations on scan costs or time. To facilitate curation of training sets, here we introduce the first semi-supervised model for MRI contrast translation (ssGAN) that can be trained directly using undersampled k-space data. To enable semi-supervised learning on undersampled data, ssGAN introduces novel multi-coil losses in image, k-space, and adversarial domains. The multi-coil losses are selectively enforced on acquired k-space samples unlike traditional losses in single-coil synthesis models. Comprehensive experiments on retrospectively undersampled multi-contrast brain MRI datasets are provided. Our results demonstrate that ssGAN yields on par performance to a supervised model, while outperforming single-coil models trained on coil-combined magnitude images. It also outperforms cascaded reconstruction-synthesis models where a supervised synthesis model is trained following self-supervised reconstruction of undersampled data. Thus, ssGAN holds great promise to improve the feasibility of learning-based multi-contrast MRI synthesis.

Multimodal MRI Reconstruction Assisted With Spatial Alignment Network

In clinical practice, multi-modal magnetic resonance imaging (MRI) with different contrasts is usually acquired in a single study to assess different properties of the same region of interest in the human body. The whole acquisition process can be accelerated by having one or more modalities under-sampled in the k -space. Recent research has shown that, considering the redundancy between different modalities, a target MRI modality under-sampled in the k -space can be more efficiently reconstructed with a fully-sampled reference MRI modality. However, we find that the performance of the aforementioned multi-modal reconstruction can be negatively affected by subtle spatial misalignment between different modalities, which is actually common in clinical practice. In this paper, we improve the quality of multi-modal reconstruction by compensating for such spatial misalignment with a spatial alignment network. First, our spatial alignment network estimates the displacement between the fully-sampled reference and the under-sampled target images, and warps the reference image accordingly. Then, the aligned fully-sampled reference image joins the multi-modal reconstruction of the under-sampled target image. Also, considering the contrast difference between the target and reference images, we have designed a cross-modality-synthesis-based registration loss in combination with the reconstruction loss, to jointly train the spatial alignment network and the reconstruction network. The experiments on both clinical MRI and multi-coil k -space raw data demonstrate the superiority and robustness of the multi-modal MRI reconstruction empowered with our spatial alignment network. Our code is publicly available at https://github.com/woxuankai/SpatialAlignmentNetwork.

SRflow: Deep learning based super-resolution of 4D-flow MRI data

Exploiting 4D-flow magnetic resonance imaging (MRI) data to quantify hemodynamics requires an adequate spatio-temporal vector field resolution at a low noise level. To address this challenge, we provide a learned solution to super-resolve in vivo 4D-flow MRI data at a post-processing level. We propose a deep convolutional neural network (CNN) that learns the inter-scale relationship of the velocity vector map and leverages an efficient residual learning scheme to make it computationally feasible. A novel, direction-sensitive, and robust loss function is crucial to learning vector-field data. We present a detailed comparative study between the proposed super-resolution and the conventional cubic B-spline based vector-field super-resolution. Our method improves the peak-velocity to noise ratio of the flow field by 10 and 30% for in vivo cardiovascular and cerebrovascular data, respectively, for 4 × super-resolution over the state-of-the-art cubic B-spline. Significantly, our method offers 10x faster inference over the cubic B-spline. The proposed approach for super-resolution of 4D-flow data would potentially improve the subsequent calculation of hemodynamic quantities.

Elimination of stripe artifacts in light sheet fluorescence microscopy using an attention-based residual neural network

Stripe artifacts can deteriorate the quality of light sheet fluorescence microscopy (LSFM) images. Owing to the inhomogeneous, high-absorption, or scattering objects located in the excitation light path, stripe artifacts are generated in LSFM images in various directions and types, such as horizontal, anisotropic, or multidirectional anisotropic. These artifacts severely degrade the quality of LSFM images. To address this issue, we proposed a new deep-learning-based approach for the elimination of stripe artifacts. This method utilizes an encoder-decoder structure of UNet integrated with residual blocks and attention modules between successive convolutional layers. Our attention module was implemented in the residual blocks to learn useful features and suppress the residual features. The proposed network was trained and validated by generating three different degradation datasets with different types of stripe artifacts in LSFM images. Our method can effectively remove different stripes in generated and actual LSFM images distorted by stripe artifacts. Besides, quantitative analysis and extensive comparison results demonstrated that our method performs the best compared with classical image-based processing algorithms and other powerful deep-learning-based destriping methods for all three generated datasets. Thus, our method has tremendous application prospects to LSFM, and its use can be easily extended to images reconstructed by other modalities affected by the presence of stripe artifacts.