Deep Generative Adversarial Neural Networks for Compressive Sensing MRI

Correlation of objective image quality metrics with radiologists' diagnostic confidence depends on the clinical task performed

Purpose

Objective image quality metrics (IQMs) are widely used as outcome measures to assess acquisition and reconstruction strategies for diagnostic images. For nonpathological magnetic resonance (MR) images, these IQMs correlate to varying degrees with expert radiologists' confidence scores of overall perceived diagnostic image quality. However, it is unclear whether IQMs also correlate with task-specific diagnostic image quality or expert radiologists' confidence in performing a specific diagnostic task, which calls into question their use as surrogates for radiologist opinion.

Approach

0.5 T MR images from 16 stroke patients and two healthy volunteers were retrospectively undersampled ( R = 1 to 7 × ) and reconstructed via compressed sensing. Three neuroradiologists reported the presence/absence of acute ischemic stroke (AIS) and assigned a Fazekas score describing the extent of chronic ischemic lesion burden. Neuroradiologists ranked their confidence in performing each task using a 1 to 5 Likert scale. Confidence scores were correlated with noise quality measure, the visual information fidelity criterion, the feature similarity index, root mean square error, and structural similarity (SSIM) via nonlinear regression modeling.

Results

Although acceleration alters image quality, neuroradiologists remain able to report pathology. All of the IQMs tested correlated to some degree with diagnostic confidence for assessing chronic ischemic lesion burden, but none correlated with diagnostic confidence in diagnosing the presence/absence of AIS due to consistent radiologist performance regardless of image degradation.

Conclusions

Accelerated images were helpful for understanding the ability of IQMs to assess task-specific diagnostic image quality in the context of chronic ischemic lesion burden, although not in the case of AIS diagnosis. These findings suggest that commonly used IQMs, such as the SSIM index, do not necessarily indicate an image's utility when performing certain diagnostic tasks.

Using deep feature distances for evaluating the perceptual quality of MR image reconstructions

PURPOSE

Commonly used MR image quality (IQ) metrics have poor concordance with radiologist-perceived diagnostic IQ. Here, we develop and explore deep feature distances (DFDs)-distances computed in a lower-dimensional feature space encoded by a convolutional neural network (CNN)-as improved perceptual IQ metrics for MR image reconstruction. We further explore the impact of distribution shifts between images in the DFD CNN encoder training data and the IQ metric evaluation.

METHODS

We compare commonly used IQ metrics (PSNR and SSIM) to two "out-of-domain" DFDs with encoders trained on natural images, an "in-domain" DFD trained on MR images alone, and two domain-adjacent DFDs trained on large medical imaging datasets. We additionally compare these with several state-of-the-art but less commonly reported IQ metrics, visual information fidelity (VIF), noise quality metric (NQM), and the high-frequency error norm (HFEN). IQ metric performance is assessed via correlations with five expert radiologist reader scores of perceived diagnostic IQ of various accelerated MR image reconstructions. We characterize the behavior of these IQ metrics under common distortions expected during image acquisition, including their sensitivity to acquisition noise.

RESULTS

All DFDs and HFEN correlate more strongly with radiologist-perceived diagnostic IQ than SSIM, PSNR, and other state-of-the-art metrics, with correlations being comparable to radiologist inter-reader variability. Surprisingly, out-of-domain DFDs perform comparably to in-domain and domain-adjacent DFDs.

CONCLUSION

A suite of IQ metrics, including DFDs and HFEN, should be used alongside commonly-reported IQ metrics for a more holistic evaluation of MR image reconstruction perceptual quality. We also observe that general vision encoders are capable of assessing visual IQ even for MR images.

FDuDoCLNet: Fully dual-domain contrastive learning network for parallel MRI reconstruction

Magnetic resonance imaging (MRI) is a non-invasive medical imaging technique that is widely used for high-resolution imaging of soft tissues and organs. However, the slow speed of MRI imaging, especially in high-resolution or dynamic scans, makes MRI reconstruction an important research topic. Currently, MRI reconstruction methods based on deep learning (DL) have garnered significant attention, and they improve the reconstruction quality by learning complex image features. However, DL-based MR image reconstruction methods exhibit certain limitations. First, the existing reconstruction networks seldom account for the diverse frequency features in the wavelet domain. Second, existing dual-domain reconstruction methods may pay too much attention to the features of a single domain (such as the global information in the image domain or the local details in the wavelet domain), resulting in the loss of either critical global structures or fine details in certain regions of the reconstructed image. In this work, inspired by the lifting scheme in wavelet theory, we propose a novel Fully Dual-Domain Contrastive Learning Network (FDuDoCLNet) based on variational networks (VarNet) for accelerating PI in both the image and wavelet domains. It is composed of several cascaded dual-domain regularization units and data consistency (DC) layers, in which a novel dual-domain contrastive loss is introduced to optimize the reconstruction performance effectively. The proposed FDuDoCLNet was evaluated on the publicly available fastMRI multi-coil knee dataset under a 6× acceleration factor, achieving a PSNR of 34.439 dB and a SSIM of 0.895.

AI-driven 3D bioprinting for regenerative medicine: From bench to bedside

In recent decades, 3D bioprinting has garnered significant research attention due to its ability to manipulate biomaterials and cells to create complex structures precisely. However, due to technological and cost constraints, the clinical translation of 3D bioprinted products (BPPs) from bench to bedside has been hindered by challenges in terms of personalization of design and scaling up of production. Recently, the emerging applications of artificial intelligence (AI) technologies have significantly improved the performance of 3D bioprinting. However, the existing literature remains deficient in a methodological exploration of AI technologies' potential to overcome these challenges in advancing 3D bioprinting toward clinical application. This paper aims to present a systematic methodology for AI-driven 3D bioprinting, structured within the theoretical framework of Quality by Design (QbD). This paper commences by introducing the QbD theory into 3D bioprinting, followed by summarizing the technology roadmap of AI integration in 3D bioprinting, including multi-scale and multi-modal sensing, data-driven design, and in-line process control. This paper further describes specific AI applications in 3D bioprinting's key elements, including bioink formulation, model structure, printing process, and function regulation. Finally, the paper discusses current prospects and challenges associated with AI technologies to further advance the clinical translation of 3D bioprinting.

DONet: Dual-Octave Network for Fast MR Image Reconstruction

Magnetic resonance (MR) image acquisition is an inherently prolonged process, whose acceleration has long been the subject of research. This is commonly achieved by obtaining multiple undersampled images, simultaneously, through parallel imaging. In this article, we propose the dual-octave network (DONet), which is capable of learning multiscale spatial-frequency features from both the real and imaginary components of MR data, for parallel fast MR image reconstruction. More specifically, our DONet consists of a series of dual-octave convolutions (Dual-OctConvs), which are connected in a dense manner for better reuse of features. In each Dual-OctConv, the input feature maps and convolutional kernels are first split into two components (i.e., real and imaginary) and then divided into four groups according to their spatial frequencies. Then, our Dual-OctConv conducts intragroup information updating and intergroup information exchange to aggregate the contextual information across different groups. Our framework provides three appealing benefits: 1) it encourages information interaction and fusion between the real and imaginary components at various spatial frequencies to achieve richer representational capacity; 2) the dense connections between the real and imaginary groups in each Dual-OctConv make the propagation of features more efficient by feature reuse; and 3) DONet enlarges the receptive field by learning multiple spatial-frequency features of both the real and imaginary components. Extensive experiments on two popular datasets (i.e., clinical knee and fastMRI), under different undersampling patterns and acceleration factors, demonstrate the superiority of our model in accelerated parallel MR image reconstruction.

Advancing MRI Reconstruction: A Systematic Review of Deep Learning and Compressed Sensing Integration

Magnetic resonance imaging (MRI) is a non-invasive imaging modality and provides comprehensive anatomical and functional insights into the human body. However, its long acquisition times can lead to patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, strategies such as parallel imaging have been applied, which utilize multiple receiver coils to speed up the data acquisition process. Additionally, compressed sensing (CS) is a method that facilitates image reconstruction from sparse data, significantly reducing image acquisition time by minimizing the amount of data collection needed. Recently, deep learning (DL) has emerged as a powerful tool for improving MRI reconstruction. It has been integrated with parallel imaging and CS principles to achieve faster and more accurate MRI reconstructions. This review comprehensively examines DL-based techniques for MRI reconstruction. We categorize and discuss various DL-based methods, including end-to-end approaches, unrolled optimization, and federated learning, highlighting their potential benefits. Our systematic review highlights significant contributions and underscores the potential of DL in MRI reconstruction. Additionally, we summarize key results and trends in DL-based MRI reconstruction, including quantitative metrics, the dataset, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based MRI reconstruction in advancing medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based MRI reconstruction publications and public datasets-https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Physics-guided self-supervised learning: Demonstration for generalized RF pulse design

PURPOSE

To introduce a new method for generalized RF pulse design using physics-guided self-supervised learning (GPS), which uses the Bloch equations as the guiding physics model.

THEORY AND METHODS

The GPS framework consists of a neural network module and a physics module, where the physics module is a Bloch simulator for MRI applications. For RF pulse design, the neural network module maps an input target profile to an RF pulse, which is subsequently loaded into the physics module. Through the supervision of the physics module, the neural network module designs an RF pulse corresponding to the target profile. GPS was applied to design 1D selective,B 1 $$ {B}_1 $$ -insensitive, saturation, and multidimensional RF pulses, each conventionally requiring dedicated design algorithms. We further demonstrate our method's flexibility and versatility by compensating for experimental and scanner imperfections through online adaptation.

RESULTS

Both simulations and experiments show that GPS can design a variety of RF pulses with corresponding profiles that agree well with the target input. Despite these verifications, GPS-designed pulses have unique differences compared to conventional designs, such as achievingB 1 $$ {B}_1 $$ -insensitivity using different mechanisms and using non-sampled regions of the conventional design to lower its peak power. Experiments, both ex vivo and in vivo, further verify that it can also be used for online adaptation to correct system imperfections, such asB 0 $$ {B}_0 $$ /B 1 + $$ {B}_1^{+} $$ inhomogeneity.

CONCLUSION

This work demonstrates the generalizability, versatility, and flexibility of the GPS method for designing RF pulses and showcases its utility in several applications.

Generalizable Reconstruction for Accelerating MR Imaging via Federated Learning With Neural Architecture Search

Heterogeneous data captured by different scanning devices and imaging protocols can affect the generalization performance of the deep learning magnetic resonance (MR) reconstruction model. While a centralized training model is effective in mitigating this problem, it raises concerns about privacy protection. Federated learning is a distributed training paradigm that can utilize multi-institutional data for collaborative training without sharing data. However, existing federated learning MR image reconstruction methods rely on models designed manually by experts, which are complex and computationally expensive, suffering from performance degradation when facing heterogeneous data distributions. In addition, these methods give inadequate consideration to fairness issues, namely ensuring that the model's training does not introduce bias towards any specific dataset's distribution. To this end, this paper proposes a generalizable federated neural architecture search framework for accelerating MR imaging (GAutoMRI). Specifically, automatic neural architecture search is investigated for effective and efficient neural network representation learning of MR images from different centers. Furthermore, we design a fairness adjustment approach that can enable the model to learn features fairly from inconsistent distributions of different devices and centers, and thus facilitate the model to generalize well to the unseen center. Extensive experiments show that our proposed GAutoMRI has better performances and generalization ability compared with seven state-of-the-art federated learning methods. Moreover, the GAutoMRI model is significantly more lightweight, making it an efficient choice for MR image reconstruction tasks. The code will be made available at https://github.com/ternencewu123/GAutoMRI.

Fourier Convolution Block with global receptive field for MRI reconstruction

Reconstructing images from under-sampled Magnetic Resonance Imaging (MRI) signals significantly reduces scan time and improves clinical practice. However, Convolutional Neural Network (CNN)-based methods, while demonstrating great performance in MRI reconstruction, may face limitations due to their restricted receptive field (RF), hindering the capture of global features. This is particularly crucial for reconstruction, as aliasing artifacts are distributed globally. Recent advancements in Vision Transformers have further emphasized the significance of a large RF. In this study, we proposed a novel global Fourier Convolution Block (FCB) with whole image RF and low computational complexity by transforming the regular spatial domain convolutions into frequency domain. Visualizations of the effective RF and trained kernels demonstrated that FCB improves the RF of reconstruction models in practice. The proposed FCB was evaluated on four popular CNN architectures using brain and knee MRI datasets. Models with FCB achieved superior PSNR and SSIM than baseline models and exhibited more details and texture recovery. The code is publicly available at https://github.com/Haozhoong/FCB.

An improved low-rank plus sparse unrolling network method for dynamic magnetic resonance imaging

BACKGROUND

Recent advances in deep learning have sparked new research interests in dynamic magnetic resonance imaging (MRI) reconstruction. However, existing deep learning-based approaches suffer from insufficient reconstruction efficiency and accuracy due to the lack of time correlation modeling during the reconstruction procedure.

PURPOSE

Inappropriate tensor processing steps and deep learning models may lead to not only a lack of modeling in the time dimension but also an increase in the overall size of the network. Therefore, this study aims to find suitable tensor processing methods and deep learning models to achieve better reconstruction results and a smaller network size.

METHODS

We propose a novel unrolling network method that enhances the reconstruction quality and reduces the parameter redundancy by introducing time correlation modeling into MRI reconstruction with low-rank core matrix and convolutional long short-term memory (ConvLSTM) unit.

RESULTS

We conduct extensive experiments on AMRG Cardiac MRI dataset to evaluate our proposed approach. The results demonstrate that compared to other state-of-the-art approaches, our approach achieves higher peak signal-to-noise ratios and structural similarity indices at different accelerator factors with significantly fewer parameters.

CONCLUSIONS

The improved reconstruction performance demonstrates that our proposed time correlation modeling is simple and effective for accelerating MRI reconstruction. We hope our approach can serve as a reference for future research in dynamic MRI reconstruction.

MRI denoising with a non-blind deep complex-valued convolutional neural network

MR images with high signal-to-noise ratio (SNR) provide more diagnostic information. Various methods for MRI denoising have been developed, but the majority of them operate on the magnitude image and neglect the phase information. Therefore, the goal of this work is to design and implement a complex-valued convolutional neural network (CNN) for MRI denoising. A complex-valued CNN incorporating the noise level map (non-blindℂ $$ \mathbb{C} $$ DnCNN) was trained with ground truth and simulated noise-corrupted image pairs. The proposed method was validated using both simulated and in vivo data collected from low-field scanners. Its denoising performance was quantitively and qualitatively evaluated, and it was compared with the real-valued CNN and several other algorithms. For the simulated noise-corrupted testing dataset, the complex-valued models had superior normalized root-mean-square error, peak SNR, structural similarity index, and phase ABSD. By incorporating the noise level map, the non-blindℂ $$ \mathbb{C} $$ DnCNN showed better performance in dealing with spatially varying parallel imaging noise. For in vivo low-field data, the non-blindℂ $$ \mathbb{C} $$ DnCNN significantly improved the SNR and visual quality of the image. The proposed non-blindℂ $$ \mathbb{C} $$ DnCNN provides an efficient and effective approach for MRI denoising. This is the first application of non-blindℂ $$ \mathbb{C} $$ DnCNN to medical imaging. The method holds the potential to enable improved low-field MRI, facilitating enhanced diagnostic imaging in under-resourced areas.

AutoSamp: Autoencoding k-Space Sampling via Variational Information Maximization for 3D MRI

Accelerated MRI protocols routinely involve a predefined sampling pattern that undersamples the k-space. Finding an optimal pattern can enhance the reconstruction quality, however this optimization is a challenging task. To address this challenge, we introduce a novel deep learning framework, AutoSamp, based on variational information maximization that enables joint optimization of sampling pattern and reconstruction of MRI scans. We represent the encoder as a non-uniform Fast Fourier Transform that allows continuous optimization of k-space sample locations on a non-Cartesian plane, and the decoder as a deep reconstruction network. Experiments on public 3D acquired MRI datasets show improved reconstruction quality of the proposed AutoSamp method over the prevailing variable density and variable density Poisson disc sampling for both compressed sensing and deep learning reconstructions. We demonstrate that our data-driven sampling optimization method achieves 4.4dB, 2.0dB, 0.75dB, 0.7dB PSNR improvements over reconstruction with Poisson Disc masks for acceleration factors of R =5, 10, 15, 25, respectively. Prospectively accelerated acquisitions with 3D FSE sequences using our optimized sampling patterns exhibit improved image quality and sharpness. Furthermore, we analyze the characteristics of the learned sampling patterns with respect to changes in acceleration factor, measurement noise, underlying anatomy, and coil sensitivities. We show that all these factors contribute to the optimization result by affecting the sampling density, k-space coverage and point spread functions of the learned sampling patterns.

Systems Engineering Approach Towards Sensitive Cellular Fluorine-19 MRI

In vivo fluorine-19 MRI using F-based tracer media has shown utility and versatility for a wide range of biomedical uses, particularly immune and stem cell detection, as well as biosensing. As with many advanced MRI acquisition techniques, the sensitivity and limit of detection (LOD) in vivo is a key consideration for a successful study outcome. In this review, we analyze the primary factors that limit cell LOD. The achievable sensitivity is strongly dependent on the specific composition of tracer, cell type of interest, cell activity, data acquisition and reconstruction methods, and MRI hardware design. Recent innovations in molecular 19F tracer design and image acquisition-reconstruction methods have achieved significant leaps in 19F MRI sensitivity, and integration of these new materials and methods into studies can result in > 10-fold improvement in LOD. These developments will help unlock the full potential of clinical 19F MRI for biomedical applications.

ACGRHA-Net: Accelerated multi-contrast MR imaging with adjacency complementary graph assisted residual hybrid attention network

Multi-contrast magnetic resonance (MR) imaging is an advanced technology used in medical diagnosis, but the long acquisition process can lead to patient discomfort and limit its broader application. Shortening acquisition time by undersampling k-space data introduces noticeable aliasing artifacts. To address this, we propose a method that reconstructs multi-contrast MR images from zero-filled data by utilizing a fully-sampled auxiliary contrast MR image as a prior to learn an adjacency complementary graph. This graph is then combined with a residual hybrid attention network, forming the adjacency complementary graph assisted residual hybrid attention network (ACGRHA-Net) for multi-contrast MR image reconstruction. Specifically, the optimal structural similarity is represented by a graph learned from the fully sampled auxiliary image, where the node features and adjacency matrices are designed to precisely capture structural information among different contrast images. This structural similarity enables effective fusion with the target image, improving the detail reconstruction. Additionally, a residual hybrid attention module is designed in parallel with the graph convolution network, allowing it to effectively capture key features and adaptively emphasize these important features in target contrast MR images. This strategy prioritizes crucial information while preserving shallow features, thereby achieving comprehensive feature fusion at deeper levels to enhance multi-contrast MR image reconstruction. Extensive experiments on the different datasets, using various sampling patterns and accelerated factors demonstrate that the proposed method outperforms the current state-of-the-art reconstruction methods.

Research on deep unfolding network reconstruction method based on scalable sampling of transient signals

In order to solve the problems of long reconstruction time and low reconstruction accuracy of compressed sensing reconstruction algorithm in the measurement of transient signals, a deep unfolding network reconstruction method based on scalable sampling is proposed to achieve fast and high-quality reconstruction of transient signal under low number of measurements. Firstly, the measurement process of compressed sensing is embedded into the neural network to realize automatic design and optimization of the observation matrix, which can reduce the number of measurements. Secondly, scalable sampling is introduced into the measurement process of compressed sensing, which can realize the training of data with different sampling ratios in the same model. Finally, a deep unfolding network model is designed to reconstruct the transient signal, which not only realizes the interpretability of the reconstructed network, but also achieves fast and high-quality reconstruction of the transient signal under the low number of measurements. Experimental results show that compared with the traditional compressed sensing reconstruction algorithms, the proposed method can obtain high-quality reconstruction accuracy with lower measurement times, and the reconstruction time is greatly reduced. The algorithm in this paper also obtains good reconstruction results under different sampling ratios, which shows that the method in this paper has good adaptability and effectiveness.

A review of deep learning-based reconstruction methods for accelerated MRI using spatiotemporal and multi-contrast redundancies

Accelerated magnetic resonance imaging (MRI) has played an essential role in reducing data acquisition time for MRI. Acceleration can be achieved by acquiring fewer data points in k-space, which results in various artifacts in the image domain. Conventional reconstruction methods have resolved the artifacts by utilizing multi-coil information, but with limited robustness. Recently, numerous deep learning-based reconstruction methods have been developed, enabling outstanding reconstruction performances with higher acceleration. Advances in hardware and developments of specialized network architectures have produced such achievements. Besides, MRI signals contain various redundant information including multi-coil redundancy, multi-contrast redundancy, and spatiotemporal redundancy. Utilization of the redundant information combined with deep learning approaches allow not only higher acceleration, but also well-preserved details in the reconstructed images. Consequently, this review paper introduces the basic concepts of deep learning and conventional accelerated MRI reconstruction methods, followed by review of recent deep learning-based reconstruction methods that exploit various redundancies. Lastly, the paper concludes by discussing the challenges, limitations, and potential directions of future developments.

FEFA: Frequency Enhanced Multi-Modal MRI Reconstruction With Deep Feature Alignment

Integrating complementary information from multiple magnetic resonance imaging (MRI) modalities is often necessary to make accurate and reliable diagnostic decisions. However, the different acquisition speeds of these modalities mean that obtaining information can be time consuming and require significant effort. Reference-based MRI reconstruction aims to accelerate slower, under-sampled imaging modalities, such as T2-modality, by utilizing redundant information from faster, fully sampled modalities, such as T1-modality. Unfortunately, spatial misalignment between different modalities often negatively impacts the final results. To address this issue, we propose FEFA, which consists of cascading FEFA blocks. The FEFA block first aligns and fuses the two modalities at the feature level. The combined features are then filtered in the frequency domain to enhance the important features while simultaneously suppressing the less essential ones, thereby ensuring accurate reconstruction. Furthermore, we emphasize the advantages of combining the reconstruction results from multiple cascaded blocks, which also contributes to stabilizing the training process. Compared to existing registration-then-reconstruction and cross-attention-based approaches, our method is end-to-end trainable without requiring additional supervision, extensive parameters, or heavy computation. Experiments on the public fastMRI, IXI and in-house datasets demonstrate that our approach is effective across various under-sampling patterns and ratios.

Physics-Informed DeepMRI: k-Space Interpolation Meets Heat Diffusion

Recently, diffusion models have shown considerable promise for MRI reconstruction. However, extensive experimentation has revealed that these models are prone to generating artifacts due to the inherent randomness involved in generating images from pure noise. To achieve more controlled image reconstruction, we reexamine the concept of interpolatable physical priors in k-space data, focusing specifically on the interpolation of high-frequency (HF) k-space data from low-frequency (LF) k-space data. Broadly, this insight drives a shift in the generation paradigm from random noise to a more deterministic approach grounded in the existing LF k-space data. Building on this, we first establish a relationship between the interpolation of HF k-space data from LF k-space data and the reverse heat diffusion process, providing a fundamental framework for designing diffusion models that generate missing HF data. To further improve reconstruction accuracy, we integrate a traditional physics-informed k-space interpolation model into our diffusion framework as a data fidelity term. Experimental validation using publicly available datasets demonstrates that our approach significantly surpasses traditional k-space interpolation methods, deep learning-based k-space interpolation techniques, and conventional diffusion models, particularly in HF regions. Finally, we assess the generalization performance of our model across various out-of-distribution datasets. Our code are available at https://github.com/ZhuoxuCui/Heat-Diffusion.

Numerical and Clinical Evaluation of the Robustness of Open-source Networks for Parallel MR Imaging Reconstruction

PURPOSE

Deep neural networks (DNNs) for MRI reconstruction often require large datasets for training. Still, in clinical settings, the domains of datasets are diverse, and how robust DNNs are to domain differences between training and testing datasets has been an open question. Here, we numerically and clinically evaluate the generalization of the reconstruction networks across various domains under clinically practical conditions and provide practical guidance on what points to consider when selecting models for clinical application.

METHODS

We compare the reconstruction performance between four network models: U-Net, the deep cascade of convolutional neural networks (DC-CNNs), Hybrid Cascade, and variational network (VarNet). We used the public multicoil dataset fastMRI for training and testing and performed a single-domain test, where the domains of the dataset used for training and testing were the same, and cross-domain tests, where the source and target domains were different. We conducted a single-domain test (Experiment 1) and cross-domain tests (Experiments 2-4), focusing on six factors (the number of images, sampling pattern, acceleration factor, noise level, contrast, and anatomical structure) both numerically and clinically.

RESULTS

U-Net had lower performance than the three model-based networks and was less robust to domain shifts between training and testing datasets. VarNet had the highest performance and robustness among the three model-based networks, followed by Hybrid Cascade and DC-CNN. Especially, VarNet showed high performance even with a limited number of training images (200 images/10 cases). U-Net was more robust to domain shifts concerning noise level than the other model-based networks. Hybrid Cascade showed slightly better performance and robustness than DC-CNN, except for robustness to noise-level domain shifts. The results of the clinical evaluations generally agreed with the results of the quantitative metrics.

CONCLUSION

In this study, we numerically and clinically evaluated the robustness of the publicly available networks using the multicoil data. Therefore, this study provided practical guidance for clinical applications.

Hallucination Index: An Image Quality Metric for Generative Reconstruction Models

Generative image reconstruction algorithms such as measurement conditioned diffusion models are increasingly popular in the field of medical imaging. These powerful models can transform low signal-to-noise ratio (SNR) inputs into outputs with the appearance of high SNR. However, the outputs can have a new type of error called hallucinations. In medical imaging, these hallucinations may not be obvious to a Radiologist but could cause diagnostic errors. Generally, hallucination refers to error in estimation of object structure caused by a machine learning model, but there is no widely accepted method to evaluate hallucination magnitude. In this work, we propose a new image quality metric called the hallucination index. Our approach is to compute the Hellinger distance from the distribution of reconstructed images to a zero hallucination reference distribution. To evaluate our approach, we conducted a numerical experiment with electron microscopy images, simulated noisy measurements, and applied diffusion based reconstructions. We sampled the measurements and the generative reconstructions repeatedly to compute the sample mean and covariance. For the zero hallucination reference, we used the forward diffusion process applied to ground truth. Our results show that higher measurement SNR leads to lower hallucination index for the same apparent image quality. We also evaluated the impact of early stopping in the reverse diffusion process and found that more modest denoising strengths can reduce hallucination. We believe this metric could be useful for evaluation of generative image reconstructions or as a warning label to inform radiologists about the degree of hallucinations in medical images.

Using a deep learning prior for accelerating hyperpolarized 13C MRSI on synthetic cancer datasets

PURPOSE

We aimed to incorporate a deep learning prior with k-space data fidelity for accelerating hyperpolarized carbon-13 MRSI, demonstrated on synthetic cancer datasets.

METHODS

A two-site exchange model, derived from the Bloch equation of MR signal evolution, was firstly used in simulating training and testing data, that is, synthetic phantom datasets. Five singular maps generated from each simulated dataset were used to train a deep learning prior, which was then employed with the fidelity term to reconstruct the undersampled MRI k-space data. The proposed method was assessed on synthetic human brain tumor images (N = 33), prostate cancer images (N = 72), and mouse tumor images (N = 58) for three undersampling factors and 2.5% additive Gaussian noise. Furthermore, varied levels of Gaussian noise with SDs of 2.5%, 5%, and 10% were added on synthetic prostate cancer data, and corresponding reconstruction results were evaluated.

RESULTS

For quantitative evaluation, peak SNRs were approximately 32 dB, and the accuracy was generally improved for 5 to 8 dB compared with those from compressed sensing with L1-norm regularization or total variation regularization. Reasonable normalized RMS error were obtained. Our method also worked robustly against noise, even on a data with noise SD of 10%.

CONCLUSION

The proposed singular value decomposition + iterative deep learning model could be considered as a general framework that extended the application of deep learning MRI reconstruction to metabolic imaging. The morphology of tumors and metabolic images could be measured robustly in six times acceleration using our method.

Model-based federated learning for accurate MR image reconstruction from undersampled k-space data

Deep learning-based methods have achieved encouraging performances in the field of Magnetic Resonance (MR) image reconstruction. Nevertheless, building powerful and robust deep learning models requires collecting large and diverse datasets from multiple centers. This raises concerns about ethics and data privacy. Recently, federated learning has emerged as a promising solution, enabling the utilization of multi-center data without the need for data transfer between institutions. Despite its potential, existing federated learning methods face challenges due to the high heterogeneity of data from different centers. Aggregation methods based on simple averaging, which are commonly used to combine the client's information, have shown limited reconstruction and generalization capabilities. In this paper, we propose a Model-based Federated learning framework (ModFed) to address these challenges. ModFed has three major contributions: (1) Different from existing data-driven federated learning methods, ModFed designs attention-assisted model-based neural networks that can alleviate the need for large amounts of data on each client; (2) To address the data heterogeneity issue, ModFed proposes an adaptive dynamic aggregation scheme, which can improve the generalization capability and robustness of the trained neural network models; (3) ModFed incorporates a spatial Laplacian attention mechanism and a personalized client-side loss regularization to capture the detailed information for accurate image reconstruction. The effectiveness of the proposed ModFed is evaluated on three in-vivo datasets. Experimental results show that when compared to six existing state-of-the-art federated learning approaches, ModFed achieves better MR image reconstruction performance with increased generalization capability. Codes will be made available at https://github.com/ternencewu123/ModFed.

Diffusion Modeling with Domain-conditioned Prior Guidance for Accelerated MRI and qMRI Reconstruction

This study introduces a novel image reconstruction technique based on a diffusion model that is conditioned on the native data domain. Our method is applied to multi-coil MRI and quantitative MRI (qMRI) reconstruction, leveraging the domain-conditioned diffusion model within the frequency and parameter domains. The prior MRI physics are used as embeddings in the diffusion model, enforcing data consistency to guide the training and sampling process, characterizing MRI k-space encoding in MRI reconstruction, and leveraging MR signal modeling for qMRI reconstruction. Furthermore, a gradient descent optimization is incorporated into the diffusion steps, enhancing feature learning and improving denoising. The proposed method demonstrates a significant promise, particularly for reconstructing images at high acceleration factors. Notably, it maintains great reconstruction accuracy for static and quantitative MRI reconstruction across diverse anatomical structures. Beyond its immediate applications, this method provides potential generalization capability, making it adaptable to inverse problems across various domains.

Knowledge-driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un-supervised learning

Deep learning (DL) has emerged as a leading approach in accelerating MRI. It employs deep neural networks to extract knowledge from available datasets and then applies the trained networks to reconstruct accurate images from limited measurements. Unlike natural image restoration problems, MRI involves physics-based imaging processes, unique data properties, and diverse imaging tasks. This domain knowledge needs to be integrated with data-driven approaches. Our review will introduce the significant challenges faced by such knowledge-driven DL approaches in the context of fast MRI along with several notable solutions, which include learning neural networks and addressing different imaging application scenarios. The traits and trends of these techniques have also been given which have shifted from supervised learning to semi-supervised learning, and finally, to unsupervised learning methods. In addition, MR vendors' choices of DL reconstruction have been provided along with some discussions on open questions and future directions, which are critical for the reliable imaging systems.

Mutli-modal straight flow matching for accelerated MR imaging

Diffusion models have garnered great interest lately in Magnetic Resonance (MR) image reconstruction. A key component of generating high-quality samples from noise is iterative denoising for thousands of steps. However, the complexity of inference steps has limited its applications. To solve the challenge in obtaining high-quality reconstructed images with fewer inference steps and computational complexity, we introduce a novel straight flow matching, based on a neural ordinary differential equation (ODE) generative model. Our model creates a linear path between undersampled images and reconstructed images, which can be accurately simulated with a few Euler steps. Furthermore, we propose a multi-modal straight flow matching model, which uses relatively easily available modalities as supplementary information to guide the reconstruction of target modalities. We introduce the low frequency fusion layer and the high frequency fusion layer into our multi-modal model, which has been proved to produce promising results in fusion tasks. The proposed multi-modal straight flow matching (MMSflow) achieves state-of-the-art performances in task of reconstruction in fastMRI and Brats-2020 and improves the sampling rate by an order of magnitude than other methods based on stochastic differential equations (SDE).

Accelerated cardiac cine magnetic resonance imaging using deep low-rank plus sparse network: validation in patients

Background

Accurate and reproducible assessment of left ventricular (LV) volumes is important in managing various cardiac conditions. However, patients are required to hold their breath multiple times during data acquisition, which may result in discomfort and restrict cardiac motion, potentially compromising the accuracy of the detected results. Accelerated imaging techniques can help reduce the number of breath holds needed, potentially improving patient comfort and the reliability of the LV assessment. This study aimed to prospectively evaluate the feasibility and accuracy of LV assessment with a model-based low-rank plus sparse network (L+S-Net) for accelerated magnetic resonance (MR) cine imaging.

Methods

Fourty-one patients with different cardiac conditions were recruited in this study. Both accelerated MR cine imaging with L+S-Net and traditional electrocardiogram (ECG)-gated segmented cine were performed for each patient. Subjective image quality (IQ) score and quantitative LV volume function parameters were measured and compared between L+S-Net and traditional standards. The IQ score and LV volume measurements of cardiovascular magnetic resonance (CMR) images reconstructed by L+S-Net and standard cine were compared by paired t-test. The acquisition time of the two methods was also calculated.

Results

In a quantitative analysis, L+S-Net and standard cine yielded similar measurements for all parameters of LV function (ejection fraction: 35±22 for standard vs. 33±23 for L+S-Net), although L+S-Net had slightly lower IQ scores than standard cine CMR (4.2±0.5 for L+S-Net vs. 4.8±0.4 for standard cine; P<0.001). The mean acquisition time of L+S-Net and standard cine was 0.83±0.08 vs. 6.35±0.78 s per slice (P<0.001).

Conclusions

Assessment of LV function with L+S-Net at 3.0 T yields comparable results to the reference standard, albeit with a reduced acquisition time. This feature enhances the clinical applicability of the L+S-Net approach, helping alleviate patient discomfort and motion artifacts that may arise due to prolonged acquisition time.

Deep learning for accelerated and robust MRI reconstruction

Deep learning (DL) has recently emerged as a pivotal technology for enhancing magnetic resonance imaging (MRI), a critical tool in diagnostic radiology. This review paper provides a comprehensive overview of recent advances in DL for MRI reconstruction, and focuses on various DL approaches and architectures designed to improve image quality, accelerate scans, and address data-related challenges. It explores end-to-end neural networks, pre-trained and generative models, and self-supervised methods, and highlights their contributions to overcoming traditional MRI limitations. It also discusses the role of DL in optimizing acquisition protocols, enhancing robustness against distribution shifts, and tackling biases. Drawing on the extensive literature and practical insights, it outlines current successes, limitations, and future directions for leveraging DL in MRI reconstruction, while emphasizing the potential of DL to significantly impact clinical imaging practices.

Paired conditional generative adversarial network for highly accelerated liver 4D MRI

Purpose.4D MRI with high spatiotemporal resolution is desired for image-guided liver radiotherapy. Acquiring densely sampling k-space data is time-consuming. Accelerated acquisition with sparse samples is desirable but often causes degraded image quality or long reconstruction time. We propose the Reconstruct Paired Conditional Generative Adversarial Network (Re-Con-GAN) to shorten the 4D MRI reconstruction time while maintaining the reconstruction quality.Methods.Patients who underwent free-breathing liver 4D MRI were included in the study. Fully- and retrospectively under-sampled data at 3, 6 and 10 times (3×, 6× and 10×) were first reconstructed using the nuFFT algorithm. Re-Con-GAN then trained input and output in pairs. Three types of networks, ResNet9, UNet and reconstruction swin transformer (RST), were explored as generators. PatchGAN was selected as the discriminator. Re-Con-GAN processed the data (3D +t) as temporal slices (2D +t). A total of 48 patients with 12 332 temporal slices were split into training (37 patients with 10 721 slices) and test (11 patients with 1611 slices). Compressed sensing (CS) reconstruction with spatiotemporal sparsity constraint was used as a benchmark. Reconstructed image quality was further evaluated with a liver gross tumor volume (GTV) localization task using Mask-RCNN trained from a separate 3D static liver MRI dataset (70 patients; 103 GTV contours).Results.Re-Con-GAN consistently achieved comparable/better PSNR, SSIM, and RMSE scores compared to CS/UNet models. The inference time of Re-Con-GAN, UNet and CS are 0.15, 0.16, and 120 s. The GTV detection task showed that Re-Con-GAN and CS, compared to UNet, better improved the dice score (3× Re-Con-GAN 80.98%; 3× CS 80.74%; 3× UNet 79.88%) of unprocessed under-sampled images (3× 69.61%).Conclusion.A generative network with adversarial training is proposed with promising and efficient reconstruction results demonstrated on an in-house dataset. The rapid and qualitative reconstruction of 4D liver MR has the potential to facilitate online adaptive MR-guided radiotherapy for liver cancer.

Synthesized 7T MPRAGE From 3T MPRAGE Using Generative Adversarial Network and Validation in Clinical Brain Imaging: A Feasibility Study

BACKGROUND

Ultra-high field 7T MRI can provide excellent tissue contrast and anatomical details, but is often cost prohibitive, and is not widely accessible in clinical practice.

PURPOSE

To generate synthetic 7T images from widely acquired 3T images with deep learning and to evaluate the feasibility of this approach for brain imaging.

STUDY TYPE

Prospective.

POPULATION

33 healthy volunteers and 89 patients with brain diseases, divided into training, and evaluation datasets in the ratio 4:1.

SEQUENCE AND FIELD STRENGTH

T1-weighted nonenhanced or contrast-enhanced magnetization-prepared rapid acquisition gradient-echo sequence at both 3T and 7T.

ASSESSMENT

A generative adversarial network (SynGAN) was developed to produce synthetic 7T images from 3T images as input. SynGAN training and evaluation were performed separately for nonenhanced and contrast-enhanced paired acquisitions. Qualitative image quality of acquired 3T and 7T images and of synthesized 7T images was evaluated by three radiologists in terms of overall image quality, artifacts, sharpness, contrast, and visualization of vessel using 5-point Likert scales.

STATISTICAL TESTS

Wilcoxon signed rank tests to compare synthetic 7T images with acquired 7T and 3T images and intraclass correlation coefficients to evaluate interobserver variability. P < 0.05 was considered significant.

RESULTS

Of the 122 paired 3T and 7T MRI scans, 66 were acquired without contrast agent and 56 with contrast agent. The average time to generate synthetic images was ~11.4 msec per slice (2.95 sec per participant). The synthetic 7T images achieved significantly improved tissue contrast and sharpness in comparison to 3T images in both nonenhanced and contrast-enhanced subgroups. Meanwhile, there was no significant difference between acquired 7T and synthetic 7T images in terms of all the evaluation criteria for both nonenhanced and contrast-enhanced subgroups (P ≥ 0.180).

DATA CONCLUSION

The deep learning model has potential to generate synthetic 7T images with similar image quality to acquired 7T images.

LEVEL OF EVIDENCE

2 TECHNICAL EFFICACY: Stage 1.

Cerebrospinal fluid flow artifact reduction with deep learning to optimize the evaluation of spinal canal stenosis on spine MRI

PURPOSE

The aim of study was to employ the Cycle Generative Adversarial Network (CycleGAN) deep learning model to diminish the cerebrospinal fluid (CSF) flow artifacts in cervical spine MRI. We also evaluate the agreement in quantifying spinal canal stenosis.

METHODS

For training model, we collected 9633 axial MR image pairs from 399 subjects. Then, additional 104 image pairs from 19 subjects were gathered for the test set. The deep learning model was developed using CycleGAN to reduce CSF flow artifacts, where T2 TSE images served as input, and T2 FFE images, known for fewer CSF flow artifacts. Post training, CycleGAN-generated images were subjected to both quantitative and qualitative evaluations for CSF artifacts. For assessing the agreement of spinal canal stenosis, four raters utilized an additional 104 pairs of original and CycleGAN-generated images, with inter-rater agreement evaluated using a weighted kappa value.

RESULTS

CSF flow artifacts were reduced in the CycleGAN-generated images compared to the T2 TSE and FFE images in both quantitative and qualitative analysis. All raters concordantly displayed satisfactory estimation results when assessing spinal canal stenosis using the CycleGAN-generated images with T2 TSE images (kappa = 0.61-0.75) compared to the original FFE with T2 TSE images (kappa = 0.48-0.71).

CONCLUSIONS

CycleGAN demonstrated the capability to produce images with diminished CSF flow artifacts. When paired with T2 TSE images, the CycleGAN-generated images allowed for more consistent assessment of spinal canal stenosis and exhibited agreement levels that were comparable to the combination of T2 TSE and FFE images.

Fast MRI Reconstruction Using Deep Learning-based Compressed Sensing: A Systematic Review

Magnetic resonance imaging (MRI) has revolutionized medical imaging, providing a non-invasive and highly detailed look into the human body. However, the long acquisition times of MRI present challenges, causing patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, researchers are exploring various techniques to reduce acquisition time and improve the overall efficiency of MRI. One such technique is compressed sensing (CS), which reduces data acquisition by leveraging image sparsity in transformed spaces. In recent years, deep learning (DL) has been integrated with CS-MRI, leading to a new framework that has seen remarkable growth. DL-based CS-MRI approaches are proving to be highly effective in accelerating MR imaging without compromising image quality. This review comprehensively examines DL-based CS-MRI techniques, focusing on their role in increasing MR imaging speed. We provide a detailed analysis of each category of DL-based CS-MRI including end-to-end, unroll optimization, self-supervised, and federated learning. Our systematic review highlights significant contributions and underscores the exciting potential of DL in CS-MRI. Additionally, our systematic review efficiently summarizes key results and trends in DL-based CS-MRI including quantitative metrics, the dataset used, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based CS-MRI in the advancement of medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based CS-MRI publications and publicly available datasets - https://github.com/mosaf/Awesome-DL-based-CS-MRI.

Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy

Ultrasound localization microscopy (ULM) enables deep tissue microvascular imaging by localizing and tracking intravenously injected microbubbles circulating in the bloodstream. However, conventional localization techniques require spatially isolated microbubbles, resulting in prolonged imaging time to obtain detailed microvascular maps. Here, we introduce LOcalization with Context Awareness (LOCA)-ULM, a deep learning-based microbubble simulation and localization pipeline designed to enhance localization performance in high microbubble concentrations. In silico, LOCA-ULM enhanced microbubble detection accuracy to 97.8% and reduced the missing rate to 23.8%, outperforming conventional and deep learning-based localization methods up to 17.4% in accuracy and 37.6% in missing rate reduction. In in vivo rat brain imaging, LOCA-ULM revealed dense cerebrovascular networks and spatially adjacent microvessels undetected by conventional ULM. We further demonstrate the superior localization performance of LOCA-ULM in functional ULM (fULM) where LOCA-ULM significantly increased the functional imaging sensitivity of fULM to hemodynamic responses invoked by whisker stimulations in the rat brain.

DCT-net: Dual-domain cross-fusion transformer network for MRI reconstruction

Current challenges in Magnetic Resonance Imaging (MRI) include long acquisition times and motion artifacts. To address these issues, under-sampled k-space acquisition has gained popularity as a fast imaging method. However, recovering fine details from under-sampled data remains challenging. In this study, we introduce a pioneering deep learning approach, namely DCT-Net, designed for dual-domain MRI reconstruction. DCT-Net seamlessly integrates information from the image domain (IRM) and frequency domain (FRM), utilizing a novel Cross Attention Block (CAB) and Fusion Attention Block (FAB). These innovative blocks enable precise feature extraction and adaptive fusion across both domains, resulting in a significant enhancement of the reconstructed image quality. The adaptive interaction and fusion mechanisms of CAB and FAB contribute to the method's effectiveness in capturing distinctive features and optimizing image reconstruction. Comprehensive ablation studies have been conducted to assess the contributions of these modules to reconstruction quality and accuracy. Experimental results on the FastMRI (2023) and Calgary-Campinas datasets (2021) demonstrate the superiority of our MRI reconstruction framework over other typical methods (most are illustrated in 2023 or 2022) in both qualitative and quantitative evaluations. This holds for knee and brain datasets under 4× and 8× accelerated imaging scenarios.

NPB-REC: A non-parametric Bayesian deep-learning approach for undersampled MRI reconstruction with uncertainty estimation

The ability to reconstruct high-quality images from undersampled MRI data is vital in improving MRI temporal resolution and reducing acquisition times. Deep learning methods have been proposed for this task, but the lack of verified methods to quantify the uncertainty in the reconstructed images hampered clinical applicability. We introduce "NPB-REC", a non-parametric fully Bayesian framework, for MRI reconstruction from undersampled data with uncertainty estimation. We use Stochastic Gradient Langevin Dynamics during training to characterize the posterior distribution of the network parameters. This enables us to both improve the quality of the reconstructed images and quantify the uncertainty in the reconstructed images. We demonstrate the efficacy of our approach on a multi-coil MRI dataset from the fastMRI challenge and compare it to the baseline End-to-End Variational Network (E2E-VarNet). Our approach outperforms the baseline in terms of reconstruction accuracy by means of PSNR and SSIM (34.55, 0.908 vs. 33.08, 0.897, p<0.01, acceleration rate R=8) and provides uncertainty measures that correlate better with the reconstruction error (Pearson correlation, R=0.94 vs. R=0.91). Additionally, our approach exhibits better generalization capabilities against anatomical distribution shifts (PSNR and SSIM of 32.38, 0.849 vs. 31.63, 0.836, p<0.01, training on brain data, inference on knee data, acceleration rate R=8). NPB-REC has the potential to facilitate the safe utilization of deep learning-based methods for MRI reconstruction from undersampled data. Code and trained models are available at https://github.com/samahkh/NPB-REC.

Self-Supervised Deep Unrolled Reconstruction Using Regularization by Denoising

Deep learning methods have been successfully used in various computer vision tasks. Inspired by that success, deep learning has been explored in magnetic resonance imaging (MRI) reconstruction. In particular, integrating deep learning and model-based optimization methods has shown considerable advantages. However, a large amount of labeled training data is typically needed for high reconstruction quality, which is challenging for some MRI applications. In this paper, we propose a novel reconstruction method, named DURED-Net, that enables interpretable self-supervised learning for MR image reconstruction by combining a self-supervised denoising network and a plug-and-play method. We aim to boost the reconstruction performance of Noise2Noise in MR reconstruction by adding an explicit prior that utilizes imaging physics. Specifically, the leverage of a denoising network for MRI reconstruction is achieved using Regularization by Denoising (RED). Experiment results demonstrate that the proposed method requires a reduced amount of training data to achieve high reconstruction quality among the state-of-the-art approaches utilizing Noise2Noise.

AliasNet: Alias artefact suppression network for accelerated phase-encode MRI

Sparse reconstruction is an important aspect of MRI, helping to reduce acquisition time and improve spatial-temporal resolution. Popular methods are based mostly on compressed sensing (CS), which relies on the random sampling of k-space to produce incoherent (noise-like) artefacts. Due to hardware constraints, 1D Cartesian phase-encode under-sampling schemes are popular for 2D CS-MRI. However, 1D under-sampling limits 2D incoherence between measurements, yielding structured aliasing artefacts (ghosts) that may be difficult to remove assuming a 2D sparsity model. Reconstruction algorithms typically deploy direction-insensitive 2D regularisation for these direction-associated artefacts. Recognising that phase-encode artefacts can be separated into contiguous 1D signals, we develop two decoupling techniques that enable explicit 1D regularisation and leverage the excellent 1D incoherence characteristics. We also derive a combined 1D + 2D reconstruction technique that takes advantage of spatial relationships within the image. Experiments conducted on retrospectively under-sampled brain and knee data demonstrate that combination of the proposed 1D AliasNet modules with existing 2D deep learned (DL) recovery techniques leads to an improvement in image quality. We also find AliasNet enables a superior scaling of performance compared to increasing the size of the original 2D network layers. AliasNet therefore improves the regularisation of aliasing artefacts arising from phase-encode under-sampling, by tailoring the network architecture to account for their expected appearance. The proposed 1D + 2D approach is compatible with any existing 2D DL recovery technique deployed for this application.

MedFusionGAN: multimodal medical image fusion using an unsupervised deep generative adversarial network

PURPOSE

This study proposed an end-to-end unsupervised medical fusion generative adversarial network, MedFusionGAN, to fuse computed tomography (CT) and high-resolution isotropic 3D T1-Gd Magnetic resonance imaging (MRI) image sequences to generate an image with CT bone structure and MRI soft tissue contrast to improve target delineation and to reduce the radiotherapy planning time.

METHODS

We used a publicly available multicenter medical dataset (GLIS-RT, 230 patients) from the Cancer Imaging Archive. To improve the models generalization, we consider different imaging protocols and patients with various brain tumor types, including metastases. The proposed MedFusionGAN consisted of one generator network and one discriminator network trained in an adversarial scenario. Content, style, and L1 losses were used for training the generator to preserve the texture and structure information of the MRI and CT images.

RESULTS

The MedFusionGAN successfully generates fused images with MRI soft-tissue and CT bone contrast. The results of the MedFusionGAN were quantitatively and qualitatively compared with seven traditional and eight deep learning (DL) state-of-the-art methods. Qualitatively, our method fused the source images with the highest spatial resolution without adding the image artifacts. We reported nine quantitative metrics to quantify the preservation of structural similarity, contrast, distortion level, and image edges in fused images. Our method outperformed both traditional and DL methods on six out of nine metrics. And it got the second performance rank for three and two quantitative metrics when compared with traditional and DL methods, respectively. To compare soft-tissue contrast, intensity profile along tumor and tumor contours of the fusion methods were evaluated. MedFusionGAN provides a more consistent, better intensity profile, and a better segmentation performance.

CONCLUSIONS

The proposed end-to-end unsupervised method successfully fused MRI and CT images. The fused image could improve targets and OARs delineation, which is an important aspect of radiotherapy treatment planning.

Subspace Model-Assisted Deep Learning for Improved Image Reconstruction

Image reconstruction from limited and/or sparse data is known to be an ill-posed problem and a priori information/constraints have played an important role in solving the problem. Early constrained image reconstruction methods utilize image priors based on general image properties such as sparsity, low-rank structures, spatial support bound, etc. Recent deep learning-based reconstruction methods promise to produce even higher quality reconstructions by utilizing more specific image priors learned from training data. However, learning high-dimensional image priors requires huge amounts of training data that are currently not available in medical imaging applications. As a result, deep learning-based reconstructions often suffer from two known practical issues: a) sensitivity to data perturbations (e.g., changes in data sampling scheme), and b) limited generalization capability (e.g., biased reconstruction of lesions). This paper proposes a new method to address these issues. The proposed method synergistically integrates model-based and data-driven learning in three key components. The first component uses the linear vector space framework to capture global dependence of image features; the second exploits a deep network to learn the mapping from a linear vector space to a nonlinear manifold; the third is an unrolling-based deep network that captures local residual features with the aid of a sparsity model. The proposed method has been evaluated with magnetic resonance imaging data, demonstrating improved reconstruction in the presence of data perturbation and/or novel image features. The method may enhance the practical utility of deep learning-based image reconstruction.

Developing and deploying deep learning models in brain magnetic resonance imaging: A review

Magnetic resonance imaging (MRI) of the brain has benefited from deep learning (DL) to alleviate the burden on radiologists and MR technologists, and improve throughput. The easy accessibility of DL tools has resulted in a rapid increase of DL models and subsequent peer-reviewed publications. However, the rate of deployment in clinical settings is low. Therefore, this review attempts to bring together the ideas from data collection to deployment in the clinic, building on the guidelines and principles that accreditation agencies have espoused. We introduce the need for and the role of DL to deliver accessible MRI. This is followed by a brief review of DL examples in the context of neuropathologies. Based on these studies and others, we collate the prerequisites to develop and deploy DL models for brain MRI. We then delve into the guiding principles to develop good machine learning practices in the context of neuroimaging, with a focus on explainability. A checklist based on the United States Food and Drug Administration's good machine learning practices is provided as a summary of these guidelines. Finally, we review the current challenges and future opportunities in DL for brain MRI.

Parallel-stream fusion of scan-specific and scan-general priors for learning deep MRI reconstruction in low-data regimes

Magnetic resonance imaging (MRI) is an essential diagnostic tool that suffers from prolonged scan times. Reconstruction methods can alleviate this limitation by recovering clinically usable images from accelerated acquisitions. In particular, learning-based methods promise performance leaps by employing deep neural networks as data-driven priors. A powerful approach uses scan-specific (SS) priors that leverage information regarding the underlying physical signal model for reconstruction. SS priors are learned on each individual test scan without the need for a training dataset, albeit they suffer from computationally burdening inference with nonlinear networks. An alternative approach uses scan-general (SG) priors that instead leverage information regarding the latent features of MRI images for reconstruction. SG priors are frozen at test time for efficiency, albeit they require learning from a large training dataset. Here, we introduce a novel parallel-stream fusion model (PSFNet) that synergistically fuses SS and SG priors for performant MRI reconstruction in low-data regimes, while maintaining competitive inference times to SG methods. PSFNet implements its SG prior based on a nonlinear network, yet it forms its SS prior based on a linear network to maintain efficiency. A pervasive framework for combining multiple priors in MRI reconstruction is algorithmic unrolling that uses serially alternated projections, causing error propagation under low-data regimes. To alleviate error propagation, PSFNet combines its SS and SG priors via a novel parallel-stream architecture with learnable fusion parameters. Demonstrations are performed on multi-coil brain MRI for varying amounts of training data. PSFNet outperforms SG methods in low-data regimes, and surpasses SS methods with few tens of training samples. On average across tasks, PSFNet achieves 3.1 dB higher PSNR, 2.8% higher SSIM, and 0.3 × lower RMSE than baselines. Furthermore, in both supervised and unsupervised setups, PSFNet requires an order of magnitude lower samples compared to SG methods, and enables an order of magnitude faster inference compared to SS methods. Thus, the proposed model improves deep MRI reconstruction with elevated learning and computational efficiency.

Frequency Learning via Multi-Scale Fourier Transformer for MRI Reconstruction

Since Magnetic Resonance Imaging (MRI) requires a long acquisition time, various methods were proposed to reduce the time, but they ignored the frequency information and non-local similarity, so that they failed to reconstruct images with a clear structure. In this article, we propose Frequency Learning via Multi-scale Fourier Transformer for MRI Reconstruction (FMTNet), which focuses on repairing the low-frequency and high-frequency information. Specifically, FMTNet is composed of a high-frequency learning branch (HFLB) and a low-frequency learning branch (LFLB). Meanwhile, we propose a Multi-scale Fourier Transformer (MFT) as the basic module to learn the non-local information. Unlike normal Transformers, MFT adopts Fourier convolution to replace self-attention to efficiently learn global information. Moreover, we further introduce a multi-scale learning and cross-scale linear fusion strategy in MFT to interact information between features of different scales and strengthen the representation of features. Compared with normal Transformers, the proposed MFT occupies fewer computing resources. Based on MFT, we design a Residual Multi-scale Fourier Transformer module as the main component of HFLB and LFLB. We conduct several experiments under different acceleration rates and different sampling patterns on different datasets, and the experiment results show that our method is superior to the previous state-of-the-art method.

Image Quality and Diagnostic Performance of Accelerated 2D Hip MRI with Deep Learning Reconstruction Based on a Deep Iterative Hierarchical Network

OBJECTIVES

Hip MRI using standard multiplanar sequences requires long scan times. Accelerating MRI is accompanied by reduced image quality. This study aimed to compare standard two-dimensional (2D) turbo spin echo (TSE) sequences with accelerated 2D TSE sequences with deep learning (DL) reconstruction (TSEDL) for routine clinical hip MRI at 1.5 and 3 T in terms of feasibility, image quality, and diagnostic performance.

MATERIAL AND METHODS

In this prospective, monocentric study, TSEDL was implemented clinically and evaluated in 14 prospectively enrolled patients undergoing a clinically indicated hip MRI at 1.5 and 3T between October 2020 and May 2021. Each patient underwent two examinations: For the first exam, we used standard sequences with generalized autocalibrating partial parallel acquisition reconstruction (TSES). For the second exam, we implemented prospectively undersampled TSE sequences with DL reconstruction (TSEDL). Two radiologists assessed the TSEDL and TSES regarding image quality, artifacts, noise, edge sharpness, diagnostic confidence, and delineation of anatomical structures using an ordinal five-point Likert scale (1 = non-diagnostic; 2 = poor; 3 = moderate; 4 = good; 5 = excellent). Both sequences were compared regarding the detection of common pathologies of the hip. Comparative analyses were conducted to assess the differences between TSEDL and TSES.

RESULTS

Compared with TSES, TSEDL was rated to be significantly superior in terms of image quality (p ≤ 0.020) with significantly reduced noise (p ≤ 0.001) and significantly improved edge sharpness (p = 0.003). No difference was found between TSES and TSEDL concerning the extent of artifacts, diagnostic confidence, or the delineation of anatomical structures (p > 0.05). Example acquisition time reductions for the TSE sequences of 52% at 3 Tesla and 70% at 1.5 Tesla were achieved.

CONCLUSION

TSEDL of the hip is clinically feasible, showing excellent image quality and equivalent diagnostic performance compared with TSES, reducing the acquisition time significantly.

CS-MRI Reconstruction Using an Improved GAN with Dilated Residual Networks and Channel Attention Mechanism

Compressed sensing (CS) MRI has shown great potential in enhancing time efficiency. Deep learning techniques, specifically generative adversarial networks (GANs), have emerged as potent tools for speedy CS-MRI reconstruction. Yet, as the complexity of deep learning reconstruction models increases, this can lead to prolonged reconstruction time and challenges in achieving convergence. In this study, we present a novel GAN-based model that delivers superior performance without the model complexity escalating. Our generator module, built on the U-net architecture, incorporates dilated residual (DR) networks, thus expanding the network's receptive field without increasing parameters or computational load. At every step of the downsampling path, this revamped generator module includes a DR network, with the dilation rates adjusted according to the depth of the network layer. Moreover, we have introduced a channel attention mechanism (CAM) to distinguish between channels and reduce background noise, thereby focusing on key information. This mechanism adeptly combines global maximum and average pooling approaches to refine channel attention. We conducted comprehensive experiments with the designed model using public domain MRI datasets of the human brain. Ablation studies affirmed the efficacy of the modified modules within the network. Incorporating DR networks and CAM elevated the peak signal-to-noise ratios (PSNR) of the reconstructed images by about 1.2 and 0.8 dB, respectively, on average, even at 10× CS acceleration. Compared to other relevant models, our proposed model exhibits exceptional performance, achieving not only excellent stability but also outperforming most of the compared networks in terms of PSNR and SSIM. When compared with U-net, DR-CAM-GAN's average gains in SSIM and PSNR were 14% and 15%, respectively. Its MSE was reduced by a factor that ranged from two to seven. The model presents a promising pathway for enhancing the efficiency and quality of CS-MRI reconstruction.

PARCEL: Physics-Based Unsupervised Contrastive Representation Learning for Multi-Coil MR Imaging

With the successful application of deep learning to magnetic resonance (MR) imaging, parallel imaging techniques based on neural networks have attracted wide attention. However, in the absence of high-quality, fully sampled datasets for training, the performance of these methods is limited. And the interpretability of models is not strong enough. To tackle this issue, this paper proposes a Physics-bAsed unsupeRvised Contrastive rEpresentation Learning (PARCEL) method to speed up parallel MR imaging. Specifically, PARCEL has a parallel framework to contrastively learn two branches of model-based unrolling networks from augmented undersampled multi-coil k-space data. A sophisticated co-training loss with three essential components has been designed to guide the two networks in capturing the inherent features and representations for MR images. And the final MR image is reconstructed with the trained contrastive networks. PARCEL was evaluated on two vivo datasets and compared to five state-of-the-art methods. The results show that PARCEL is able to learn essential representations for accurate MR reconstruction without relying on fully sampled datasets. The code will be made available at https://github.com/ternencewu123/PARCEL.

Twinned Residual Auto-Encoder (TRAE)-A new DL architecture for denoising super-resolution and task-aware feature learning from COVID-19 CT images

The detection of the COronaVIrus Disease 2019 (COVID-19) from Computed Tomography (CT) scans has become a very important task in modern medical diagnosis. Unfortunately, typical resolutions of state-of-the-art CT scans are still not adequate for reliable and accurate automatic detection of COVID-19 disease. Motivated by this consideration, in this paper, we propose a novel architecture that jointly affords the Single-Image Super-Resolution (SISR) and the reliable classification problems from Low Resolution (LR) and noisy CT scans. Specifically, the proposed architecture is based on a couple of Twinned Residual Auto-Encoders (TRAE), which exploits the feature vectors and the SR images recovered by a Master AE for performing transfer learning and then improves the training of a "twinned" Follower AE. In addition, we also develop a Task-Aware (TA) version of the basic TRAE architecture, namely the TA-TRAE, which further utilizes the set of feature vectors generated by the Follower AE for the joint training of an additional auxiliary classifier, so to perform automated medical diagnosis on the basis of the available LR input images without human support. Experimental results and comparisons with a number of state-of-the-art CNN/GAN/CycleGAN benchmark SISR architectures, performed by considering × 2 , × 4 , and × 8 super-resolution (i.e., upscaling) factors, support the effectiveness of the proposed TRAE/TA-TRAE architectures. In particular, the detection accuracy attained by the proposed architectures outperforms the corresponding ones of the implemented CNN, GAN and CycleGAN baselines up to 9.0%, 6.5%, and 6.0% at upscaling factors as high as × 8 .

An overview of artificial intelligence in medical physics and radiation oncology

Artificial intelligence (AI) is developing rapidly and has found widespread applications in medicine, especially radiotherapy. This paper provides a brief overview of AI applications in radiotherapy, and highlights the research directions of AI that can potentially make significant impacts and relevant ongoing research works in these directions. Challenging issues related to the clinical applications of AI, such as robustness and interpretability of AI models, are also discussed. The future research directions of AI in the field of medical physics and radiotherapy are highlighted.

Faster Elbow MRI with Deep Learning Reconstruction-Assessment of Image Quality, Diagnostic Confidence, and Anatomy Visualization Compared to Standard Imaging

OBJECTIVE

The objective of this study was to evaluate a deep learning (DL) reconstruction for turbo spin echo (TSE) sequences of the elbow regarding image quality and visualization of anatomy.

MATERIALS AND METHODS

Between October 2020 and June 2021, seventeen participants (eight patients, nine healthy subjects; mean age: 43 ± 16 (20-70) years, eight men) were prospectively included in this study. Each patient underwent two examinations: standard MRI, including TSE sequences reconstructed with a generalized autocalibrating partial parallel acquisition reconstruction (TSESTD), and prospectively undersampled TSE sequences reconstructed with a DL reconstruction (TSEDL). Two radiologists evaluated the images concerning image quality, noise, edge sharpness, artifacts, diagnostic confidence, and delineation of anatomical structures using a 5-point Likert scale, and rated the images concerning the detection of common pathologies.

RESULTS

Image quality was significantly improved in TSEDL (mean 4.35, IQR 4-5) compared to TSESTD (mean 3.76, IQR 3-4, p = 0.008). Moreover, TSEDL showed decreased noise (mean 4.29, IQR 3.5-5) compared to TSESTD (mean 3.35, IQR 3-4, p = 0.004). Ratings for delineation of anatomical structures, artifacts, edge sharpness, and diagnostic confidence did not differ significantly between TSEDL and TSESTD (p > 0.05). Inter-reader agreement was substantial to almost perfect (κ = 0.628-0.904). No difference was found concerning the detection of pathologies between the readers and between TSEDL and TSESTD. Using DL, the acquisition time could be reduced by more than 35% compared to TSESTD.

CONCLUSION

TSEDL provided improved image quality and decreased noise while receiving equal ratings for edge sharpness, artifacts, delineation of anatomical structures, diagnostic confidence, and detection of pathologies compared to TSESTD. Providing more than a 35% reduction of acquisition time, TSEDL may be clinically relevant for elbow imaging due to increased patient comfort and higher patient throughput.

[Physical model-based cascaded generative adversarial networks for accelerating quantitative multi-parametric magnetic resonance imaging]

OBJECTIVE

To explore the feasibility and interpretation of physical model- based cascaded generative adversarial networks for accelerating quantitative multi-echo multi-parametric magnetic resonance imaging using raw multi-echo multicoil k-space data.

METHODS

A physical model-based cascaded generative adversarial network is proposed to enhance image feature information to obtain high-quality reconstructed images using joint training of multi-domain information and learning of key parameters required for image reconstruction through a system matrix and adaptively optimizing the k-space generator and image generator structures. Raw multi-echo multi-coil k-space data are used to accelerate multi-contrast multi-parametric magnetic resonance imaging. A physically driven deep learning reconstruction method is used to increase the generalization capability and improve the model performance by building a system matrix function instead of direct end-to-end training of the model.

RESULTS

In terms of overall image quality, the proposed model achieved significant improvements compared to other methods. On an 80- case test set, the average PSNR value of the reconstructed images was 34.13, SSIM was 0.965, and NRMSE was 0.114. In terms of multi-contrast multi-parametric image reconstruction, the model achieved PSNR values of 38.87 for PDW, 35.62 for T1W, and 34.38 for T2* Map, which were significantly better than those of other methods for quantitative evaluation. The model also produced clearer features of the brain gray matter, white matter, and cerebrospinal fluid. Furthermore, compared with the existing methods with a reconstruction time difference of less than 10%, the proposed method achieved the highest improvement of up to 20% in the metrics of PSNR, SSIM, and NRMSE.

CONCLUSION

Compared with other existing methods, the physical model-based cascaded generative adversarial networks can reconstruct more image details and features, thus improving the quality and accuracy of the reconstructed images.

A Theoretical Framework for Self-Supervised MR Image Reconstruction Using Sub-Sampling via Variable Density Noisier2Noise

In recent years, there has been attention on leveraging the statistical modeling capabilities of neural networks for reconstructing sub-sampled Magnetic Resonance Imaging (MRI) data. Most proposed methods assume the existence of a representative fully-sampled dataset and use fully-supervised training. However, for many applications, fully sampled training data is not available, and may be highly impractical to acquire. The development and understanding of self-supervised methods, which use only sub-sampled data for training, are therefore highly desirable. This work extends the Noisier2Noise framework, which was originally constructed for self-supervised denoising tasks, to variable density sub-sampled MRI data. We use the Noisier2Noise framework to analytically explain the performance of Self-Supervised Learning via Data Undersampling (SSDU), a recently proposed method that performs well in practice but until now lacked theoretical justification. Further, we propose two modifications of SSDU that arise as a consequence of the theoretical developments. Firstly, we propose partitioning the sampling set so that the subsets have the same type of distribution as the original sampling mask. Secondly, we propose a loss weighting that compensates for the sampling and partitioning densities. On the fastMRI dataset we show that these changes significantly improve SSDU's image restoration quality and robustness to the partitioning parameters.