Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction

CYCLE-CONSISTENT SELF-SUPERVISED LEARNING FOR IMPROVED HIGHLY-ACCELERATED MRI RECONSTRUCTION

Physics-driven deep learning (PD-DL) has become a powerful tool for accelerated MRI. Recent developments have also developed unsupervised learning for PD-DL, including self-supervised learning. However, at very high acceleration rates, such approaches show performance deterioration. In this study, we propose to use cyclic-consistency (CC) to improve self-supervised learning for highly accelerated MRI. In our proposed CC, simulated measurements are obtained by undersampling the network output using patterns drawn from the same distribution as the true one. The reconstructions of these simulated measurements are obtained using the same network, which are then compared to the acquired data at the true sampling locations. This CC approach is used in conjunction with a masking-based self-supervised loss. Results show that the proposed method can substantially reduce aliasing artifacts at high acceleration rates, including rate 6 and 8 fastMRI knee imaging and 20-fold HCP-style fMRI.

FastMRI Prostate: A public, biparametric MRI dataset to advance machine learning for prostate cancer imaging

Magnetic resonance imaging (MRI) has experienced remarkable advancements in the integration of artificial intelligence (AI) for image acquisition and reconstruction. The availability of raw k-space data is crucial for training AI models in such tasks, but public MRI datasets are mostly restricted to DICOM images only. To address this limitation, the fastMRI initiative released brain and knee k-space datasets, which have since seen vigorous use. In May 2023, fastMRI was expanded to include biparametric (T2- and diffusion-weighted) prostate MRI data from a clinical population. Biparametric MRI plays a vital role in the diagnosis and management of prostate cancer. Advances in imaging methods, such as reconstructing under-sampled data from accelerated acquisitions, can improve cost-effectiveness and accessibility of prostate MRI. Raw k-space data, reconstructed images and slice, volume and exam level annotations for likelihood of prostate cancer are provided in this dataset for 47468 slices corresponding to 1560 volumes from 312 patients. This dataset facilitates AI and algorithm development for prostate image reconstruction, with the ultimate goal of enhancing prostate cancer diagnosis.

DCT-net: Dual-domain cross-fusion transformer network for MRI reconstruction

Current challenges in Magnetic Resonance Imaging (MRI) include long acquisition times and motion artifacts. To address these issues, under-sampled k-space acquisition has gained popularity as a fast imaging method. However, recovering fine details from under-sampled data remains challenging. In this study, we introduce a pioneering deep learning approach, namely DCT-Net, designed for dual-domain MRI reconstruction. DCT-Net seamlessly integrates information from the image domain (IRM) and frequency domain (FRM), utilizing a novel Cross Attention Block (CAB) and Fusion Attention Block (FAB). These innovative blocks enable precise feature extraction and adaptive fusion across both domains, resulting in a significant enhancement of the reconstructed image quality. The adaptive interaction and fusion mechanisms of CAB and FAB contribute to the method's effectiveness in capturing distinctive features and optimizing image reconstruction. Comprehensive ablation studies have been conducted to assess the contributions of these modules to reconstruction quality and accuracy. Experimental results on the FastMRI (2023) and Calgary-Campinas datasets (2021) demonstrate the superiority of our MRI reconstruction framework over other typical methods (most are illustrated in 2023 or 2022) in both qualitative and quantitative evaluations. This holds for knee and brain datasets under 4× and 8× accelerated imaging scenarios.

UNCERTAINTY-GUIDED PHYSICS-DRIVEN DEEP LEARNING RECONSTRUCTION VIA CYCLIC MEASUREMENT CONSISTENCY

Physics-driven deep learning (PD-DL) techniques have recently emerged as a powerful means for improved computational imaging, including in MRI applications. These methods use the physics information by incorporating the known forward model for data fidelity, while performing regularization using neural networks. There has been substantial progress in the training of PD-DL reconstruction methods, ranging from simple supervised learning to more practical self-supervised learning and generative models that allow training without reference data. Similarly, efforts have been made to characterize the errors associated with PD-DL methods via uncertainty quantification, mostly focusing on generative models. In this work, we devise an uncertainty estimation process that primarily focuses on the data fidelity component of PD-DL by characterizing the cyclic consistency between different forward models. Subsequently, we use this uncertainty estimate to guide the training of the PD-DL method. Results show that the proposed uncertainty-guided PD-DL strategy improves reconstruction quality.

Deep-learning-based reconstruction of undersampled MRI to reduce scan times: a multicentre, retrospective, cohort study

BACKGROUND

The extended acquisition times required for MRI limit its availability in resource-constrained settings. Consequently, accelerating MRI by undersampling k-space data, which is necessary to reconstruct an image, has been a long-standing but important challenge. We aimed to develop a deep convolutional neural network (dCNN) optimisation method for MRI reconstruction and to reduce scan times and evaluate its effect on image quality and accuracy of oncological imaging biomarkers.

METHODS

In this multicentre, retrospective, cohort study, MRI data from patients with glioblastoma treated at Heidelberg University Hospital (775 patients and 775 examinations) and from the phase 2 CORE trial (260 patients, 1083 examinations, and 58 institutions) and the phase 3 CENTRIC trial (505 patients, 3147 examinations, and 139 institutions) were used to develop, train, and test dCNN for reconstructing MRI from highly undersampled single-coil k-space data with various acceleration rates (R=2, 4, 6, 8, 10, and 15). Independent testing was performed with MRIs from the phase 2/3 EORTC-26101 trial (528 patients with glioblastoma, 1974 examinations, and 32 institutions). The similarity between undersampled dCNN-reconstructed and original MRIs was quantified with various image quality metrics, including structural similarity index measure (SSIM) and the accuracy of undersampled dCNN-reconstructed MRI on downstream radiological assessment of imaging biomarkers in oncology (automated artificial intelligence-based quantification of tumour burden and treatment response) was performed in the EORTC-26101 test dataset. The public NYU Langone Health fastMRI brain test dataset (558 patients and 558 examinations) was used to validate the generalisability and robustness of the dCNN for reconstructing MRIs from available multi-coil (parallel imaging) k-space data.

FINDINGS

In the EORTC-26101 test dataset, the median SSIM of undersampled dCNN-reconstructed MRI ranged from 0·88 to 0·99 across different acceleration rates, with 0·92 (95% CI 0·92-0·93) for 10-times acceleration (R=10). The 10-times undersampled dCNN-reconstructed MRI yielded excellent agreement with original MRI when assessing volumes of contrast-enhancing tumour (median DICE for spatial agreement of 0·89 [95% CI 0·88 to 0·89]; median volume difference of 0·01 cm3 [95% CI 0·00 to 0·03] equalling 0·21%; p=0·0036 for equivalence) or non-enhancing tumour or oedema (median DICE of 0·94 [95% CI 0·94 to 0·95]; median volume difference of -0·79 cm3 [95% CI -0·87 to -0·72] equalling -1·77%; p=0·023 for equivalence) in the EORTC-26101 test dataset. Automated volumetric tumour response assessment in the EORTC-26101 test dataset yielded an identical median time to progression of 4·27 months (95% CI 4·14 to 4·57) when using 10-times-undersampled dCNN-reconstructed or original MRI (log-rank p=0·80) and agreement in the time to progression in 374 (95·2%) of 393 patients with data. The dCNN generalised well to the fastMRI brain dataset, with significant improvements in the median SSIM when using multi-coil compared with single-coil k-space data (p<0·0001).

INTERPRETATION

Deep-learning-based reconstruction of undersampled MRI allows for a substantial reduction of scan times, with a 10-times acceleration demonstrating excellent image quality while preserving the accuracy of derived imaging biomarkers for the assessment of oncological treatment response. Our developments are available as open source software and hold considerable promise for increasing the accessibility to MRI, pending further prospective validation.

FUNDING

Deutsche Forschungsgemeinschaft (German Research Foundation) and an Else Kröner Clinician Scientist Endowed Professorship by the Else Kröner Fresenius Foundation.

T2LR-Net: An unrolling network learning transformed tensor low-rank prior for dynamic MR image reconstruction

The tensor low-rank prior has attracted considerable attention in dynamic MR reconstruction. Tensor low-rank methods preserve the inherent high-dimensional structure of data, allowing for improved extraction and utilization of intrinsic low-rank characteristics. However, most current methods are still confined to utilizing low-rank structures either in the image domain or predefined transformed domains. Designing an optimal transformation adaptable to dynamic MRI reconstruction through manual efforts is inherently challenging. In this paper, we propose a deep unrolling network that utilizes the convolutional neural network (CNN) to adaptively learn the transformed domain for leveraging tensor low-rank priors. Under the supervised mechanism, the learning of the tensor low-rank domain is directly guided by the reconstruction accuracy. Specifically, we generalize the traditional t-SVD to a transformed version based on arbitrary high-dimensional unitary transformations and introduce a novel unitary transformed tensor nuclear norm (UTNN). Subsequently, we present a dynamic MRI reconstruction model based on UTNN and devise an efficient iterative optimization algorithm using ADMM, which is finally unfolded into the proposed T2LR-Net. Experiments on two dynamic cardiac MRI datasets demonstrate that T2LR-Net outperforms the state-of-the-art optimization-based and unrolling network-based methods.

A Systematic Review and Identification of the Challenges of Deep Learning Techniques for Undersampled Magnetic Resonance Image Reconstruction

Deep learning (DL) in magnetic resonance imaging (MRI) shows excellent performance in image reconstruction from undersampled k-space data. Artifact-free and high-quality MRI reconstruction is essential for ensuring accurate diagnosis, supporting clinical decision-making, enhancing patient safety, facilitating efficient workflows, and contributing to the validity of research studies and clinical trials. Recently, deep learning has demonstrated several advantages over conventional MRI reconstruction methods. Conventional methods rely on manual feature engineering to capture complex patterns and are usually computationally demanding due to their iterative nature. Conversely, DL methods use neural networks with hundreds of thousands of parameters and automatically learn relevant features and representations directly from the data. Nevertheless, there are some limitations to DL-based techniques concerning MRI reconstruction tasks, such as the need for large, labeled datasets, the possibility of overfitting, and the complexity of model training. Researchers are striving to develop DL models that are more efficient, adaptable, and capable of providing valuable information for medical practitioners. We provide a comprehensive overview of the current developments and clinical uses by focusing on state-of-the-art DL architectures and tools used in MRI reconstruction. This study has three objectives. Our main objective is to describe how various DL designs have changed over time and talk about cutting-edge tactics, including their advantages and disadvantages. Hence, data pre- and post-processing approaches are assessed using publicly available MRI datasets and source codes. Secondly, this work aims to provide an extensive overview of the ongoing research on transformers and deep convolutional neural networks for rapid MRI reconstruction. Thirdly, we discuss several network training strategies, like supervised, unsupervised, transfer learning, and federated learning for rapid and efficient MRI reconstruction. Consequently, this article provides significant resources for future improvement of MRI data pre-processing and fast image reconstruction.

Promoting fast MR imaging pipeline by full-stack AI

Magnetic resonance imaging (MRI) is a widely used imaging modality in clinics for medical disease diagnosis, staging, and follow-up. Deep learning has been extensively used to accelerate k-space data acquisition, enhance MR image reconstruction, and automate tissue segmentation. However, these three tasks are usually treated as independent tasks and optimized for evaluation by radiologists, thus ignoring the strong dependencies among them; this may be suboptimal for downstream intelligent processing. Here, we present a novel paradigm, full-stack learning (FSL), which can simultaneously solve these three tasks by considering the overall imaging process and leverage the strong dependence among them to further improve each task, significantly boosting the efficiency and efficacy of practical MRI workflows. Experimental results obtained on multiple open MR datasets validate the superiority of FSL over existing state-of-the-art methods on each task. FSL has great potential to optimize the practical workflow of MRI for medical diagnosis and radiotherapy.

Intrafraction Motion Management With MR-Guided Radiation Therapy

High quality radiation therapy requires highly accurate and precise dose delivery. MR-guided radiotherapy (MRgRT), integrating an MRI scanner with a linear accelerator, offers excellent quality images in the treatment room without subjecting patient to ionizing radiation. MRgRT therefore provides a powerful tool for intrafraction motion management. This paper summarizes different sources of intrafraction motion for different disease sites and describes the MR imaging techniques available to visualize and quantify intrafraction motion. It provides an overview of MR guided motion management strategies and of the current technical capabilities of the commercially available MRgRT systems. It describes how these motion management capabilities are currently being used in clinical studies, protocols and provides a future outlook.

Unveiling the future of breast cancer assessment: a critical review on generative adversarial networks in elastography ultrasound

Elastography Ultrasound provides elasticity information of the tissues, which is crucial for understanding the density and texture, allowing for the diagnosis of different medical conditions such as fibrosis and cancer. In the current medical imaging scenario, elastograms for B-mode Ultrasound are restricted to well-equipped hospitals, making the modality unavailable for pocket ultrasound. To highlight the recent progress in elastogram synthesis, this article performs a critical review of generative adversarial network (GAN) methodology for elastogram generation from B-mode Ultrasound images. Along with a brief overview of cutting-edge medical image synthesis, the article highlights the contribution of the GAN framework in light of its impact and thoroughly analyzes the results to validate whether the existing challenges have been effectively addressed. Specifically, This article highlights that GANs can successfully generate accurate elastograms for deep-seated breast tumors (without having artifacts) and improve diagnostic effectiveness for pocket US. Furthermore, the results of the GAN framework are thoroughly analyzed by considering the quantitative metrics, visual evaluations, and cancer diagnostic accuracy. Finally, essential unaddressed challenges that lie at the intersection of elastography and GANs are presented, and a few future directions are shared for the elastogram synthesis research.

Deep learning-based reconstruction for acceleration of lumbar spine MRI: a prospective comparison with standard MRI

OBJECTIVE

To compare the image quality and diagnostic performance between standard turbo spin-echo MRI and accelerated MRI with deep learning (DL)-based image reconstruction for degenerative lumbar spine diseases.

MATERIALS AND METHODS

Fifty patients who underwent both the standard and accelerated lumbar MRIs at a 1.5-T scanner for degenerative lumbar spine diseases were prospectively enrolled. DL reconstruction algorithm generated coarse (DL_coarse) and fine (DL_fine) images from the accelerated protocol. Image quality was quantitatively assessed in terms of signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) and qualitatively assessed using five-point visual scoring systems. The sensitivity and specificity of four radiologists for the diagnosis of degenerative diseases in both protocols were compared.

RESULTS

The accelerated protocol reduced the average MRI acquisition time by 32.3% as compared to the standard protocol. As compared with standard images, DL_coarse and DL_fine showed significantly higher SNRs on T1-weighted images (T1WI; both p < .001) and T2-weighted images (T2WI; p = .002 and p < 0.001), higher CNRs on T1WI (both p < 0.001), and similar CNRs on T2WI (p = .49 and p = .27). The average radiologist assessment of overall image quality for DL_coarse and DL_fine was higher on sagittal T1WI (p = .04 and p < .001) and axial T2WI (p = .006 and p = .01) and similar on sagittal T2WI (p = .90 and p = .91). Both DL_coarse and DL_fine had better image quality of cauda equina and paraspinal muscles on axial T2WI (both p = .04 for cauda equina; p = .008 and p = .002 for paraspinal muscles). Differences in sensitivity and specificity for the detection of central canal stenosis and neural foraminal stenosis between standard and DL-reconstructed images were all statistically nonsignificant (p ≥ 0.05).

CONCLUSION

DL-based protocol reduced MRI acquisition time without degrading image quality and diagnostic performance of readers for degenerative lumbar spine diseases.

CLINICAL RELEVANCE STATEMENT

The deep learning (DL)-based reconstruction algorithm may be used to further accelerate spine MRI imaging to reduce patient discomfort and increase the cost efficiency of spine MRI imaging.

KEY POINTS

• By using deep learning (DL)-based reconstruction algorithm in combination with the accelerated MRI protocol, the average acquisition time was reduced by 32.3% as compared with the standard protocol. • DL-reconstructed images had similar or better quantitative/qualitative overall image quality and similar or better image quality for the delineation of most individual anatomical structures. • The average radiologist's sensitivity and specificity for the detection of major degenerative lumbar spine diseases, including central canal stenosis, neural foraminal stenosis, and disc herniation, on standard and DL-reconstructed images, were similar.

End-to-end memory-efficient reconstruction for cone beam CT

BACKGROUND

Cone beam computed tomography (CBCT) plays an important role in many medical fields nowadays. Unfortunately, the potential of this imaging modality is hampered by lower image quality compared to the conventional CT, and producing accurate reconstructions remains challenging. A lot of recent research has been directed towards reconstruction methods relying on deep learning, which have shown great promise for various imaging modalities. However, practical application of deep learning to CBCT reconstruction is complicated by several issues, such as exceedingly high memory costs of deep learning methods when working with fully 3D data. Additionally, deep learning methods proposed in the literature are often trained and evaluated only on data from a specific region of interest, thus raising concerns about possible lack of generalization to other regions.

PURPOSE

In this work, we aim to address these limitations and propose LIRE: a learned invertible primal-dual iterative scheme for CBCT reconstruction.

METHODS

LIRE is a learned invertible primal-dual iterative scheme for CBCT reconstruction, wherein we employ a U-Net architecture in each primal block and a residual convolutional neural network (CNN) architecture in each dual block. Memory requirements of the network are substantially reduced while preserving its expressive power through a combination of invertible residual primal-dual blocks and patch-wise computations inside each of the blocks during both forward and backward pass. These techniques enable us to train on data with isotropic 2 mm voxel spacing, clinically-relevant projection count and detector panel resolution on current hardware with 24 GB video random access memory (VRAM).

RESULTS

Two LIRE models for small and for large field-of-view (FoV) setting were trained and validated on a set of 260 + 22 thorax CT scans and tested using a set of 142 thorax CT scans plus an out-of-distribution dataset of 79 head and neck CT scans. For both settings, our method surpasses the classical methods and the deep learning baselines on both test sets. On the thorax CT set, our method achieves peak signal-to-noise ratio (PSNR) of 33.84 ± 2.28 for the small FoV setting and 35.14 ± 2.69 for the large FoV setting; U-Net baseline achieves PSNR of 33.08 ± 1.75 and 34.29 ± 2.71 respectively. On the head and neck CT set, our method achieves PSNR of 39.35 ± 1.75 for the small FoV setting and 41.21 ± 1.41 for the large FoV setting; U-Net baseline achieves PSNR of 33.08 ± 1.75 and 34.29 ± 2.71 respectively. Additionally, we demonstrate that LIRE can be finetuned to reconstruct high-resolution CBCT data with the same geometry but 1 mm voxel spacing and higher detector panel resolution, where it outperforms the U-Net baseline as well.

CONCLUSIONS

Learned invertible primal-dual schemes with additional memory optimizations can be trained to reconstruct CBCT volumes directly from the projection data with clinically-relevant geometry and resolution. Such methods can offer better reconstruction quality and generalization compared to classical deep learning baselines.

Data-efficient Bayesian learning for radial dynamic MR reconstruction

BACKGROUND

Cardiac MRI has become the gold-standard imaging technique for assessing cardiovascular morphology and function. In spite of this, its slow data acquisition process presents imaging challenges due to the motion from heartbeats, respiration, and blood flow. In recent studies, deep learning (DL) algorithms have shown promising results for the task of image reconstruction. However, there have been instances where they have introduced artifacts that may be misinterpreted as pathologies or may obscure the detection of pathologies. Therefore, it is important to obtain a metric, such as the uncertainty of the network output, that identifies such artifacts. However, this can be quite challenging for large-scale image reconstruction problems such as dynamic multi-coil non-Cartesian MRI.

PURPOSE

To efficiently quantify uncertainties of a physics-informed DL-based image reconstruction method for a large-scale accelerated 2D multi-coil dynamic radial MRI reconstruction problem, and demonstrate the benefits of physics-informed DL over model-agnostic DL in reducing uncertainties while at the same time improving image quality.

METHODS

We extended a recently proposed physics-informed 2D U-Net that learns spatio-temporal slices (named XT-YT U-Net), and employed it for the task of uncertainty quantification (UQ) by using Monte Carlo dropout and a Gaussian negative log-likelihood loss function. Our data comprised 2D dynamic MR images acquired with a radial balanced steady-state free precession sequence. The XT-YT U-Net, which allows for training with a limited amount of data, was trained and validated on a dataset of 15 healthy volunteers, and further tested on data from four patients. An extensive comparison between physics-informed and model-agnostic neural networks (NNs) concerning the obtained image quality and uncertainty estimates was performed. Further, we employed calibration plots to assess the quality of the UQ.

RESULTS

The inclusion of the MR-physics model of data acquisition as a building block in the NN architecture led to higher image quality (NRMSE:- 33 ± 8.2 % $-33 \pm 8.2 \%$ , PSNR:6.3 ± 1.3 % $6.3 \pm 1.3 \%$ , and SSIM:1.9 ± 0.96 % $1.9 \pm 0.96 \%$ ), lower uncertainties (- 46 ± 8.7 % $-46 \pm 8.7 \%$ ), and, based on the calibration plots, an improved UQ compared to its model-agnostic counterpart. Furthermore, the UQ information can be used to differentiate between anatomical structures (e.g., coronary arteries, ventricle boundaries) and artifacts.

CONCLUSIONS

Using an XT-YT U-Net, we were able to quantify uncertainties of a physics-informed NN for a high-dimensional and computationally demanding 2D multi-coil dynamic MR imaging problem. In addition to improving the image quality, embedding the acquisition model in the network architecture decreased the reconstruction uncertainties as well as quantitatively improved the UQ. The UQ provides additional information to assess the performance of different network approaches.

Clinical feasibility of deep learning reconstruction in liver diffusion-weighted imaging: Improvement of image quality and impact on apparent diffusion coefficient value

PURPOSE

Diffusion-weighted imaging (DWI) of the liver suffers from low resolution, noise, and artifacts. This study aimed to investigate the effect of deep learning reconstruction (DLR) on image quality and apparent diffusion coefficient (ADC) quantification of liver DWI at 3 Tesla.

METHOD

In this prospective study, images of the liver obtained at DWI with b-values of 0 (DWI0), 50 (DWI50) and 800 s/mm2 (DWI800) from consecutive patients with liver lesions from February 2022 to February 2023 were reconstructed with and without DLR (non-DLR). Image quality was assessed qualitatively using Likert scoring system and quantitatively using signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR) and liver/parenchyma boundary sharpness from region-of-interest (ROI) analysis. ADC value of lesion were measured. Phantom experiment was also performed to investigate the factors that determine the effect of DLR on ADC value. Qualitative score, SNR, CNR, boundary sharpness, and apparent diffusion coefficients (ADCs) for DWI were compared using paired t-test and Wilcoxon signed rank test. P < 0.05 was considered statistically significant.

RESULTS

A total of 85 patients with 170 lesions were included. DLR group showed a higher qualitative score than the non-DLR group. for example, with DWI800 the score was 4.77 ± 0.52 versus 4.30 ± 0.63 (P < 0.001). DLR group also showed higher SNRs, CNRs and boundary sharpness than the non-DLR group. DLR reduced the ADC of malignant tumors (1.105[0.904, 1.340] versus 1.114[0.904, 1.320]) (P < 0.001), but there was no significant difference in the diagnostic value of malignancy for DLR and non-DLR groups (P = 57.3). The phantom study confirmed a reduction of ADC in images with low resolution, and a stronger reduction of ADC in heterogeneous structures than in homogeneous ones (P < 0.001).

CONCLUSIONS

DLR improved image quality of liver DWI. DLR reduced the ADC value of lesions, but did not affect the diagnostic performance of ADC in distinguishing malignant tumors on a 3.0-T MRI system.

Evaluation of a Deep Learning Reconstruction for High-Quality T2-Weighted Breast Magnetic Resonance Imaging

Deep learning (DL) reconstruction techniques to improve MR image quality are becoming commercially available with the hope that they will be applicable to multiple imaging application sites and acquisition protocols. However, before clinical implementation, these methods must be validated for specific use cases. In this work, the quality of standard-of-care (SOC) T2w and a high-spatial-resolution (HR) imaging of the breast were assessed both with and without prototype DL reconstruction. Studies were performed using data collected from phantoms, 20 retrospectively collected SOC patient exams, and 56 prospectively acquired SOC and HR patient exams. Image quality was quantitatively assessed via signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), and edge sharpness. Qualitatively, all in vivo images were scored by either two or four radiologist readers using 5-point Likert scales in the following categories: artifacts, perceived sharpness, perceived SNR, and overall quality. Differences in reader scores were tested for significance. Reader preference and perception of signal intensity changes were also assessed. Application of the DL resulted in higher average SNR (1.2-2.8 times), CNR (1.0-1.8 times), and image sharpness (1.2-1.7 times). Qualitatively, the SOC acquisition with DL resulted in significantly improved image quality scores in all categories compared to non-DL images. HR acquisition with DL significantly increased SNR, sharpness, and overall quality compared to both the non-DL SOC and the non-DL HR images. The acquisition time for the HR data only required a 20% increase compared to the SOC acquisition and readers typically preferred DL images over non-DL counterparts. Overall, the DL reconstruction demonstrated improved T2w image quality in clinical breast MRI.

Complexities of deep learning-based undersampled MR image reconstruction

Artificial intelligence has opened a new path of innovation in magnetic resonance (MR) image reconstruction of undersampled k-space acquisitions. This review offers readers an analysis of the current deep learning-based MR image reconstruction methods. The literature in this field shows exponential growth, both in volume and complexity, as the capabilities of machine learning in solving inverse problems such as image reconstruction are explored. We review the latest developments, aiming to assist researchers and radiologists who are developing new methods or seeking to provide valuable feedback. We shed light on key concepts by exploring the technical intricacies of MR image reconstruction, highlighting the importance of raw datasets and the difficulty of evaluating diagnostic value using standard metrics.Relevance statement Increasingly complex algorithms output reconstructed images that are difficult to assess for robustness and diagnostic quality, necessitating high-quality datasets and collaboration with radiologists.Key points• Deep learning-based image reconstruction algorithms are increasing both in complexity and performance.• The evaluation of reconstructed images may mistake perceived image quality for diagnostic value.• Collaboration with radiologists is crucial for advancing deep learning technology.

Improving MR image quality with a multi-task model, using convolutional losses

PURPOSE

During the acquisition of MRI data, patient-, sequence-, or hardware-related factors can introduce artefacts that degrade image quality. Four of the most significant tasks for improving MRI image quality have been bias field correction, super-resolution, motion-, and noise correction. Machine learning has achieved outstanding results in improving MR image quality for these tasks individually, yet multi-task methods are rarely explored.

METHODS

In this study, we developed a model to simultaneously correct for all four aforementioned artefacts using multi-task learning. Two different datasets were collected, one consisting of brain scans while the other pelvic scans, which were used to train separate models, implementing their corresponding artefact augmentations. Additionally, we explored a novel loss function that does not only aim to reconstruct the individual pixel values, but also the image gradients, to produce sharper, more realistic results. The difference between the evaluated methods was tested for significance using a Friedman test of equivalence followed by a Nemenyi post-hoc test.

RESULTS

Our proposed model generally outperformed other commonly-used correction methods for individual artefacts, consistently achieving equal or superior results in at least one of the evaluation metrics. For images with multiple simultaneous artefacts, we show that the performance of using a combination of models, trained to correct individual artefacts depends heavily on the order that they were applied. This is not an issue for our proposed multi-task model. The model trained using our novel convolutional loss function always outperformed the model trained with a mean squared error loss, when evaluated using Visual Information Fidelity, a quality metric connected to perceptual quality.

CONCLUSION

We trained two models for multi-task MRI artefact correction of brain, and pelvic scans. We used a novel loss function that significantly improves the image quality of the outputs over using mean squared error. The approach performs well on real world data, and it provides insight into which artefacts it detects and corrects for. Our proposed model and source code were made publicly available.

AI for PET image reconstruction

Image reconstruction for positron emission tomography (PET) has been developed over many decades, with advances coming from improved modelling of the data statistics and improved modelling of the imaging physics. However, high noise and limited spatial resolution have remained issues in PET imaging, and state-of-the-art PET reconstruction has started to exploit other medical imaging modalities (such as MRI) to assist in noise reduction and enhancement of PET's spatial resolution. Nonetheless, there is an ongoing drive towards not only improving image quality, but also reducing the injected radiation dose and reducing scanning times. While the arrival of new PET scanners (such as total body PET) is helping, there is always a need to improve reconstructed image quality due to the time and count limited imaging conditions. Artificial intelligence (AI) methods are now at the frontier of research for PET image reconstruction. While AI can learn the imaging physics as well as the noise in the data (when given sufficient examples), one of the most common uses of AI arises from exploiting databases of high-quality reference examples, to provide advanced noise compensation and resolution recovery. There are three main AI reconstruction approaches: (i) direct data-driven AI methods which rely on supervised learning from reference data, (ii) iterative (unrolled) methods which combine our physics and statistical models with AI learning from data, and (iii) methods which exploit AI with our known models, but crucially can offer benefits even in the absence of any example training data whatsoever. This article reviews these methods, considering opportunities and challenges of AI for PET reconstruction.

Improved characterization of lenticulostriate arteries using compressed sensing time-of-flight at 7T

OBJECTIVES

To evaluate the feasibility of 0.2-mm isotropic lenticulostriate arteries (LSAs) imaging using compressed sensing time-of-flight (CS TOF) at around 10 min on 7T, and compare the delineation and characterization of LSAs using conventional TOF and CS TOF.

METHODS

Thirty healthy volunteers were examined with CS TOF and conventional TOF at 7T for around 10 min each. CS TOF was optimized to achieve 0.2-mm isotropic LSA imaging. The numbers of LSA stems and branches were counted and compared on a vascular skeleton. The length and distance were measured and compared on the most prominent branch in each hemisphere. Another patient with intracranial artery stenosis was studied to compare LSA delineation in CS TOF and digital subtraction angiography (DSA).

RESULTS

The number of stems visualized with CS TOF was significantly higher than with conventional TOF in both left (p = 0.002, ICC = 0.884) and right (p < 0.001, ICC = 0.938) hemispheres. The number of branches visualized by conventional TOF was significantly lower than that by CS TOF in both left (p < 0.001, ICC = 0.893) and right (p < 0.001, ICC = 0.896) hemispheres. The lengths were statistically higher in CS TOF than in conventional TOF (left: p < 0.001, ICC = 0.868; right: p < 0.001, ICC = 0.876).

CONCLUSIONS

The high-resolution CS TOF improves the delineation and characterization of LSAs over conventional TOF. High-resolution LSA imaging using CS TOF can be a promising tool for clinical research and applications in patients with neurologic diseases.

KEY POINTS

• 0.2-mm isotropic LSA imaging for around 10 min using CS TOF at 7T is feasible. • More stems and branches of LSAs with longer lengths can be delineated with CS TOF than with conventional TOF at the same scan time. • High-resolution CS TOF can be a promising tool for research and applications on LSA.

Keeping Patient Data Secure in the Age of Radiology Artificial Intelligence: Cybersecurity Considerations and Future Directions

Artificial intelligence (AI)-based solutions are increasingly being incorporated into radiology workflows. Implementation of AI comes along with cybersecurity risks and challenges that practices should be aware of and mitigate for a successful and secure deployment. In this article, these cybersecurity issues are examined through the lens of the "CIA" triad framework-confidentiality, integrity, and availability. We discuss the implications of implementation configurations and development approaches on data security and confidentiality and the potential impact that the insertion of AI can have on the truthfulness of data, access to data, and the cybersecurity attack surface. Finally, we provide a checklist to address important security considerations before deployment of an AI application, and discuss future advances in AI addressing some of these security concerns.

Bayesian reconstruction of magnetic resonance images using Gaussian processes

A central goal of modern magnetic resonance imaging (MRI) is to reduce the time required to produce high-quality images. Efforts have included hardware and software innovations such as parallel imaging, compressed sensing, and deep learning-based reconstruction. Here, we propose and demonstrate a Bayesian method to build statistical libraries of magnetic resonance (MR) images in k-space and use these libraries to identify optimal subsampling paths and reconstruction processes. Specifically, we compute a multivariate normal distribution based upon Gaussian processes using a publicly available library of T1-weighted images of healthy brains. We combine this library with physics-informed envelope functions to only retain meaningful correlations in k-space. This covariance function is then used to select a series of ring-shaped subsampling paths using Bayesian optimization such that they optimally explore space while remaining practically realizable in commercial MRI systems. Combining optimized subsampling paths found for a range of images, we compute a generalized sampling path that, when used for novel images, produces superlative structural similarity and error in comparison to previously reported reconstruction processes (i.e. 96.3% structural similarity and < 0.003 normalized mean squared error from sampling only 12.5% of the k-space data). Finally, we use this reconstruction process on pathological data without retraining to show that reconstructed images are clinically useful for stroke identification. Since the model trained on images of healthy brains could be directly used for predictions in pathological brains without retraining, it shows the inherent transferability of this approach and opens doors to its widespread use.

Towards clinical applicability and computational efficiency in automatic cranial implant design: An overview of the AutoImplant 2021 cranial implant design challenge

Cranial implants are commonly used for surgical repair of craniectomy-induced skull defects. These implants are usually generated offline and may require days to weeks to be available. An automated implant design process combined with onsite manufacturing facilities can guarantee immediate implant availability and avoid secondary intervention. To address this need, the AutoImplant II challenge was organized in conjunction with MICCAI 2021, catering for the unmet clinical and computational requirements of automatic cranial implant design. The first edition of AutoImplant (AutoImplant I, 2020) demonstrated the general capabilities and effectiveness of data-driven approaches, including deep learning, for a skull shape completion task on synthetic defects. The second AutoImplant challenge (i.e., AutoImplant II, 2021) built upon the first by adding real clinical craniectomy cases as well as additional synthetic imaging data. The AutoImplant II challenge consisted of three tracks. Tracks 1 and 3 used skull images with synthetic defects to evaluate the ability of submitted approaches to generate implants that recreate the original skull shape. Track 3 consisted of the data from the first challenge (i.e., 100 cases for training, and 110 for evaluation), and Track 1 provided 570 training and 100 validation cases aimed at evaluating skull shape completion algorithms at diverse defect patterns. Track 2 also made progress over the first challenge by providing 11 clinically defective skulls and evaluating the submitted implant designs on these clinical cases. The submitted designs were evaluated quantitatively against imaging data from post-craniectomy as well as by an experienced neurosurgeon. Submissions to these challenge tasks made substantial progress in addressing issues such as generalizability, computational efficiency, data augmentation, and implant refinement. This paper serves as a comprehensive summary and comparison of the submissions to the AutoImplant II challenge. Codes and models are available at https://github.com/Jianningli/Autoimplant_II.

Stochastic optimization of three-dimensional non-Cartesian sampling trajectory

PURPOSE

Optimizing three-dimensional (3D) k-space sampling trajectories is important for efficient MRI yet presents a challenging computational problem. This work proposes a generalized framework for optimizing 3D non-Cartesian sampling patterns via data-driven optimization.

METHODS

We built a differentiable simulation model to enable gradient-based methods for sampling trajectory optimization. The algorithm can simultaneously optimize multiple properties of sampling patterns, including image quality, hardware constraints (maximum slew rate and gradient strength), reduced peripheral nerve stimulation (PNS), and parameter-weighted contrast. The proposed method can either optimize the gradient waveform (spline-based freeform optimization) or optimize properties of given sampling trajectories (such as the rotation angle of radial trajectories). Notably, the method can optimize sampling trajectories synergistically with either model-based or learning-based reconstruction methods. We proposed several strategies to alleviate the severe nonconvexity and huge computation demand posed by the large scale. The corresponding code is available as an open-source toolbox.

RESULTS

We applied the optimized trajectory to multiple applications including structural and functional imaging. In the simulation studies, the image quality of a 3D kooshball trajectory was improved from 0.29 to 0.22 (NRMSE) with Stochastic optimization framework for 3D NOn-Cartesian samPling trajectorY (SNOPY) optimization. In the prospective studies, by optimizing the rotation angles of a stack-of-stars (SOS) trajectory, SNOPY reduced the NRMSE of reconstructed images from 1.19 to 0.97 compared to the best empirical method (RSOS-GR). Optimizing the gradient waveform of a rotational EPI trajectory improved participants' rating of the PNS from "strong" to "mild."

CONCLUSION

SNOPY provides an efficient data-driven and optimization-based method to tailor non-Cartesian sampling trajectories.

Generalizable Deep Learning Method for Suppressing Unseen and Multiple MRI Artifacts Using Meta-learning

Magnetic Resonance (MR) images suffer from various types of artifacts due to motion, spatial resolution, and under-sampling. Conventional deep learning methods deal with removing a specific type of artifact, leading to separately trained models for each artifact type that lack the shared knowledge generalizable across artifacts. Moreover, training a model for each type and amount of artifact is a tedious process that consumes more training time and storage of models. On the other hand, the shared knowledge learned by jointly training the model on multiple artifacts might be inadequate to generalize under deviations in the types and amounts of artifacts. Model-agnostic meta-learning (MAML), a nested bi-level optimization framework is a promising technique to learn common knowledge across artifacts in the outer level of optimization, and artifact-specific restoration in the inner level. We propose curriculum-MAML (CMAML), a learning process that integrates MAML with curriculum learning to impart the knowledge of variable artifact complexity to adaptively learn restoration of multiple artifacts during training. Comparative studies against Stochastic Gradient Descent and MAML, using two cardiac datasets reveal that CMAML exhibits (i) better generalization with improved PSNR for 83% of unseen types and amounts of artifacts and improved SSIM in all cases, and (ii) better artifact suppression in 4 out of 5 cases of composite artifacts (scans with multiple artifacts).Clinical relevance- Our results show that CMAML has the potential to minimize the number of artifact-specific models; which is essential to deploy deep learning models for clinical use. Furthermore, we have also taken another practical scenario of an image affected by multiple artifacts and show that our method performs better in 80% of cases.

A Hybrid Residual Attention Convolutional Neural Network for Compressed Sensing Magnetic Resonance Image Reconstruction

We propose a dual-domain deep learning technique for accelerating compressed sensing magnetic resonance image reconstruction. An advanced convolutional neural network with residual connectivity and an attention mechanism was developed for frequency and image domains. First, the sensor domain subnetwork estimates the unmeasured frequencies of k-space to reduce aliasing artifacts. Second, the image domain subnetwork performs a pixel-wise operation to remove blur and noisy artifacts. The skip connections efficiently concatenate the feature maps to alleviate the vanishing gradient problem. An attention gate in each decoder layer enhances network generalizability and speeds up image reconstruction by eliminating irrelevant activations. The proposed technique reconstructs real-valued clinical images from sparsely sampled k-spaces that are identical to the reference images. The performance of this novel approach was compared with state-of-the-art direct mapping, single-domain, and multi-domain methods. With acceleration factors (AFs) of 4 and 5, our method improved the mean peak signal-to-noise ratio (PSNR) to 8.67 and 9.23, respectively, compared with the single-domain Unet model; similarly, our approach increased the average PSNR to 3.72 and 4.61, respectively, compared with the multi-domain W-net. Remarkably, using an AF of 6, it enhanced the PSNR by 9.87 ± 1.55 and 6.60 ± 0.38 compared with Unet and W-net, respectively.

Federated End-to-End Unrolled Models for Magnetic Resonance Image Reconstruction

Image reconstruction is the process of recovering an image from raw, under-sampled signal measurements, and is a critical step in diagnostic medical imaging, such as magnetic resonance imaging (MRI). Recently, data-driven methods have led to improved image quality in MRI reconstruction using a limited number of measurements, but these methods typically rely on the existence of a large, centralized database of fully sampled scans for training. In this work, we investigate federated learning for MRI reconstruction using end-to-end unrolled deep learning models as a means of training global models across multiple clients (data sites), while keeping individual scans local. We empirically identify a low-data regime across a large number of heterogeneous scans, where a small number of training samples per client are available and non-collaborative models lead to performance drops. In this regime, we investigate the performance of adaptive federated optimization algorithms as a function of client data distribution and communication budget. Experimental results show that adaptive optimization algorithms are well suited for the federated learning of unrolled models, even in a limited-data regime (50 slices per data site), and that client-sided personalization can improve reconstruction quality for clients that did not participate in training.

Synthesizing Complex-Valued Multicoil MRI Data from Magnitude-Only Images

Despite the proliferation of deep learning techniques for accelerated MRI acquisition and enhanced image reconstruction, the construction of large and diverse MRI datasets continues to pose a barrier to effective clinical translation of these technologies. One major challenge is in collecting the MRI raw data (required for image reconstruction) from clinical scanning, as only magnitude images are typically saved and used for clinical assessment and diagnosis. The image phase and multi-channel RF coil information are not retained when magnitude-only images are saved in clinical imaging archives. Additionally, preprocessing used for data in clinical imaging can lead to biased results. While several groups have begun concerted efforts to collect large amounts of MRI raw data, current databases are limited in the diversity of anatomy, pathology, annotations, and acquisition types they contain. To address this, we present a method for synthesizing realistic MR data from magnitude-only data, allowing for the use of diverse data from clinical imaging archives in advanced MRI reconstruction development. Our method uses a conditional GAN-based framework to generate synthetic phase images from input magnitude images. We then applied ESPIRiT to derive RF coil sensitivity maps from fully sampled real data to generate multi-coil data. The synthetic data generation method was evaluated by comparing image reconstruction results from training Variational Networks either with real data or synthetic data. We demonstrate that the Variational Network trained on synthetic MRI data from our method, consisting of GAN-derived synthetic phase and multi-coil information, outperformed Variational Networks trained on data with synthetic phase generated using current state-of-the-art methods. Additionally, we demonstrate that the Variational Networks trained with synthetic k-space data from our method perform comparably to image reconstruction networks trained on undersampled real k-space data.

Bayesian MRI reconstruction with joint uncertainty estimation using diffusion models

PURPOSE

We introduce a framework that enables efficient sampling from learned probability distributions for MRI reconstruction.

METHOD

Samples are drawn from the posterior distribution given the measured k-space using the Markov chain Monte Carlo (MCMC) method, different from conventional deep learning-based MRI reconstruction techniques. In addition to the maximum a posteriori estimate for the image, which can be obtained by maximizing the log-likelihood indirectly or directly, the minimum mean square error estimate and uncertainty maps can also be computed from those drawn samples. The data-driven Markov chains are constructed with the score-based generative model learned from a given image database and are independent of the forward operator that is used to model the k-space measurement.

RESULTS

We numerically investigate the framework from these perspectives: (1) the interpretation of the uncertainty of the image reconstructed from undersampled k-space; (2) the effect of the number of noise scales used to train the generative models; (3) using a burn-in phase in MCMC sampling to reduce computation; (4) the comparison to conventional ℓ 1 $$ {\ell}_1 $$ -wavelet regularized reconstruction; (5) the transferability of learned information; and (6) the comparison to fastMRI challenge.

CONCLUSION

A framework is described that connects the diffusion process and advanced generative models with Markov chains. We demonstrate its flexibility in terms of contrasts and sampling patterns using advanced generative priors and the benefits of also quantifying the uncertainty for every pixel.

Deep Learning-Based Reconstruction for Cardiac MRI: A Review

Cardiac magnetic resonance (CMR) is an essential clinical tool for the assessment of cardiovascular disease. Deep learning (DL) has recently revolutionized the field through image reconstruction techniques that allow unprecedented data undersampling rates. These fast acquisitions have the potential to considerably impact the diagnosis and treatment of cardiovascular disease. Herein, we provide a comprehensive review of DL-based reconstruction methods for CMR. We place special emphasis on state-of-the-art unrolled networks, which are heavily based on a conventional image reconstruction framework. We review the main DL-based methods and connect them to the relevant conventional reconstruction theory. Next, we review several methods developed to tackle specific challenges that arise from the characteristics of CMR data. Then, we focus on DL-based methods developed for specific CMR applications, including flow imaging, late gadolinium enhancement, and quantitative tissue characterization. Finally, we discuss the pitfalls and future outlook of DL-based reconstructions in CMR, focusing on the robustness, interpretability, clinical deployment, and potential for new methods.

Connections Between Deep Equilibrium and Sparse Representation Models With Application to Hyperspectral Image Denoising

In this study, the problem of computing a sparse representation of multi-dimensional visual data is considered. In general, such data e.g., hyperspectral images, color images or video data consists of signals that exhibit strong local dependencies. A new computationally efficient sparse coding optimization problem is derived by employing regularization terms that are adapted to the properties of the signals of interest. Exploiting the merits of the learnable regularization techniques, a neural network is employed to act as structure prior and reveal the underlying signal dependencies. To solve the optimization problem Deep unrolling and Deep equilibrium based algorithms are developed, forming highly interpretable and concise deep-learning-based architectures, that process the input dataset in a block-by-block fashion. Extensive simulation results, in the context of hyperspectral image denoising, are provided, which demonstrate that the proposed algorithms outperform significantly other sparse coding approaches and exhibit superior performance against recent state-of-the-art deep-learning-based denoising models. In a wider perspective, our work provides a unique bridge between a classic approach, that is the sparse representation theory, and modern representation tools that are based on deep learning modeling.

Deep Learning Denoising of Low-Dose Computed Tomography Chest Images: A Quantitative and Qualitative Image Analysis

PURPOSE

To assess deep learning denoised (DLD) computed tomography (CT) chest images at various low doses by both quantitative and qualitative perceptual image analysis.

METHODS

Simulated noise was inserted into sinogram data from 32 chest CTs acquired at 100 mAs, generating anatomically registered images at 40, 20, 10, and 5 mAs. A DLD model was developed, with 23 scans selected for training, 5 for validation, and 4 for test.Quantitative analysis of perceptual image quality was assessed with Structural SIMilarity Index (SSIM) and Fréchet Inception Distance (FID). Four thoracic radiologists graded overall diagnostic image quality, image artifact, visibility of small structures, and lesion conspicuity. Noise-simulated and denoised image series were evaluated in comparison with one another, and in comparison with standard 100 mAs acquisition at the 4 mAs levels. Statistical tests were conducted at the 2-sided 5% significance level, with multiple comparison correction.

RESULTS

At the same mAs levels, SSIM and FID between noise-simulated and reconstructed DLD images indicated that images were closer to a perfect match with increasing mAs (closer to 1 for SSIM, and 0 for FID).In comparing noise-simulated and DLD images to standard-dose 100-mAs images, DLD improved SSIM and FID. Deep learning denoising improved SSIM of 40-, 20-, 10-, and 5-mAs simulations in comparison with standard-dose 100-mAs images, with change in SSIM from 0.91 to 0.94, 0.87 to 0.93, 0.67 to 0.87, and 0.54 to 0.84, respectively. Deep learning denoising improved FID of 40-, 20-, 10-, and 5-mAs simulations in comparison with standard-dose 100-mAs images, with change in FID from 20 to 13, 46 to 21, 104 to 41, and 148 to 69, respectively.Qualitative image analysis showed no significant difference in lesion conspicuity between DLD images at any mAs in comparison with 100-mAs images. Deep learning denoising images at 10 and 5 mAs were rated lower for overall diagnostic image quality ( P < 0.001), and at 5 mAs lower for overall image artifact and visibility of small structures ( P = 0.002), in comparison with 100 mAs.

CONCLUSIONS

Deep learning denoising resulted in quantitative improvements in image quality. Qualitative assessment demonstrated DLD images at or less than 10 mAs to be rated inferior to standard-dose images.

Calibrationless reconstruction of uniformly‐undersampled multi‐channel MR data with deep learning estimated ESPIRiT maps

PURPOSE

To develop a truly calibrationless reconstruction method that derives An Eigenvalue Approach to Autocalibrating Parallel MRI (ESPIRiT) maps from uniformly-undersampled multi-channel MR data by deep learning.

METHODS

ESPIRiT, one commonly used parallel imaging reconstruction technique, forms the images from undersampled MR k-space data using ESPIRiT maps that effectively represents coil sensitivity information. Accurate ESPIRiT map estimation requires quality coil sensitivity calibration or autocalibration data. We present a U-Net based deep learning model to estimate the multi-channel ESPIRiT maps directly from uniformly-undersampled multi-channel multi-slice MR data. The model is trained using fully-sampled multi-slice axial brain datasets from the same MR receiving coil system. To utilize subject-coil geometric parameters available for each dataset, the training imposes a hybrid loss on ESPIRiT maps at the original locations as well as their corresponding locations within the standard reference multi-slice axial stack. The performance of the approach was evaluated using publicly available T1-weighed brain and cardiac data.

RESULTS

The proposed model robustly predicted multi-channel ESPIRiT maps from uniformly-undersampled k-space data. They were highly comparable to the reference ESPIRiT maps directly computed from 24 consecutive central k-space lines. Further, they led to excellent ESPIRiT reconstruction performance even at high acceleration, exhibiting a similar level of errors and artifacts to that by using reference ESPIRiT maps.

CONCLUSION

A new deep learning approach is developed to estimate ESPIRiT maps directly from uniformly-undersampled MR data. It presents a general strategy for calibrationless parallel imaging reconstruction through learning from the coil and protocol-specific data.

K2S Challenge: From Undersampled K-Space to Automatic Segmentation

Magnetic Resonance Imaging (MRI) offers strong soft tissue contrast but suffers from long acquisition times and requires tedious annotation from radiologists. Traditionally, these challenges have been addressed separately with reconstruction and image analysis algorithms. To see if performance could be improved by treating both as end-to-end, we hosted the K2S challenge, in which challenge participants segmented knee bones and cartilage from 8× undersampled k-space. We curated the 300-patient K2S dataset of multicoil raw k-space and radiologist quality-checked segmentations. 87 teams registered for the challenge and there were 12 submissions, varying in methodologies from serial reconstruction and segmentation to end-to-end networks to another that eschewed a reconstruction algorithm altogether. Four teams produced strong submissions, with the winner having a weighted Dice Similarity Coefficient of 0.910 ± 0.021 across knee bones and cartilage. Interestingly, there was no correlation between reconstruction and segmentation metrics. Further analysis showed the top four submissions were suitable for downstream biomarker analysis, largely preserving cartilage thicknesses and key bone shape features with respect to ground truth. K2S thus showed the value in considering reconstruction and image analysis as end-to-end tasks, as this leaves room for optimization while more realistically reflecting the long-term use case of tools being developed by the MR community.

Virtual elastography ultrasound via generative adversarial network for breast cancer diagnosis

Elastography ultrasound (EUS) imaging is a vital ultrasound imaging modality. The current use of EUS faces many challenges, such as vulnerability to subjective manipulation, echo signal attenuation, and unknown risks of elastic pressure in certain delicate tissues. The hardware requirement of EUS also hinders the trend of miniaturization of ultrasound equipment. Here we show a cost-efficient solution by designing a deep neural network to synthesize virtual EUS (V-EUS) from conventional B-mode images. A total of 4580 breast tumor cases were collected from 15 medical centers, including a main cohort with 2501 cases for model establishment, an external dataset with 1730 cases and a portable dataset with 349 cases for testing. In the task of differentiating benign and malignant breast tumors, there is no significant difference between V-EUS and real EUS on high-end ultrasound, while the diagnostic performance of pocket-sized ultrasound can be improved by about 5% after V-EUS is equipped.

A densely interconnected network for deep learning accelerated MRI

OBJECTIVE

To improve accelerated MRI reconstruction through a densely connected cascading deep learning reconstruction framework.

MATERIALS AND METHODS

A cascading deep learning reconstruction framework (reference model) was modified by applying three architectural modifications: input-level dense connections between cascade inputs and outputs, an improved deep learning sub-network, and long-range skip-connections between subsequent deep learning networks. An ablation study was performed, where five model configurations were trained on the NYU fastMRI neuro dataset with an end-to-end scheme conjunct on four- and eightfold acceleration. The trained models were evaluated by comparing their respective structural similarity index measure (SSIM), normalized mean square error (NMSE), and peak signal to noise ratio (PSNR).

RESULTS

The proposed densely interconnected residual cascading network (DIRCN), utilizing all three suggested modifications achieved a SSIM improvement of 8% and 11%, a NMSE improvement of 14% and 23%, and a PSNR improvement of 2% and 3% for four- and eightfold acceleration, respectively. In an ablation study, the individual architectural modifications all contributed to this improvement for both acceleration factors, by improving the SSIM, NMSE, and PSNR with approximately 2-4%, 4-9%, and 0.5-1%, respectively.

CONCLUSION

The proposed architectural modifications allow for simple adjustments on an already existing cascading framework to further improve the resulting reconstructions.

Physics-Driven Deep Learning for Computational Magnetic Resonance Imaging: Combining physics and machine learning for improved medical imaging

Physics-driven deep learning methods have emerged as a powerful tool for computational magnetic resonance imaging (MRI) problems, pushing reconstruction performance to new limits. This article provides an overview of the recent developments in incorporating physics information into learning-based MRI reconstruction. We consider inverse problems with both linear and non-linear forward models for computational MRI, and review the classical approaches for solving these. We then focus on physics-driven deep learning approaches, covering physics-driven loss functions, plug-and-play methods, generative models, and unrolled networks. We highlight domain-specific challenges such as real- and complex-valued building blocks of neural networks, and translational applications in MRI with linear and non-linear forward models. Finally, we discuss common issues and open challenges, and draw connections to the importance of physics-driven learning when combined with other downstream tasks in the medical imaging pipeline.

Signal intensity informed multi-coil encoding operator for physics-guided deep learning reconstruction of highly accelerated myocardial perfusion CMR

PURPOSE

To develop a physics-guided deep learning (PG-DL) reconstruction strategy based on a signal intensity informed multi-coil (SIIM) encoding operator for highly-accelerated simultaneous multislice (SMS) myocardial perfusion cardiac MRI (CMR).

METHODS

First-pass perfusion CMR acquires highly-accelerated images with dynamically varying signal intensity/SNR following the administration of a gadolinium-based contrast agent. Thus, using PG-DL reconstruction with a conventional multi-coil encoding operator leads to analogous signal intensity variations across different time-frames at the network output, creating difficulties in generalization for varying SNR levels. We propose to use a SIIM encoding operator to capture the signal intensity/SNR variations across time-frames in a reformulated encoding operator. This leads to a more uniform/flat contrast at the output of the PG-DL network, facilitating generalizability across time-frames. PG-DL reconstruction with the proposed SIIM encoding operator is compared to PG-DL with conventional encoding operator, split slice-GRAPPA, locally low-rank (LLR) regularized reconstruction, low-rank plus sparse (L + S) reconstruction, and regularized ROCK-SPIRiT.

RESULTS

Results on highly accelerated free-breathing first pass myocardial perfusion CMR at three-fold SMS and four-fold in-plane acceleration show that the proposed method improves upon the reconstruction methods use for comparison. Substantial noise reduction is achieved compared to split slice-GRAPPA, and aliasing artifacts reduction compared to LLR regularized reconstruction, L + S reconstruction and PG-DL with conventional encoding. Furthermore, a qualitative reader study indicated that proposed method outperformed all methods.

CONCLUSION

PG-DL reconstruction with the proposed SIIM encoding operator improves generalization across different time-frames /SNRs in highly accelerated perfusion CMR.

De-Aliasing and Accelerated Sparse Magnetic Resonance Image Reconstruction Using Fully Dense CNN with Attention Gates

When sparsely sampled data are used to accelerate magnetic resonance imaging (MRI), conventional reconstruction approaches produce significant artifacts that obscure the content of the image. To remove aliasing artifacts, we propose an advanced convolutional neural network (CNN) called fully dense attention CNN (FDA-CNN). We updated the Unet model with the fully dense connectivity and attention mechanism for MRI reconstruction. The main benefit of FDA-CNN is that an attention gate in each decoder layer increases the learning process by focusing on the relevant image features and provides a better generalization of the network by reducing irrelevant activations. Moreover, densely interconnected convolutional layers reuse the feature maps and prevent the vanishing gradient problem. Additionally, we also implement a new, proficient under-sampling pattern in the phase direction that takes low and high frequencies from the k-space both randomly and non-randomly. The performance of FDA-CNN was evaluated quantitatively and qualitatively with three different sub-sampling masks and datasets. Compared with five current deep learning-based and two compressed sensing MRI reconstruction techniques, the proposed method performed better as it reconstructed smoother and brighter images. Furthermore, FDA-CNN improved the mean PSNR by 2 dB, SSIM by 0.35, and VIFP by 0.37 compared with Unet for the acceleration factor of 5.

Physics-Informed Compressed Sensing for PC-MRI: An Inverse Navier-Stokes Problem

We formulate a physics-informed compressed sensing (PICS) method for the reconstruction of velocity fields from noisy and sparse phase-contrast magnetic resonance signals. The method solves an inverse Navier-Stokes boundary value problem, which permits us to jointly reconstruct and segment the velocity field, and at the same time infer hidden quantities such as the hydrodynamic pressure and the wall shear stress. Using a Bayesian framework, we regularize the problem by introducing a priori information about the unknown parameters in the form of Gaussian random fields. This prior information is updated using the Navier-Stokes problem, an energy-based segmentation functional, and by requiring that the reconstruction is consistent with the $k$ -space signals. We create an algorithm that solves this inverse problem, and test it for noisy and sparse $k$ -space signals of the flow through a converging nozzle. We find that the method is capable of reconstructing and segmenting the velocity fields from sparsely-sampled (15% $k$ -space coverage), low ( $\sim 10$ ) signal-to-noise ratio (SNR) signals, and that the reconstructed velocity field compares well with that derived from fully-sampled (100% $k$ -space coverage) high ( $> 40$ ) SNR signals of the same flow.

A unified model for reconstruction and R2* mapping of accelerated 7T data using the quantitative recurrent inference machine

Quantitative MRI (qMRI) acquired at the ultra-high field of 7 Tesla has been used in visualizing and analyzing subcortical structures. qMRI relies on the acquisition of multiple images with different scan settings, leading to extended scanning times. Data redundancy and prior information from the relaxometry model can be exploited by deep learning to accelerate the imaging process. We propose the quantitative Recurrent Inference Machine (qRIM), with a unified forward model for joint reconstruction and R₂^*-mapping from sparse data, embedded in a Recurrent Inference Machine (RIM), an iterative inverse problem-solving network. To study the dependency of the proposed extension of the unified forward model to network architecture, we implemented and compared a quantitative End-to-End Variational Network (qE2EVN). Experiments were performed with high-resolution multi-echo gradient echo data of the brain at 7T of a cohort study covering the entire adult life span. The error in reconstructed R₂^* from undersampled data relative to reference data significantly decreased for the unified model compared to sequential image reconstruction and parameter fitting using the RIM. With increasing acceleration factor, an increasing reduction in the reconstruction error was observed, pointing to a larger benefit for sparser data. Qualitatively, this was following an observed reduction of image blurriness in R₂^*-maps. In contrast, when using the U-Net as network architecture, a negative bias in R₂^* in selected regions of interest was observed. Compressed Sensing rendered accurate, but less precise estimates of R₂^*. The qE2EVN showed slightly inferior reconstruction quality compared to the qRIM but better quality than the U-Net and Compressed Sensing. Subcortical maturation over age measured by a linearly increasing interquartile range of R₂^* in the striatum was preserved up to an acceleration factor of 9. With the integrated prior of the unified forward model, the proposed qRIM can exploit the redundancy among repeated measurements and shared information between tasks, facilitating relaxometry in accelerated MRI.

Wave-Encoded Model-Based Deep Learning for Highly Accelerated Imaging with Joint Reconstruction

A recently introduced model-based deep learning (MoDL) technique successfully incorporates convolutional neural network (CNN)-based regularizers into physics-based parallel imaging reconstruction using a small number of network parameters. Wave-controlled aliasing in parallel imaging (CAIPI) is an emerging parallel imaging method that accelerates imaging acquisition by employing sinusoidal gradients in the phase- and slice/partition-encoding directions during the readout to take better advantage of 3D coil sensitivity profiles. We propose wave-encoded MoDL (wave-MoDL) combining the wave-encoding strategy with unrolled network constraints for highly accelerated 3D imaging while enforcing data consistency. We extend wave-MoDL to reconstruct multicontrast data with CAIPI sampling patterns to leverage similarity between multiple images to improve the reconstruction quality. We further exploit this to enable rapid quantitative imaging using an interleaved look-locker acquisition sequence with T₂ preparation pulse (3D-QALAS). Wave-MoDL enables a 40 s MPRAGE acquisition at 1 mm resolution at 16-fold acceleration. For quantitative imaging, wave-MoDL permits a 1:50 min acquisition for T₁, T₂, and proton density mapping at 1 mm resolution at 12-fold acceleration, from which contrast-weighted images can be synthesized as well. In conclusion, wave-MoDL allows rapid MR acquisition and high-fidelity image reconstruction and may facilitate clinical and neuroscientific applications by incorporating unrolled neural networks into wave-CAIPI reconstruction.

Exploring the Acceleration Limits of Deep Learning Variational Network-based Two-dimensional Brain MRI

Purpose

To explore the limits of deep learning-based brain MRI reconstruction and identify useful acceleration ranges for general-purpose imaging and potential screening.

Materials and Methods

In this retrospective study conducted from 2019 through 2021, a model was trained for reconstruction on 5847 brain MR images. Performance was evaluated across a wide range of accelerations (up to 100-fold along a single phase-encoded direction for two-dimensional [2D] sections) on the fastMRI test set collected at New York University, consisting of 558 image volumes. In a sample of 69 volumes, reconstructions were classified by radiologists for identification of two clinical thresholds: (a) general-purpose diagnostic imaging and (b) potential use in a screening protocol. A Monte Carlo procedure was developed to estimate reconstruction error with only undersampled data. The model was evaluated on both in-domain and out-of-domain data. The 95% CIs were calculated using the percentile bootstrap method.

Results

Radiologists rated 100% of 69 volumes as having sufficient image quality for general-purpose imaging at up to 4× acceleration and 65 of 69 volumes (94%) as having sufficient image quality for screening at up to 14× acceleration. The Monte Carlo procedure estimated ground truth peak signal-to-noise ratio and mean squared error with coefficients of determination greater than 0.5 at 2× to 20× acceleration levels. Out-of-distribution experiments demonstrated the model's ability to produce images substantially distinct from the training set, even at 100× acceleration.

Conclusion

For 2D brain images using deep learning-based reconstruction, maximum acceleration for potential screening was three to four times higher than that for diagnostic general-purpose imaging.Keywords: MRI Reconstruction, High Acceleration, Deep Learning, Screening, Out of Distribution Supplemental material is available for this article. © RSNA, 2022.

[Applications of Artificial Intelligence in MR Image Acquisition and Reconstruction]

Recently, artificial intelligence (AI) technology has shown potential clinical utility in a wide range of MRI fields. In particular, AI models for improving the efficiency of the image acquisition process and the quality of reconstructed images are being actively developed by the MR research community. AI is expected to further reduce acquisition times in various MRI protocols used in clinical practice when compared to current parallel imaging techniques. Additionally, AI can help with tasks such as planning, parameter optimization, artifact reduction, and quality assessment. Furthermore, AI is being actively applied to automate MR image analysis such as image registration, segmentation, and object detection. For this reason, it is important to consider the effects of protocols or devices in MR image analysis. In this review article, we briefly introduced issues related to AI application of MR image acquisition and reconstruction.

An End-to-End Recurrent Neural Network for Radial MR Image Reconstruction

Recent advances in deep learning have contributed greatly to the field of parallel MR imaging, where a reduced amount of k-space data are acquired to accelerate imaging time. In our previous work, we have proposed a deep learning method to reconstruct MR images directly from k-space data acquired with Cartesian trajectories. However, MRI utilizes various non-Cartesian trajectories, such as radial trajectories, with various numbers of multi-channel RF coils according to the purpose of an MRI scan. Thus, it is important for a reconstruction network to efficiently unfold aliasing artifacts due to undersampling and to combine multi-channel k-space data into single-channel data. In this work, a neural network named 'ETER-net' is utilized to reconstruct an MR image directly from k-space data acquired with Cartesian and non-Cartesian trajectories and multi-channel RF coils. In the proposed image reconstruction network, the domain transform network converts k-space data into a rough image, which is then refined in the following network to reconstruct a final image. We also analyze loss functions including adversarial and perceptual losses to improve the network performance. For experiments, we acquired k-space data at a 3T MRI scanner with Cartesian and radial trajectories to show the learning mechanism of the direct mapping relationship between the k-space and the corresponding image by the proposed network and to demonstrate the practical applications. According to our experiments, the proposed method showed satisfactory performance in reconstructing images from undersampled single- or multi-channel k-space data with reduced image artifacts. In conclusion, the proposed method is a deep-learning-based MR reconstruction network, which can be used as a unified solution for parallel MRI, where k-space data are acquired with various scanning trajectories.

Accelerated two-dimensional phase-contrast for cardiovascular MRI using deep learning-based reconstruction with complex difference estimation

PURPOSE

To develop and validate a deep learning-based reconstruction framework for highly accelerated two-dimensional (2D) phase contrast (PC-MRI) data with accurate and precise quantitative measurements.

METHODS

We propose a modified DL-ESPIRiT reconstruction framework for 2D PC-MRI, comprised of an unrolled neural network architecture with a Complex Difference estimation (CD-DL). CD-DL was trained on 155 fully sampled 2D PC-MRI pediatric clinical datasets. The fully sampled data ( n = 29 $$ n=29 $$ ) was retrospectively undersampled (6-11 × $$ \times $$ ) and reconstructed using CD-DL and a parallel imaging and compressed sensing method (PICS). Measurements of peak velocity and total flow were compared to determine the highest acceleration rate that provided accuracy and precision within ± 5 % $$ \pm 5\% $$ . Feasibility of CD-DL was demonstrated on prospectively undersampled datasets acquired in pediatric clinical patients ( n = 5 $$ n=5 $$ ) and compared to traditional parallel imaging (PI) and PICS.

RESULTS

The retrospective evaluation showed that 9 × $$ \times $$ accelerated 2D PC-MRI images reconstructed with CD-DL provided accuracy and precision (bias, [95 % $$ \% $$ confidence intervals]) within ± 5 % $$ \pm 5\% $$ . CD-DL showed higher accuracy and precision compared to PICS for measurements of peak velocity (2.8 % $$ \% $$ [ - 2 . 9 $$ -2.9 $$ , 4.5] vs. 3.9 % $$ \% $$ [ - 11 . 0 $$ -11.0 $$ , 4.9]) and total flow (1.8 % $$ \% $$ [ - 3 . 9 $$ -3.9 $$ , 3.4] vs. 2.9 % $$ \% $$ [ - 7 . 1 $$ -7.1 $$ , 6.9]). The prospective feasibility study showed that CD-DL provided higher accuracy and precision than PICS for measurements of peak velocity and total flow.

CONCLUSION

In a retrospective evaluation, CD-DL produced quantitative measurements of 2D PC-MRI peak velocity and total flow with ≤ 5 % $$ \le 5\% $$ error in both accuracy and precision for up to 9 × $$ \times $$ acceleration. Clinical feasibility was demonstrated using a prospective clinical deployment of our 8 × $$ \times $$ undersampled acquisition and CD-DL reconstruction in a cohort of pediatric patients.

Deep learning for compressive sensing: a ubiquitous systems perspective

Compressive sensing (CS) is a mathematically elegant tool for reducing the sensor sampling rate, potentially bringing context-awareness to a wider range of devices. Nevertheless, practical issues with the sampling and reconstruction algorithms prevent further proliferation of CS in real world domains, especially among heterogeneous ubiquitous devices. Deep learning (DL) naturally complements CS for adapting the sampling matrix, reconstructing the signal, and learning from the compressed samples. While the CS–DL integration has received substantial research interest recently, it has not yet been thoroughly surveyed, nor has any light been shed on practical issues towards bringing the CS–DL to real world implementations in the ubiquitous computing domain. In this paper we identify main possible ways in which CS and DL can interplay, extract key ideas for making CS–DL efficient, outline major trends in the CS–DL research space, and derive guidelines for the future evolution of CS–DL within the ubiquitous computing domain.

Improving high frequency image features of deep learning reconstructions via k-space refinement with null-space kernel

PURPOSE

Deep learning (DL) based reconstruction using unrolled neural networks has shown great potential in accelerating MRI. However, one of the major drawbacks is the loss of high-frequency details and textures in the output. The purpose of the study is to propose a novel refinement method that uses null-space kernel to refine k-space and improve blurred image details and textures.

METHODS

The proposed method constrains the output of the DL to comply to the linear neighborhood relationship calibrated in the auto-calibration lines. To demonstrate efficacy, we tested our refinement method on the DL reconstruction under a variety of conditions (i.e., dataset, unrolled neural networks, and under-sampling scheme). Specifically, the method was tested on three large-scale public datasets (knee and brain) from fastMRI's multi-coil track.

RESULTS

The proposed scheme visually reduces the structural error in the k-space domain, enhance the homogeneity of the k-space intensity. Consequently, reconstructed image shows sharper images with enhanced details and textures. The proposed method is also successful in improving high-frequency image details (SSIM, GMSD) without sacrificing overall image error (PSNR).

CONCLUSION

Our findings imply that refining DL output using the proposed method may generally improve DL reconstruction as tested with various large-scale dataset and networks.

Multimodal MRI Reconstruction Assisted With Spatial Alignment Network

In clinical practice, multi-modal magnetic resonance imaging (MRI) with different contrasts is usually acquired in a single study to assess different properties of the same region of interest in the human body. The whole acquisition process can be accelerated by having one or more modalities under-sampled in the k -space. Recent research has shown that, considering the redundancy between different modalities, a target MRI modality under-sampled in the k -space can be more efficiently reconstructed with a fully-sampled reference MRI modality. However, we find that the performance of the aforementioned multi-modal reconstruction can be negatively affected by subtle spatial misalignment between different modalities, which is actually common in clinical practice. In this paper, we improve the quality of multi-modal reconstruction by compensating for such spatial misalignment with a spatial alignment network. First, our spatial alignment network estimates the displacement between the fully-sampled reference and the under-sampled target images, and warps the reference image accordingly. Then, the aligned fully-sampled reference image joins the multi-modal reconstruction of the under-sampled target image. Also, considering the contrast difference between the target and reference images, we have designed a cross-modality-synthesis-based registration loss in combination with the reconstruction loss, to jointly train the spatial alignment network and the reconstruction network. The experiments on both clinical MRI and multi-coil k -space raw data demonstrate the superiority and robustness of the multi-modal MRI reconstruction empowered with our spatial alignment network. Our code is publicly available at https://github.com/woxuankai/SpatialAlignmentNetwork.