ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing

A soft sensor model based on long&short-term memory dual pathways convolutional gated recurrent unit network for predicting cement specific surface area

The specific surface area of cement is an important index for the quality of cement products. But the time-varying delay, non-linearity and data redundancy in the process industry data make it difficult to establish an accurate online monitoring model. To solve the problems, a soft sensor model based on long&short-term memory dual pathways convolutional gated recurrent unit network (L/S-ConvGRU) is proposed for predicting the cement specific surface area. In this paper, first, as the linear coupling constraint inside the gated recurrent unit network (GRU) hinders the flow of information, parameters L and S are introduced into convolutional gated recurrent unit network (ConvGRU). L and S are decimals in the range (0, 1) which changed its internal linear constraint relationship and enhanced the feature extraction capability of the model. Then, two spatio-temporal feature extraction pathways are designed: long-term memory enhancement pathway and short-term dependence pathway, which capture long-term and short-term time-varying delay information from the sample data. Finally, the two feature extraction pathways mentioned above are applied to the L/S-ConvGRU model and the extracted spatio-temporal features are fused to achieve accurate prediction of the specific surface area of cement. The model was trained using raw data from the cement plant and the experimental results show that L/S-ConvGRU has higher precision and better generalization capability.

Guided Event Filtering: Synergy Between Intensity Images and Neuromorphic Events for High Performance Imaging

Many visual and robotics tasks in real-world scenarios rely on robust handling of high speed motion and high dynamic range (HDR) with effectively high spatial resolution and low noise. Such stringent requirements, however, cannot be directly satisfied by a single imager or imaging modality, rather by multi-modal sensors with complementary advantages. In this paper, we address high performance imaging by exploring the synergy between traditional frame-based sensors with high spatial resolution and low sensor noise, and emerging event-based sensors with high speed and high dynamic range. We introduce a novel computational framework, termed Guided Event Filtering (GEF), to process these two streams of input data and output a stream of super-resolved yet noise-reduced events. To generate high quality events, GEF first registers the captured noisy events onto the guidance image plane according to our flow model. it then performs joint image filtering that inherits the mutual structure from both inputs. Lastly, GEF re-distributes the filtered event frame in the space-time volume while preserving the statistical characteristics of the original events. When the guidance images under-perform, GEF incorporates an event self-guiding mechanism that resorts to neighbor events for guidance. We demonstrate the benefits of GEF by applying the output high quality events to existing event-based algorithms across diverse application categories, including high speed object tracking, depth estimation, high frame-rate video synthesis, and super resolution/HDR/color image restoration.

A Model-Driven Deep Unfolding Method for JPEG Artifacts Removal

Deep learning-based methods have achieved notable progress in removing blocking artifacts caused by lossy JPEG compression on images. However, most deep learning-based methods handle this task by designing black-box network architectures to directly learn the relationships between the compressed images and their clean versions. These network architectures are always lack of sufficient interpretability, which limits their further improvements in deblocking performance. To address this issue, in this article, we propose a model-driven deep unfolding method for JPEG artifacts removal, with interpretable network structures. First, we build a maximum posterior (MAP) model for deblocking using convolutional dictionary learning and design an iterative optimization algorithm using proximal operators. Second, we unfold this iterative algorithm into a learnable deep network structure, where each module corresponds to a specific operation of the iterative algorithm. In this way, our network inherits the benefits of both the powerful model ability of data-driven deep learning method and the interpretability of traditional model-driven method. By training the proposed network in an end-to-end manner, all learnable modules can be automatically explored to well characterize the representations of both JPEG artifacts and image content. Experiments on synthetic and real-world datasets show that our method is able to generate competitive or even better deblocking results, compared with state-of-the-art methods both quantitatively and qualitatively.

A cascade of preconditioned conjugate gradient networks for accelerated magnetic resonance imaging

BACKGROUND AND OBJECTIVE

Recent unfolding based compressed sensing magnetic resonance imaging (CS-MRI) methods only reinterpret conventional CS-MRI optimization algorithms and, consequently, inherit the weaknesses of the alternating optimization strategy. In order to avoid the structural complexity of the alternating optimization strategy and achieve better reconstruction performance, we propose to directly optimize the ℓ₁ regularized convex optimization problem using a deep learning approach.

METHOD

In order to achieve direct optimization, a system of equations solving the ℓ₁ regularized optimization problem is constructed from the optimality conditions of a novel primal-dual form proposed for the effective training of the sparsifying transform. The optimal solution is obtained by a cascade of unfolding networks of the preconditioned conjugate gradient (PCG) algorithm trained to minimize the mean element-wise absolute difference (ℓ₁ loss) between the terminal output and ground truth image in an end-to-end manner. The performance of the proposed method was compared with that of U-Net, PD-Net, ISTA-Net+, and the recently proposed projection-based cascaded U-Net, using single-coil knee MR images of the fastMRI dataset.

RESULTS

In our experiment, the proposed network outperformed existing unfolding-based networks and the complex version of U-Net in several subsampling scenarios. In particular, when using the random Cartesian subsampling mask with 25 % sampling rate, the proposed model outperformed PD-Net by 0.76 dB, ISTA-Net+ by 0.43 dB, and U-Net by 1.21 dB on the positron density without suppression (PD) dataset in term of peak signal to noise ratio. In comparison with the projection-based cascade U-Net, the proposed algorithm achieved approximately the same performance when the sampling rate was 25% with only 1.62% number of network parameters at the cost of a longer reconstruction time (approximately twice).

CONCLUSION

A cascade of unfolding networks of the PCG algorithm was proposed to directly optimize the ℓ₁ regularized CS-MRI optimization problem. The proposed network achieved improved reconstruction performance compared with U-Net, PD-Net, and ISTA-Net+, and achieved approximately the same performance as the projection-based cascaded U-Net while using significantly fewer network parameters.

Accelerating 3D MTC-BOOST in patients with congenital heart disease using a joint multi-scale variational neural network reconstruction

PURPOSE

Free-breathing Magnetization Transfer Contrast Bright blOOd phase SensiTive (MTC-BOOST) is a prototype balanced-Steady-State Free Precession sequence for 3D whole-heart imaging, that employs the endogenous magnetisation transfer contrast mechanism. This achieves reduction of flow and off-resonance artefacts, that often arise with the clinical T2prepared balanced-Steady-State Free Precession sequence, enabling high quality, contrast-agent free imaging of the thoracic cardiovascular anatomy. Fully-sampled MTC-BOOST acquisition requires long scan times (~10-24 min) and therefore acceleration is needed to permit its clinical incorporation. The aim of this study is to enable and clinically validate the 5-fold accelerated MTC-BOOST acquisition with joint Multi-Scale Variational Neural Network (jMS-VNN) reconstruction.

METHODS

Thirty-six patients underwent free-breathing, 3D whole-heart imaging with the MTC-BOOST sequence, which is combined with variable density spiral-like Cartesian sampling and 2D image navigators for translational motion estimation. This sequence acquires two differently weighted bright-blood volumes in an interleaved fashion, which are then joined in a phase sensitive inversion recovery reconstruction to obtain a complementary fully co-registered black-blood volume. Data from eighteen patients were used for training, whereas data from the remaining eighteen patients were used for testing/evaluation. The proposed deep-learning based approach adopts a supervised multi-scale variational neural network for joint reconstruction of the two differently weighted bright-blood volumes acquired with the 5-fold accelerated MTC-BOOST. The two contrast images are stacked as different channels in the network to exploit the shared information. The proposed approach is compared to the fully-sampled MTC-BOOST and 5-fold undersampled MTC-BOOST acquisition with Compressed Sensing (CS) reconstruction in terms of scan/reconstruction time and bright-blood image quality. Comparison against conventional 2-fold undersampled T2-prepared 3D bright-blood whole-heart clinical sequence (T2prep-3DWH) is also included.

RESULTS

Acquisition time was 3.0 ± 1.0 min for the 5-fold accelerated MTC-BOOST versus 9.0 ± 1.1 min for the fully-sampled MTC-BOOST and 11.1 ± 2.6 min for the T2prep-3DWH (p < 0.001 and p < 0.001, respectively). Reconstruction time was significantly lower with the jMS-VNN method compared to CS (10 ± 0.5 min vs 20 ± 2 s, p < 0.001). Image quality was higher for the proposed 5-fold undersampled jMS-VNN versus conventional CS, comparable or higher to the corresponding T2prep-3DWH dataset and similar to the fully-sampled MTC-BOOST.

CONCLUSION

The proposed 5-fold accelerated jMS-VNN MTC-BOOST framework provides efficient 3D whole-heart bright-blood imaging in fast acquisition and reconstruction time with concomitant reduction of flow and off-resonance artefacts, that are frequently encountered with the clinical sequence. Image quality of the cardiac anatomy and thoracic vasculature is comparable or superior to the clinical scan and 5-fold CS reconstruction in faster reconstruction time, promising potential clinical adoption.

Highly accelerated MR parametric mapping by undersampling the k-space and reducing the contrast number simultaneously with deep learning

Introduction. To propose a novel deep learning-based method called RG-Net (reconstruction and generation network) for highly accelerated MR parametric mapping by undersampling k-space and reducing the acquired contrast number simultaneously. Methods. The proposed framework consists of a reconstruction module and a generative module. The reconstruction module reconstructs MR images from the acquired few undersampled k-space data with the help of a data prior. The generative module then synthesizes the remaining multi-contrast images from the reconstructed images, where the exponential model is implicitly incorporated into the image generation through the supervision of fully sampled labels. The RG-Net was trained and tested on the T1ρ mapping data from 8 volunteers at net acceleration rates of 17, respectively. Regional T1ρ analysis for cartilage and the brain was performed to assess the performance of RG-Net. Results. RG-Net yields a high-quality T1ρ map at a high acceleration rate of 17. Compared with the competing methods that only undersample k-space, our framework achieves better performance in T1ρ value analysis. Conclusion. The proposed RG-Net can achieve a high acceleration rate while maintaining good reconstruction quality by undersampling k-space and reducing the contrast number simultaneously for fast MR parametric mapping. The generative module of our framework can also be used as an insertable module in other fast MR parametric mapping methods.

Deep learning for compressive sensing: a ubiquitous systems perspective

Compressive sensing (CS) is a mathematically elegant tool for reducing the sensor sampling rate, potentially bringing context-awareness to a wider range of devices. Nevertheless, practical issues with the sampling and reconstruction algorithms prevent further proliferation of CS in real world domains, especially among heterogeneous ubiquitous devices. Deep learning (DL) naturally complements CS for adapting the sampling matrix, reconstructing the signal, and learning from the compressed samples. While the CS–DL integration has received substantial research interest recently, it has not yet been thoroughly surveyed, nor has any light been shed on practical issues towards bringing the CS–DL to real world implementations in the ubiquitous computing domain. In this paper we identify main possible ways in which CS and DL can interplay, extract key ideas for making CS–DL efficient, outline major trends in the CS–DL research space, and derive guidelines for the future evolution of CS–DL within the ubiquitous computing domain.

Iterator-Net: sinogram-based CT image reconstruction

Image reconstruction is extremely important for computed tomography (CT) imaging, so it is significant to be continuously improved. The unfolding dynamics method combines a deep learning model with a traditional iterative algorithm. It is interpretable and has a fast reconstruction speed, but the essence of the algorithm is to replace the approximation operator in the optimization objective with a learning operator in the form of a convolutional neural network. In this paper, we firstly design a new iterator network (iNet), which is based on the universal approximation theorem and tries to simulate the functional relationship between the former and the latter in the maximum-likelihood expectation maximization (MLEM) algorithm. To evaluate the effectiveness of the method, we conduct experiments on a CT dataset, and the results show that our iNet method improves the quality of reconstructed images.

Dual-Domain Reconstruction Network with V-Net and K-Net for Fast MRI

PURPOSE

To introduce a dual-domain reconstruction network with V-Net and K-Net for accurate MR image reconstruction from undersampled k-space data.

METHODS

Most state-of-the-art reconstruction methods apply U-Net or cascaded U-Nets in the image domain and/or k-space domain. Nevertheless, these methods have the following problems: (1) directly applying U-Net in the k-space domain is not optimal for extracting features; (2) classical image-domain-oriented U-Net is heavyweighted and hence inefficient when cascaded many times to yield good reconstruction accuracy; (3) classical image-domain-oriented U-Net does not make full use of information of the encoder network for extracting features in the decoder network; and (4) existing methods are ineffective in simultaneously extracting and fusing features in the image domain and its dual k-space domain. To tackle these problems, we present 3 different methods: (1) V-Net, an image-domain encoder-decoder subnetwork that is more lightweight for cascading and effective in fully utilizing features in the encoder for decoding; (2) K-Net, a k-space domain subnetwork that is more suitable for extracting hierarchical features in the k-space domain, and (3) KV-Net, a dual-domain reconstruction network in which V-Nets and K-Nets are effectively combined and cascaded.

RESULTS

Extensive experimental results on the fastMRI dataset demonstrate that the proposed KV-Net can reconstruct high-quality images and outperform state-of-the-art approaches with fewer parameters.

CONCLUSIONS

To reconstruct images effectively and efficiently from incomplete k-space data, we have presented a dual-domain KV-Net to combine K-Nets and V-Nets. The KV-Net achieves better results with 9% and 5% parameters than comparable methods (XPD-Net and i-RIM).

Variational HyperAdam: A Meta-Learning Approach to Network Training

Stochastic optimization algorithms have been popular for training deep neural networks. Recently, there emerges a new approach of learning-based optimizer, which has achieved promising performance for training neural networks. However, these black-box learning-based optimizers do not fully take advantage of the experience in human-designed optimizers and heavily rely on learning from meta-training tasks, therefore have limited generalization ability. In this paper, we propose a novel optimizer, dubbed as Variational HyperAdam, which is based on a parametric generalized Adam algorithm, i.e., HyperAdam, in a variational framework. With Variational HyperAdam as optimizer for training neural network, the parameter update vector of the neural network at each training step is considered as random variable, whose approximate posterior distribution given the training data and current network parameter vector is predicted by Variational HyperAdam. The parameter update vector for network training is sampled from this approximate posterior distribution. Specifically, in Variational HyperAdam, we design a learnable generalized Adam algorithm for estimating expectation, paired with a VarBlock for estimating the variance of the approximate posterior distribution of parameter update vector. The Variational HyperAdam is learned in a meta-learning approach with meta-training loss derived by variational inference. Experiments verify that the learned Variational HyperAdam achieved state-of-the-art network training performance for various types of networks on different datasets, such as multilayer perceptron, CNN, LSTM and ResNet.

DU-CG-STAP Method Based on Sparse Recovery and Unsupervised Learning for Airborne Radar Clutter Suppression

With a small number of training range cells, sparse recovery (SR)-based space–time adaptive processing (STAP) methods can help to suppress clutter and detect targets effectively for airborne radar. However, SR algorithms usually have problems of high computational complexity and parameter-setting difficulties. More importantly, non-ideal factors in practice will lead to the degraded clutter suppression performance of SR-STAP methods. Based on the idea of deep unfolding (DU), a space–time two-dimensional (2D)-decoupled SR network, namely 2DMA-Net, is constructed in this paper to achieve a fast clutter spectrum estimation without complicated parameter tuning. For 2DMA-Net, without using labeled data, a self-supervised training method based on raw radar data is implemented. Then, to filter out the interferences caused by non-ideal factors, a cycle-consistent adversarial network (CycleGAN) is used as the image enhancement process for the clutter spectrum obtained using 2DMA-Net. For CycleGAN, an unsupervised training method based on unpaired data is implemented. Finally, 2DMA-Net and CycleGAN are cascaded to achieve a fast and accurate estimation of the clutter spectrum, resulting in the DU-CG-STAP method with unsupervised learning, as demonstrated in this paper. The simulation results show that, compared to existing typical SR-STAP methods, the proposed method can simultaneously improve clutter suppression performance and reduce computational complexity.

Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers

Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here we introduce a novel unsupervised MRI reconstruction method based on zero-Shot Learned Adversarial TransformERs (SLATER). SLATER embodies a deep adversarial network with cross-attention transformers to map noise and latent variables onto coil-combined MR images. During pre-training, this unconditional network learns a high-quality MRI prior in an unsupervised generative modeling task. During inference, a zero-shot reconstruction is then performed by incorporating the imaging operator and optimizing the prior to maximize consistency to undersampled data. Comprehensive experiments on brain MRI datasets clearly demonstrate the superior performance of SLATER against state-of-the-art unsupervised methods.

Accelerating multi-echo chemical shift encoded water-fat MRI using model-guided deep learning

PURPOSE

To accelerate chemical shift encoded (CSE) water-fat imaging by applying a model-guided deep learning water-fat separation (MGDL-WF) framework to the undersampled k-space data.

METHODS

A model-guided deep learning water-fat separation framework is proposed for the acceleration using Cartesian/radial undersampling data. The proposed MGDL-WF combines the power of CSE water-fat imaging model and data-driven deep learning by jointly using a multi-peak fat model and a modified residual U-net network. The model is used to guide the image reconstruction, and the network is used to capture the artifacts induced by the undersampling. A data consistency layer is used in MGDL-WF to ensure the output images to be consistent with the k-space measurements. A Gauss-Newton iteration algorithm is adapted for the gradient updating of the networks.

RESULTS

Compared with the compressed sensing water-fat separation (CS-WF) algorithm/2-step procedure algorithm, the MGDL-WF increased peak signal-to-noise ratio (PSNR) by 5.31/5.23, 6.11/4.54, and 4.75 dB/1.88 dB with Cartesian sampling, and by 4.13/6.53, 2.90/4.68, and 1.68 dB/3.48 dB with radial sampling, at acceleration rates (R) of 4, 6, and 8, respectively. By using MGDL-WF, radial sampling increased the PSNR by 2.07 dB at R = 8, compared with Cartesian sampling.

CONCLUSIONS

The proposed MGDL-WF enables exploiting features of the water images and fat images from the undersampled multi-echo data, leading to improved performance in the accelerated CSE water-fat imaging. By using MGDL-WF, radial sampling can further improve the image quality with comparable scan time in comparison with Cartesian sampling.

A Sparse-Model-Driven Network for Efficient and High-Accuracy InSAR Phase Filtering

Phase filtering is a vital step for interferometric synthetic aperture radar (InSAR) terrain elevation measurements. Existing phase filtering methods can be divided into two categories: traditional model-based and deep learning (DL)-based. Previous studies have shown that DL-based methods are frequently superior to traditional ones. However, most of the existing DL-based methods are purely data-driven and neglect the filtering model, so that they often need to use a large-scale complex architecture to fit the huge training sets. The issue brings a challenge to improve the accuracy of interferometric phase filtering without sacrificing speed. Therefore, we propose a sparse-model-driven network (SMD-Net) for efficient and high-accuracy InSAR phase filtering by unrolling the sparse regularization (SR) algorithm to solve the filtering model into a network. Unlike the existing DL-based filtering methods, the SMD-Net models the physical process of filtering in the network and contains fewer layers and parameters. It is thus expected to ensure the accuracy of the filtering without sacrificing speed. In addition, unlike the traditional SR algorithm setting the spare transform by handcrafting, a convolutional neural network (CNN) module was established to adaptively learn such a transform, which significantly improved the filtering performance. Extensive experimental results on the simulated and measured data demonstrated that the proposed method outperformed several advanced InSAR phase filtering methods in both accuracy and speed. In addition, to verify the filtering performance of the proposed method under small training samples, the training samples were reduced to 10%. The results show that the performance of the proposed method was comparable on the simulated data and superior on the real data compared with another DL-based method, which demonstrates that our method is not constrained by the requirement of a huge number of training samples.

The use of deep learning methods in low-dose computed tomography image reconstruction: a systematic review

Conventional reconstruction techniques, such as filtered back projection (FBP) and iterative reconstruction (IR), which have been utilised widely in the image reconstruction process of computed tomography (CT) are not suitable in the case of low-dose CT applications, because of the unsatisfying quality of the reconstructed image and inefficient reconstruction time. Therefore, as the demand for CT radiation dose reduction continues to increase, the use of artificial intelligence (AI) in image reconstruction has become a trend that attracts more and more attention. This systematic review examined various deep learning methods to determine their characteristics, availability, intended use and expected outputs concerning low-dose CT image reconstruction. Utilising the methodology of Kitchenham and Charter, we performed a systematic search of the literature from 2016 to 2021 in Springer, Science Direct, arXiv, PubMed, ACM, IEEE, and Scopus. This review showed that algorithms using deep learning technology are superior to traditional IR methods in noise suppression, artifact reduction and structure preservation, in terms of improving the image quality of low-dose reconstructed images. In conclusion, we provided an overview of the use of deep learning approaches in low-dose CT image reconstruction together with their benefits, limitations, and opportunities for improvement.

Dual Alternating Direction Method of Multipliers for Inverse Imaging

Inverse imaging covers a wide range of imaging applications, including super-resolution, deblurring, and compressive sensing. We propose a novel scheme to solve such problems by combining duality and the alternating direction method of multipliers (ADMM). In addition to a conventional ADMM process, we introduce a second one that solves the dual problem to find the estimated nontrivial lower bound of the objective function, and the related iteration results are used in turn to guide the primal iterations. We call this D-ADMM, and show that it converges to the global minimum when the regularization function is convex and the optimization problem has at least one optimizer. Furthermore, we show how the scheme can give rise to two specific algorithms, called D-ADMM-L2 and D-ADMM-TV, by having different regularization functions. We compare D-ADMM-TV with other methods on image super-resolution and demonstrate comparable or occasionally slightly better quality results. This paves the way of incorporating advanced operators and strategies designed for basic ADMM into the D-ADMM method as well to further improve the performances of those methods.

ADMM-SVNet: An ADMM-Based Sparse-View CT Reconstruction Network

In clinical medical applications, sparse-view computed tomography (CT) imaging is an effective method for reducing radiation doses. The iterative reconstruction method is usually adopted for sparse-view CT. In the process of optimizing the iterative model, the approach of directly solving the quadratic penalty function of the objective function can be expected to perform poorly. Compared with the direct solution method, the alternating direction method of multipliers (ADMM) algorithm can avoid the ill-posed problem associated with the quadratic penalty function. However, the regularization items, sparsity transform, and parameters in the traditional ADMM iterative model need to be manually adjusted. In this paper, we propose a data-driven ADMM reconstruction method that can automatically optimize the above terms that are difficult to choose within an iterative framework. The main contribution of this paper is that a modified U-net represents the sparse transformation, and the prior information and related parameters are automatically trained by the network. Based on a comparison with other state-of-the-art reconstruction algorithms, the qualitative and quantitative results show the effectiveness of our method for sparse-view CT image reconstruction. The experimental results show that the proposed method performs well in streak artifact elimination and detail structure preservation. The proposed network can deal with a wide range of noise levels and has exceptional performance in low-dose reconstruction tasks.

Green Visual Sensor of Plant: An Energy-Efficient Compressive Video Sensing in the Internet of Things

Internet of Things (IoT) realizes the real-time video monitoring of plant propagation or growth in the wild. However, the monitoring time is seriously limited by the battery capacity of the visual sensor, which poses a challenge to the long-working plant monitoring. Video coding is the most consuming component in a visual sensor, it is important to design an energy-efficient video codec in order to extend the time of monitoring plants. This article presents an energy-efficient Compressive Video Sensing (CVS) system to make the visual sensor green. We fuse a context-based allocation into CVS to improve the reconstruction quality with fewer computations. Especially, considering the practicality of CVS, we extract the contexts of video frames from compressive measurements but not from original pixels. Adapting to these contexts, more measurements are allocated to capture the complex structures but fewer to the simple structures. This adaptive allocation enables the low-complexity recovery algorithm to produce high-quality reconstructed video sequences. Experimental results show that by deploying the proposed context-based CVS system on the visual sensor, the rate-distortion performance is significantly improved when comparing it with some state-of-the-art methods, and the computational complexity is also reduced, resulting in a low energy consumption.

LU-Net: combining LSTM and U-Net for sinogram synthesis in sparse-view SPECT reconstruction

Lowering the dose in single-photon emission computed tomography (SPECT) imaging to reduce the radiation damage to patients has become very significant. In SPECT imaging, lower radiation dose can be achieved by reducing the activity of administered radiotracer, which will lead to projection data with either sparse projection views or reduced photon counts per view. Direct reconstruction of sparse-view projection data may lead to severe ray artifacts in the reconstructed image. Many existing works use neural networks to synthesize the projection data of sparse-view to address the issue of ray artifacts. However, these methods rarely consider the sequence feature of projection data along projection view. This work is dedicated to developing a neural network architecture that accounts for the sequence feature of projection data at adjacent view angles. In this study, we propose a network architecture combining Long Short-Term Memory network (LSTM) and U-Net, dubbed LU-Net, to learn the mapping from sparse-view projection data to full-view data. In particular, the LSTM module in the proposed network architecture can learn the sequence feature of projection data at adjacent angles to synthesize the missing views in the sinogram. All projection data used in the numerical experiment are generated by the Monte Carlo simulation software SIMIND. We evenly sample the full-view sinogram and obtain the 1/2-, 1/3- and 1/4-view projection data, respectively, representing three different levels of view sparsity. We explore the performance of the proposed network architecture at the three simulated view levels. Finally, we employ the preconditioned alternating projection algorithm (PAPA) to reconstruct the synthesized projection data. Compared with U-Net and traditional iterative reconstruction method with total variation regularization as well as PAPA solver (TV-PAPA), the proposed network achieves significant improvement in both global and local quality metrics.

A Review of Deep Learning Methods for Compressed Sensing Image Reconstruction and Its Medical Applications

Compressed sensing (CS) and its medical applications are active areas of research. In this paper, we review recent works using deep learning method to solve CS problem for images or medical imaging reconstruction including computed tomography (CT), magnetic resonance imaging (MRI) and positron-emission tomography (PET). We propose a novel framework to unify traditional iterative algorithms and deep learning approaches. In short, we define two projection operators toward image prior and data consistency, respectively, and any reconstruction algorithm can be decomposed to the two parts. Though deep learning methods can be divided into several categories, they all satisfies the framework. We built the relationship between different reconstruction methods of deep learning, and connect them to traditional methods through the proposed framework. It also indicates that the key to solve CS problem and its medical applications is how to depict the image prior. Based on the framework, we analyze the current deep learning methods and point out some important directions of research in the future.

AI musculoskeletal clinical applications: how can AI increase my day-to-day efficiency?

Artificial intelligence (AI) is expected to bring greater efficiency in radiology by performing tasks that would otherwise require human intelligence, also at a much faster rate than human performance. In recent years, milestone deep learning models with unprecedented low error rates and high computational efficiency have shown remarkable performance for lesion detection, classification, and segmentation tasks. However, the growing field of AI has significant implications for radiology that are not limited to visual tasks. These are essential applications for optimizing imaging workflows and improving noninterpretive tasks. This article offers an overview of the recent literature on AI, focusing on the musculoskeletal imaging chain, including initial patient scheduling, optimized protocoling, magnetic resonance imaging reconstruction, image enhancement, medical image-to-image translation, and AI-aided image interpretation. The substantial developments of advanced algorithms, the emergence of massive quantities of medical data, and the interest of researchers and clinicians reveal the potential for the growing applications of AI to augment the day-to-day efficiency of musculoskeletal radiologists.

Task-Oriented Convex Bilevel Optimization With Latent Feasibility

This paper firstly proposes a convex bilevel optimization paradigm to formulate and optimize popular learning and vision problems in real-world scenarios. Different from conventional approaches, which directly design their iteration schemes based on given problem formulation, we introduce a task-oriented energy as our latent constraint which integrates richer task information. By explicitly re- characterizing the feasibility, we establish an efficient and flexible algorithmic framework to tackle convex models with both shrunken solution space and powerful auxiliary (based on domain knowledge and data distribution of the task). In theory, we present the convergence analysis of our latent feasibility re- characterization based numerical strategy. We also analyze the stability of the theoretical convergence under computational error perturbation. Extensive numerical experiments are conducted to verify our theoretical findings and evaluate the practical performance of our method on different applications.

Progressive Residual Learning with Memory Upgrade for Ultrasound Image Blind Super-resolution

For clinical medical diagnosis and treatment, image super-resolution (SR) technology will be helpful to improve the ultrasonic imaging quality so as to enhance the accuracy of disease diagnosis. However, due to the differences of sensing devices or transmission media, the resolution degradation process of ultrasound imaging in real scenes is uncontrollable, especially when the blur kernel is usually unknown. This issue makes current endto- end SR networks poor performance when applied to ultrasonic images. Aiming to achieve effective SR in real ultrasound medical scenes, in this work, we propose a blind deep SR method based on progressive residual learning and memory upgrade. Specifically, we estimate the accurate blur kernel from the spatial attention map block of low resolution (LR) ultrasound image through a multi-label classification network, then we construct three modules - up-sampling (US) module, residual learning (RL) model and memory upgrading (MU) model for ultrasound image blind SR. The US module is designed to upscale the input information and the up-sampled residual result will be used for SR reconstruction. The RL module is employed to approximate the original LR and continuously generate the updated residual and feed it to the next US module. The last MU module can store all progressively learned residuals, which offers increased interactions between the US and RL modules, augmenting the details recovery. Extensive experiments and evaluations on the benchmark CCA-US and US-CASE datasets demonstrate the proposed approach achieves better performance against the state-ofthe- art methods.

Undersampled Multi-contrast MRI Reconstruction Based on Double-domain Generative Adversarial Network

Multi-contrast magnetic resonance imaging can provide comprehensive information for clinical diagnosis. However, multi-contrast imaging suffers from long acquisition time, which makes it inhibitive for daily clinical practice. Subsampling k-space is one of the main methods to speed up scan time. Missing k-space samples will lead to inevitable serious artifacts and noise. Considering the assumption that different contrast modalities share some mutual information, it may be possible to exploit this redundancy to accelerate multi-contrast imaging acquisition. Recently, generative adversarial network shows superior performance in image reconstruction and synthesis. Some studies based on k-space reconstruction also exhibit superior performance over conventional state-of-art method. In this study, we propose a cross-domain two-stage generative adversarial network for multi-contrast images reconstruction based on prior full-sampled contrast and undersampled information. The new approach integrates reconstruction and synthesis, which estimates and completes the missing k-space and then refines in image space. It takes one fully-sampled contrast modality data and highly undersampled data from several other modalities as input, and outputs high quality images for each contrast simultaneously. The network is trained and tested on a public brain dataset from healthy subjects. Quantitative comparisons against baseline clearly indicate that the proposed method can effectively reconstruct undersampled images. Even under high acceleration, the network still can recover texture details and reduce artifacts.

2D probabilistic undersampling pattern optimization for MR image reconstruction

With 3D magnetic resonance imaging (MRI), a tradeoff exists between higher image quality and shorter scan time. One way to solve this problem is to reconstruct high-quality MRI images from undersampled k-space. There have been many recent studies exploring effective k-space undersampling patterns and designing MRI reconstruction methods from undersampled k-space, which are two necessary steps. Most studies separately considered these two steps, although in theory, their performance is dependent on each other. In this study, we propose a joint optimization model, trained end-to-end, to simultaneously optimize the undersampling pattern in the Fourier domain and the reconstruction model in the image domain. A 2D probabilistic undersampling layer was designed to optimize the undersampling pattern and probability distribution in a differentiable manner. A 2D inverse Fourier transform layer was implemented to connect the Fourier domain and the image domain during the forward and back propagation. Finally, we discovered an optimized relationship between the probability distribution of the undersampling pattern and its corresponding sampling rate. Further testing was performed using 3D T1-weighted MR images of the brain from the MICCAI 2013 Grand Challenge on Multi-Atlas Labeling dataset and locally acquired brain 3D T1-weighted MR images of healthy volunteers and contrast-enhanced 3D T1-weighted MR images of high-grade glioma patients. The results showed that the recovered MR images using our 2D probabilistic undersampling pattern (with or without the reconstruction network) significantly outperformed those using the existing start-of-the-art undersampling strategies for both qualitative and quantitative comparison, suggesting the advantages and some extent of the generalization of our proposed method.

Accelerate gas diffusion-weighted MRI for lung morphometry with deep learning

OBJECTIVES

Multiple b-value gas diffusion-weighted MRI (DW-MRI) enables non-invasive and quantitative assessment of lung morphometry, but its long acquisition time is not well-tolerated by patients. We aimed to accelerate multiple b-value gas DW-MRI for lung morphometry using deep learning.

METHODS

A deep cascade of residual dense network (DC-RDN) was developed to reconstruct high-quality DW images from highly undersampled k-space data. Hyperpolarized ¹²⁹Xe lung ventilation images were acquired from 101 participants and were retrospectively collected to generate synthetic DW-MRI data to train the DC-RDN. Afterwards, the performance of the DC-RDN was evaluated on retrospectively and prospectively undersampled multiple b-value ¹²⁹Xe MRI datasets.

RESULTS

Each slice with size of 64 × 64 × 5 could be reconstructed within 7.2 ms. For the retrospective test data, the DC-RDN showed significant improvement on all quantitative metrics compared with the conventional reconstruction methods (p < 0.05). The apparent diffusion coefficient (ADC) and morphometry parameters were not significantly different between the fully sampled and DC-RDN reconstructed images (p > 0.05). For the prospectively accelerated acquisition, the required breath-holding time was reduced from 17.8 to 4.7 s with an acceleration factor of 4. Meanwhile, the prospectively reconstructed results showed good agreement with the fully sampled images, with a mean difference of -0.72% and -0.74% regarding global mean ADC and mean linear intercept (L_m) values.

CONCLUSIONS

DC-RDN is effective in accelerating multiple b-value gas DW-MRI while maintaining accurate estimation of lung microstructural morphometry, facilitating the clinical potential of studying lung diseases with hyperpolarized DW-MRI.

KEY POINTS

• The deep cascade of residual dense network allowed fast and high-quality reconstruction of multiple b-value gas diffusion-weighted MRI at an acceleration factor of 4. • The apparent diffusion coefficient and morphometry parameters were not significantly different between the fully sampled images and the reconstructed results (p > 0.05). • The required breath-holding time was reduced from 17.8 to 4.7 s and each slice with size of 64 × 64 × 5 could be reconstructed within 7.2 ms.

Evaluation on the generalization of a learned convolutional neural network for MRI reconstruction

Recently, deep learning approaches with various network architectures have drawn significant attention from the magnetic resonance imaging (MRI) community because of their great potential for image reconstruction from undersampled k-space data in fast MRI. However, the robustness of a trained network when applied to test data deviated from training data is still an important open question. In this work, we focus on quantitatively evaluating the influence of image contrast, human anatomy, sampling pattern, undersampling factor, and noise level on the generalization of a trained network composed by a cascade of several CNNs and a data consistency layer, called a deep cascade of convolutional neural network (DC-CNN). The DC-CNN is trained from datasets with different image contrast, human anatomy, sampling pattern, undersampling factor, and noise level, and then applied to test datasets consistent or inconsistent with the training datasets to assess the generalizability of the learned DC-CNN network. The results of our experiments show that reconstruction quality from the DC-CNN network is highly sensitive to sampling pattern, undersampling factor, and noise level, which are closely related to signal-to-noise ratio (SNR), and is relatively less sensitive to the image contrast. We also show that a deviation of human anatomy between training and test data leads to a substantial reduction of image quality for the brain dataset, whereas comparable performance for the chest and knee dataset having fewer anatomy details than brain images. This work further provides some empirical understanding of the generalizability of trained networks when there are deviations between training and test data. It also demonstrates the potential of transfer learning for image reconstruction from datasets different from those used in training the network.

Soft Compression for Lossless Image Coding Based on Shape Recognition

Soft compression is a lossless image compression method that is committed to eliminating coding redundancy and spatial redundancy simultaneously. To do so, it adopts shapes to encode an image. In this paper, we propose a compressible indicator function with regard to images, which gives a threshold of the average number of bits required to represent a location and can be used for illustrating the working principle. We investigate and analyze soft compression for binary image, gray image and multi-component image with specific algorithms and compressible indicator value. In terms of compression ratio, the soft compression algorithm outperforms the popular classical standards PNG and JPEG2000 in lossless image compression. It is expected that the bandwidth and storage space needed when transmitting and storing the same kind of images (such as medical images) can be greatly reduced with applying soft compression.

A Novel Approach to Large-Scale Dynamically Weighted Directed Network Representation

A dynamically weighted directed network (DWDN) is frequently encountered in various big data-related applications like a terminal interaction pattern analysis system (TIPAS) concerned in this study. It consists of large-scale dynamic interactions among numerous nodes. As the involved nodes increase drastically, it becomes impossible to observe their full interactions at each time slot, making a resultant DWDN High Dimensional and Incomplete (HDI). An HDI DWDN, in spite of its incompleteness, contains rich knowledge regarding involved nodes various behavior patterns. To extract such knowledge from an HDI DWDN, this paper proposes a novel Alternating direction method of multipliers (ADMM)-based Nonnegative Latent-factorization of Tensors (ANLT) model. It adopts three-fold ideas: a) building a data density-oriented augmented Lagrangian function for efficiently handling an HDI tensors incompleteness and nonnegativity; b) splitting the optimization task in each iteration into an elaborately designed subtask series where each one is solved based on the previously solved ones following the ADMM principle to achieve fast convergence; and c) theoretically proving that its convergence is guaranteed with its efficient learning scheme. Experimental results on six DWDNs from real applications demonstrate that the proposed ANLT outperforms state-of-the-art models significantly in both computational efficiency and prediction accuracy.

Artificial intelligence with deep learning in nuclear medicine and radiology

The use of deep learning in medical imaging has increased rapidly over the past few years, finding applications throughout the entire radiology pipeline, from improved scanner performance to automatic disease detection and diagnosis. These advancements have resulted in a wide variety of deep learning approaches being developed, solving unique challenges for various imaging modalities. This paper provides a review on these developments from a technical point of view, categorizing the different methodologies and summarizing their implementation. We provide an introduction to the design of neural networks and their training procedure, after which we take an extended look at their uses in medical imaging. We cover the different sections of the radiology pipeline, highlighting some influential works and discussing the merits and limitations of deep learning approaches compared to other traditional methods. As such, this review is intended to provide a broad yet concise overview for the interested reader, facilitating adoption and interdisciplinary research of deep learning in the field of medical imaging.

Generalized deep iterative reconstruction for sparse-view CT imaging

Sparse-view CT is a promising approach in reducing the X-ray radiation dose in clinical CT imaging. However, the CT images reconstructed from the conventional filtered backprojection (FBP) algorithm suffer from severe streaking artifacts. Iterative reconstruction (IR) algorithms have been widely adopted to mitigate these streaking artifacts, but they may prolong the CT imaging time due to the intense data-specific computations. Recently, model-driven deep learning (DL) CT image reconstruction method, which unrolls the iterative optimization procedures into the deep neural network, has shown exciting prospect in improving the image quality and shortening the reconstruction time. In this work, we explore the generalized unrolling scheme for such iterative model to further enhance its performance on sparse-view CT imaging. By using it, the iteration parameters, regularizer term, data-fidelity term and even the mathematical operations are all assumed to be learned and optimized via the network training. Results from the numerical and experimental sparse-view CT imaging demonstrate that the newly proposed network with the maximum generalization provides the best reconstruction performance.

Deep learning for digital holography: a review

Recent years have witnessed the unprecedented progress of deep learning applications in digital holography (DH). Nevertheless, there remain huge potentials in how deep learning can further improve performance and enable new functionalities for DH. Here, we survey recent developments in various DH applications powered by deep learning algorithms. This article starts with a brief introduction to digital holographic imaging, then summarizes the most relevant deep learning techniques for DH, with discussions on their benefits and challenges. We then present case studies covering a wide range of problems and applications in order to highlight research achievements to date. We provide an outlook of several promising directions to widen the use of deep learning in various DH applications.

Learning Data Consistency and its Application to Dynamic MR Imaging

Magnetic resonance (MR) image reconstruction from undersampled k-space data can be formulated as a minimization problem involving data consistency and image prior. Existing deep learning (DL)-based methods for MR reconstruction employ deep networks to exploit the prior information and integrate the prior knowledge into the reconstruction under the explicit constraint of data consistency, without considering the real distribution of the noise. In this work, we propose a new DL-based approach termed Learned DC that implicitly learns the data consistency with deep networks, corresponding to the actual probability distribution of system noise. The data consistency term and the prior knowledge are both embedded in the weights of the networks, which provides an utterly implicit manner of learning reconstruction model. We evaluated the proposed approach with highly undersampled dynamic data, including the dynamic cardiac cine data with up to 24-fold acceleration and dynamic rectum data with the acceleration factor equal to the number of phases. Experimental results demonstrate the superior performance of the Learned DC both quantitatively and qualitatively than the state-of-the-art.

Forward Model and Deep Learning Based Iterative Deconvolution for Robust Dynamic CT Perfusion

Perfusion maps obtained from low-dose computed tomography (CT) data suffer from poor signal to noise ratio. To enhance the quality of the perfusion maps, several works rely on denoising the low-dose CT (LD-CT) images followed by conventional regularized deconvolution. Recent works employ deep neural networks (DNN) for learning a direct mapping between the noisy and the clean perfusion maps ignoring the convolution-based forward model. DNN-based methods are not robust to practical variations in the data that are seen in real-world applications such as stroke. In this work, we propose an iterative framework that combines the perfusion forward model with a DNN-based regularizer to obtain perfusion maps directly from the LD-CT dynamic data. To improve the robustness of the DNN, we leverage the anatomical information from the contrast-enhanced LD-CT images to learn the mapping between low-dose and standard-dose perfusion maps. Through empirical experiments, we show that our model is robust both qualitatively and quantitatively to practical perturbations in the data.

XCloud-pFISTA: A Medical Intelligence Cloud for Accelerated MRI

Machine learning and artificial intelligence have shown remarkable performance in accelerated magnetic resonance imaging (MRI). Cloud computing technologies have great advantages in building an easily accessible platform to deploy advanced algorithms. In this work, we develop an open-access, easy-to-use and high-performance medical intelligence cloud computing platform (XCloud-pFISTA) to reconstruct MRI images from undersampled k-space data. Two state-of-the-art approaches of the Projected Fast Iterative Soft-Thresholding Algorithm (pFISTA) family have been successfully implemented on the cloud. This work can be considered as a good example of cloud-based medical image reconstruction and may benefit the future development of integrated reconstruction and online diagnosis system.

Deep learning based timing calibration for PET

Neural network has been found an increasingly wide utilization in all fields. Owing to the fact that the traditional optimized algorithm, Iterative Shrinkage-Thresholding Algorithm (ISTA) or Alternating Direction Method of Multi-pliers (ADMM), could be presented by a form of network, and it could overcome some shortcomings of traditional algorithms, which inspired us to introduce the structured deep network into PET timing calibration. In this paper, by reformulating an ADMM algorithm to a deep network, we introduce a ADMM-Net framework for calibration, which combines the advantage of compatibility of consistency condition method. To verify the performance, several experiments of Monte Carlo simulation in GATE are performed.

Patient-specific hyperparameter learning for optimization-based CT image reconstruction

We propose a hyperparameter learning framework that learnspatient-specifichyperparameters for optimization-based image reconstruction problems for x-ray CT applications. The framework consists of two functional modules: (1) a hyperparameter learning module parameterized by a convolutional neural network, (2) an image reconstruction module that takes as inputs both the noisy sinogram and the hyperparameters from (1) and generates the reconstructed images. As a proof-of-concept study, in this work we focus on a subclass of optimization-based image reconstruction problems with exactly computable solutions so that the whole network can be trained end-to-end in an efficient manner. Unlike existing hyperparameter learning methods, our proposed framework generates patient-specific hyperparameters from the sinogram of the same patient. Numerical studies demonstrate the effectiveness of our proposed approach compared to bi-level optimization.

Deep-Learned Regularization and Proximal Operator for Image Compressive Sensing

Deep learning has recently been intensively studied in the context of image compressive sensing (CS) to discover and represent complicated image structures. These approaches, however, either suffer from nonflexibility for an arbitrary sampling ratio or lack an explicit deep-learned regularization term. This paper aims to solve the CS reconstruction problem by combining the deep-learned regularization term and proximal operator. We first introduce a regularization term using a carefully designed residual-regressive net, which can measure the distance between a corrupted image and a clean image set and accurately identify to which subspace the corrupted image belongs. We then address a proximal operator with a tailored dilated residual channel attention net, which enables the learned proximal operator to map the distorted image into the clean image set. We adopt an adaptive proximal selection strategy to embed the network into the loop of the CS image reconstruction algorithm. Moreover, a self-ensemble strategy is presented to improve CS recovery performance. We further utilize state evolution to analyze the effectiveness of the designed networks. Extensive experiments also demonstrate that our method can yield superior accurate reconstruction (PSNR gain over 1 dB) compared to other competing approaches while achieving the current state-of-the-art image CS reconstruction performance. The test code is available at https://github.com/zjut-gwl/CSDRCANet.

Deep learning in magnetic resonance image reconstruction

Magnetic resonance (MR) imaging visualises soft tissue contrast in exquisite detail without harmful ionising radiation. In this work, we provide a state-of-the-art review on the use of deep learning in MR image reconstruction from different image acquisition types involving compressed sensing techniques, parallel image acquisition and multi-contrast imaging. Publications with deep learning-based image reconstruction for MR imaging were identified from the literature (PubMed and Google Scholar), and a comprehensive description of each of the works was provided. A detailed comparison that highlights the differences, the data used and the performance of each of these works were also made. A discussion of the potential use cases for each of these methods is provided. The sparse image reconstruction methods were found to be most popular in using deep learning for improved performance, accelerating acquisitions by around 4-8 times. Multi-contrast image reconstruction methods rely on at least one pre-acquired image, but can achieve 16-fold, and even up to 32- to 50-fold acceleration depending on the set-up. Parallel imaging provides frameworks to be integrated in many of these methods for additional speed-up potential. The successful use of compressed sensing techniques and multi-contrast imaging with deep learning and parallel acquisition methods could yield significant MR acquisition speed-ups within clinical routines in the near future.

Fully Learnable Model for Task-Driven Image Compressed Sensing

This study primarily investigates image sensing at low sampling rates with convolutional neural networks (CNN) for specific applications. To improve the image acquisition efficiency in energy-limited systems, this study, inspired by compressed sensing, proposes a fully learnable model for task-driven image-compressed sensing (FLCS). The FLCS, based on Deep Convolution Generative Adversarial Networks (DCGAN) and Variational Auto-encoder (VAE), divides the image-compressed sensing model into three learnable parts, i.e., the Sampler, the Solver and the Rebuilder. To be specific, a measurement matrix suitable for a type of image is obtained by training the Sampler. The Solver calculates the image’s low-dimensional representation with the measurements. The Rebuilder learns a mapping from the low-dimensional latent space to the image space. All the mentioned could be trained jointly or individually for a range of application scenarios. The pre-trained FLCS reconstructs images with few iterations for task-driven compressed sensing. As indicated from the experimental results, compared with existing approaches, the proposed method could significantly improve the reconstructed images’ quality while decreasing the running time. This study is of great significance for the application of image-compressed sensing at low sampling rates.

Deep K-SVD Denoising

This work considers noise removal from images, focusing on the well-known K-SVD denoising algorithm. This sparsity-based method was proposed in 2006, and for a short while it was considered as state-of-the-art. However, over the years it has been surpassed by other methods, including the recent deep-learning-based newcomers. The question we address in this paper is whether K-SVD was brought to its peak in its original conception, or whether it can be made competitive again. The approach we take in answering this question is to redesign the algorithm to operate in a supervised manner. More specifically, we propose an end-to-end deep architecture with the exact K-SVD computational path, and train it for optimized denoising. Our work shows how to overcome difficulties arising in turning the K-SVD scheme into a differentiable, and thus learnable, machine. With a small number of parameters to learn and while preserving the original K-SVD essence, the proposed architecture is shown to outperform the classical K-SVD algorithm substantially, and getting closer to recent state-of-the-art learning-based denoising methods. Adopting a broader context, this work touches on themes around the design of deep-learning solutions for image processing tasks, while paving a bridge between classic methods and novel deep-learning-based ones.

Which GAN? A comparative study of generative adversarial network-based fast MRI reconstruction

Fast magnetic resonance imaging (MRI) is crucial for clinical applications that can alleviate motion artefacts and increase patient throughput. K-space undersampling is an obvious approach to accelerate MR acquisition. However, undersampling of k-space data can result in blurring and aliasing artefacts for the reconstructed images. Recently, several studies have been proposed to use deep learning-based data-driven models for MRI reconstruction and have obtained promising results. However, the comparison of these methods remains limited because the models have not been trained on the same datasets and the validation strategies may be different. The purpose of this work is to conduct a comparative study to investigate the generative adversarial network (GAN)-based models for MRI reconstruction. We reimplemented and benchmarked four widely used GAN-based architectures including DAGAN, ReconGAN, RefineGAN and KIGAN. These four frameworks were trained and tested on brain, knee and liver MRI images using twofold, fourfold and sixfold accelerations, respectively, with a random undersampling mask. Both quantitative evaluations and qualitative visualization have shown that the RefineGAN method has achieved superior performance in reconstruction with better accuracy and perceptual quality compared to other GAN-based methods. This article is part of the theme issue 'Synergistic tomographic image reconstruction: part 1'.

Triply Complementary Priors for Image Restoration

Recent works that utilized deep models have achieved superior results in various image restoration (IR) applications. Such approach is typically supervised, which requires a corpus of training images with distributions similar to the images to be recovered. On the other hand, the shallow methods, which are usually unsupervised remain promising performance in many inverse problems, e.g., image deblurring and image compressive sensing (CS), as they can effectively leverage nonlocal self-similarity priors of natural images. However, most of such methods are patch-based leading to the restored images with various artifacts due to naive patch aggregation in addition to the slow speed. Using either approach alone usually limits performance and generalizability in IR tasks. In this paper, we propose a joint low-rank and deep (LRD) image model, which contains a pair of triply complementary priors, namely, internal and external, shallow and deep, and non-local and local priors. We then propose a novel hybrid plug-and-play (H-PnP) framework based on the LRD model for IR. Following this, a simple yet effective algorithm is developed to solve the proposed H-PnP based IR problems. Extensive experimental results on several representative IR tasks, including image deblurring, image CS and image deblocking, demonstrate that the proposed H-PnP algorithm achieves favorable performance compared to many popular or state-of-the-art IR methods in terms of both objective and visual perception.

High-Resolution ISAR Imaging and Autofocusing via 2D-ADMM-Net

A deep-learning architecture, dubbed as the 2D-ADMM-Net (2D-ADN), is proposed in this article. It provides effective high-resolution 2D inverse synthetic aperture radar (ISAR) imaging under scenarios of low SNRs and incomplete data, by combining model-based sparse reconstruction and data-driven deep learning. Firstly, mapping from ISAR images to their corresponding echoes in the wavenumber domain is derived. Then, a 2D alternating direction method of multipliers (ADMM) is unrolled and generalized to a deep network, where all adjustable parameters in the reconstruction layers, nonlinear transform layers, and multiplier update layers are learned by an end-to-end training through back-propagation. Since the optimal parameters of each layer are learned separately, 2D-ADN exhibits more representation flexibility and preferable reconstruction performance than model-driven methods. Simultaneously, it is able to better facilitate ISAR imaging with limited training samples than data-driven methods owing to its simple structure and small number of adjustable parameters. Additionally, benefiting from the good performance of 2D-ADN, a random phase error estimation method is proposed, through which well-focused imaging can be acquired. It is demonstrated by experiments that although trained by only a few simulated images, the 2D-ADN shows good adaptability to measured data and favorable imaging results with a clear background can be obtained in a short time.

EnGe-CSNet: A Trainable Image Compressed Sensing Model Based on Variational Encoder and Generative Networks

The present study primarily investigates the topic of image reconstruction at high compression rates. As proven from compressed sensing theory, an appropriate algorithm is capable of reconstructing natural images from a few measurements since they are sparse in several transform domains (e.g., Discrete Cosine Transform and Wavelet Transform). To enhance the quality of reconstructed images in specific applications, this paper builds a trainable deep compressed sensing model, termed as EnGe-CSNet, by combining Convolution Generative Adversarial Networks and a Variational Autoencoder. Given the significant structural similarity between a certain type of natural images collected with image sensors, deep convolutional networks are pre-trained on images that are set to learn the low dimensional manifolds of high dimensional images. The generative network is employed as the prior information, and it is used to reconstruct images from compressed measurements. As revealed from the experimental results, the proposed model exhibits a better performance than competitive algorithms at high compression rates. Furthermore, as indicated by several reconstructed samples of noisy images, the model here is robust to pattern noise. The present study is critical to facilitating the application of image compressed sensing.

FISTA-Net: Learning a Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging

Inverse problems are essential to imaging applications. In this letter, we propose a model-based deep learning network, named FISTA-Net, by combining the merits of interpretability and generality of the model-based Fast Iterative Shrinkage/Thresholding Algorithm (FISTA) and strong regularization and tuning-free advantages of the data-driven neural network. By unfolding the FISTA into a deep network, the architecture of FISTA-Net consists of multiple gradient descent, proximal mapping, and momentum modules in cascade. Different from FISTA, the gradient matrix in FISTA-Net can be updated during iteration and a proximal operator network is developed for nonlinear thresholding which can be learned through end-to-end training. Key parameters of FISTA-Net including the gradient step size, thresholding value and momentum scalar are tuning-free and learned from training data rather than hand-crafted. We further impose positive and monotonous constraints on these parameters to ensure they converge properly. The experimental results, evaluated both visually and quantitatively, show that the FISTA-Net can optimize parameters for different imaging tasks, i.e. Electromagnetic Tomography (EMT) and X-ray Computational Tomography (X-ray CT). It outperforms the state-of-the-art model-based and deep learning methods and exhibits good generalization ability over other competitive learning-based approaches under different noise levels.