Group Sparsity Residual Constraint with Non-Local Priors for Image Restoration

A Convolutional Neural Network-Based Quantization Method for Block Compressed Sensing of Images

Block compressed sensing (BCS) is a promising method for resource-constrained image/video coding applications. However, the quantization of BCS measurements has posed a challenge, leading to significant quantization errors and encoding redundancy. In this paper, we propose a quantization method for BCS measurements using convolutional neural networks (CNN). The quantization process maps measurements to quantized data that follow a uniform distribution based on the measurements' distribution, which aims to maximize the amount of information carried by the quantized data. The dequantization process restores the quantized data to data that conform to the measurements' distribution. The restored data are then modified by the correlation information of the measurements drawn from the quantized data, with the goal of minimizing the quantization errors. The proposed method uses CNNs to construct quantization and dequantization processes, and the networks are trained jointly. The distribution parameters of each block are used as side information, which is quantized with 1 bit by the same method. Extensive experiments on four public datasets showed that, compared with uniform quantization and entropy coding, the proposed method can improve the PSNR by an average of 0.48 dB without using entropy coding when the compression bit rate is 0.1 bpp.

Joint constraints of guided filtering based confidence and nonlocal sparse tensor for color polarization super-resolution imaging

This paper introduces a camera-array-based super-resolution color polarization imaging system designed to simultaneously capture color and polarization information of a scene in a single shot. Existing snapshot color polarization imaging has a complex structure and limited generalizability, which are overcome by the proposed system. In addition, a novel reconstruction algorithm is designed to exploit the complementarity and correlation between the twelve channels in acquired color polarization images for simultaneous super-resolution (SR) imaging and denoising. We propose a confidence-guided SR reconstruction algorithm based on guided filtering to enhance the constraint capability of the observed data. Additionally, by introducing adaptive parameters, we effectively balance the data fidelity constraint and the regularization constraint of nonlocal sparse tensor. Simulations were conducted to compare the proposed system with a color polarization camera. The results show that color polarization images generated by the proposed system and algorithm outperform those obtained from the color polarization camera and the state-of-the-art color polarization demosaicking algorithms. Moreover, the proposed algorithm also outperforms state-of-the-art SR algorithms based on deep learning. To evaluate the applicability of the proposed imaging system and reconstruction algorithm in practice, a prototype was constructed for color polarization image acquisition. Compared with conventional acquisition, the proposed solution demonstrates a significant improvement in the reconstructed color polarization images.

Natural image restoration based on multi-scale group sparsity residual constraints

The Group Sparse Representation (GSR) model shows excellent potential in various image restoration tasks. In this study, we propose a novel Multi-Scale Group Sparse Residual Constraint Model (MS-GSRC) which can be applied to various inverse problems, including denoising, inpainting, and compressed sensing (CS). Our new method involves the following three steps: (1) finding similar patches with an overlapping scheme for the input degraded image using a multi-scale strategy, (2) performing a group sparse coding on these patches with low-rank constraints to get an initial representation vector, and (3) under the Bayesian maximum a posteriori (MAP) restoration framework, we adopt an alternating minimization scheme to solve the corresponding equation and reconstruct the target image finally. Simulation experiments demonstrate that our proposed model outperforms in terms of both objective image quality and subjective visual quality compared to several state-of-the-art methods.

Low-Rankness Guided Group Sparse Representation for Image Restoration

As a spotlighted nonlocal image representation model, group sparse representation (GSR) has demonstrated a great potential in diverse image restoration tasks. Most of the existing GSR-based image restoration approaches exploit the nonlocal self-similarity (NSS) prior by clustering similar patches into groups and imposing sparsity to each group coefficient, which can effectively preserve image texture information. However, these methods have imposed only plain sparsity over each individual patch of the group, while neglecting other beneficial image properties, e.g., low-rankness (LR), leads to degraded image restoration results. In this article, we propose a novel low-rankness guided group sparse representation (LGSR) model for highly effective image restoration applications. The proposed LGSR jointly utilizes the sparsity and LR priors of each group of similar patches under a unified framework. The two priors serve as the complementary priors in LGSR for effectively preserving the texture and structure information of natural images. Moreover, we apply an alternating minimization algorithm with an adaptively adjusted parameter scheme to solve the proposed LGSR-based image restoration problem. Extensive experiments are conducted to demonstrate that the proposed LGSR achieves superior results compared with many popular or state-of-the-art algorithms in various image restoration tasks, including denoising, inpainting, and compressive sensing (CS).

An Improved BM3D Algorithm Based on Image Depth Feature Map and Structural Similarity Block-Matching

We propose an improved BM3D algorithm for block-matching based on UNet denoising network feature maps and structural similarity (SSIM). In response to the traditional BM3D algorithm that directly performs block-matching on a noisy image, without considering the deep-level features of the image, we propose a method that performs block-matching on the feature maps of the noisy image. In this method, we perform block-matching on multiple depth feature maps of a noisy image, and then determine the positions of the corresponding similar blocks in the noisy image based on the block-matching results, to obtain the set of similar blocks that take into account the deep-level features of the noisy image. In addition, we improve the similarity measure criterion for block-matching based on the Structural Similarity Index, which takes into account the pixel-by-pixel value differences in the image blocks while fully considering the structure, brightness, and contrast information of the image blocks. To verify the effectiveness of the proposed method, we conduct extensive comparative experiments. The experimental results demonstrate that the proposed method not only effectively enhances the denoising performance of the image, but also preserves the detailed features of the image and improves the visual quality of the denoised image.

Sparse Representation-Based Multi-Focus Image Fusion Method via Local Energy in Shearlet Domain

Multi-focus image fusion plays an important role in the application of computer vision. In the process of image fusion, there may be blurring and information loss, so it is our goal to obtain high-definition and information-rich fusion images. In this paper, a novel multi-focus image fusion method via local energy and sparse representation in the shearlet domain is proposed. The source images are decomposed into low- and high-frequency sub-bands according to the shearlet transform. The low-frequency sub-bands are fused by sparse representation, and the high-frequency sub-bands are fused by local energy. The inverse shearlet transform is used to reconstruct the fused image. The Lytro dataset with 20 pairs of images is used to verify the proposed method, and 8 state-of-the-art fusion methods and 8 metrics are used for comparison. According to the experimental results, our method can generate good performance for multi-focus image fusion.

Feasibility of a Real-Time Embedded Hyperspectral Compressive Sensing Imaging System

Hyperspectral imaging has been attracting considerable interest as it provides spectrally rich acquisitions useful in several applications, such as remote sensing, agriculture, astronomy, geology and medicine. Hyperspectral devices based on compressive acquisitions have appeared recently as an alternative to conventional hyperspectral imaging systems and allow for data-sampling with fewer acquisitions than classical imaging techniques, even under the Nyquist rate. However, compressive hyperspectral imaging requires a reconstruction algorithm in order to recover all the data from the raw compressed acquisition. The reconstruction process is one of the limiting factors for the spread of these devices, as it is generally time-consuming and comes with a high computational burden. Algorithmic and material acceleration with embedded and parallel architectures (e.g., GPUs and FPGAs) can considerably speed up image reconstruction, making hyperspectral compressive systems suitable for real-time applications. This paper provides an in-depth analysis of the required performance in terms of computing power, data memory and bandwidth considering a compressive hyperspectral imaging system and a state-of-the-art reconstruction algorithm as an example. The results of the analysis show that real-time application is possible by combining several approaches, namely, exploitation of system matrix sparsity and bandwidth reduction by appropriately tuning data value encoding.

Nonconvex Structural Sparsity Residual Constraint for Image Restoration

This article proposes a novel nonconvex structural sparsity residual constraint (NSSRC) model for image restoration, which integrates structural sparse representation (SSR) with nonconvex sparsity residual constraint (NC-SRC). Although SSR itself is powerful for image restoration by combining the local sparsity and nonlocal self-similarity in natural images, in this work, we explicitly incorporate the novel NC-SRC prior into SSR. Our proposed approach provides more effective sparse modeling for natural images by applying a more flexible sparse representation scheme, leading to high-quality restored images. Moreover, an alternating minimizing framework is developed to solve the proposed NSSRC-based image restoration problems. Extensive experimental results on image denoising and image deblocking validate that the proposed NSSRC achieves better results than many popular or state-of-the-art methods over several publicly available datasets.

Group Sparsity Mixture Model and Its Application on Image Denoising

Prior learning is a fundamental problem in the field of image processing. In this paper, we conduct a detailed study on (1) how to model and learn the prior of the image patch group, which consists of a group of non-local similar image patches, and (2) how to apply the learned prior to the whole image denoising task. To tackle the first problem, we propose a new prior model named Group Sparsity Mixture Model (GSMM). With the bilateral matrix multiplication, the GSMM can model both the local feature of a single patch and the relation among non-local similar patches, and thus it is very suitable for patch group based prior learning. This is supported by the parameter analysis which demonstrates that the learned GSMM successfully captures the inherent strong sparsity embodied in the image patch group. Besides, as a mixture model, GSMM can be used for patch group classification. This makes the image denoising method based on GSMM capable of processing patch groups flexibly. To tackle the second problem, we propose an efficient and effective patch group based image denoising framework, which is plug-and-play and compatible with any patch group prior model. Using this framework, we construct two versions of GSMM based image denoising methods, both of which outperform the competing methods based on other prior models, e.g., Field of Experts (FoE) and Gaussian Mixture Model (GMM). Also, the better version is competitive with the state-of-the-art model based method WNNM with about ×8 faster average running speed.

A Hybrid Structural Sparsification Error Model for Image Restoration

Recent works on structural sparse representation (SSR), which exploit image nonlocal self-similarity (NSS) prior by grouping similar patches for processing, have demonstrated promising performance in various image restoration applications. However, conventional SSR-based image restoration methods directly fit the dictionaries or transforms to the internal (corrupted) image data. The trained internal models inevitably suffer from overfitting to data corruption, thus generating the degraded restoration results. In this article, we propose a novel hybrid structural sparsification error (HSSE) model for image restoration, which jointly exploits image NSS prior using both the internal and external image data that provide complementary information. Furthermore, we propose a general image restoration scheme based on the HSSE model, and an alternating minimization algorithm for a range of image restoration applications, including image inpainting, image compressive sensing and image deblocking. Extensive experiments are conducted to demonstrate that the proposed HSSE-based scheme outperforms many popular or state-of-the-art image restoration methods in terms of both objective metrics and visual perception.

SMDS-Net: Model Guided Spectral-Spatial Network for Hyperspectral Image Denoising

Deep learning (DL) based hyperspectral images (HSIs) denoising approaches directly learn the nonlinear mapping between noisy and clean HSI pairs. They usually do not consider the physical characteristics of HSIs. This drawback makes the models lack interpretability that is key to understanding their denoising mechanism and limits their denoising ability. In this paper, we introduce a novel model-guided interpretable network for HSI denoising to tackle this problem. Fully considering the spatial redundancy, spectral low-rankness, and spectral-spatial correlations of HSIs, we first establish a subspace-based multidimensional sparse (SMDS) model under the umbrella of tensor notation. After that, the model is unfolded into an end-to-end network named SMDS-Net, whose fundamental modules are seamlessly connected with the denoising procedure and optimization of the SMDS model. This makes SMDS-Net convey clear physical meanings, i.e., learning the low-rankness and sparsity of HSIs. Finally, all key variables are obtained by discriminative training. Extensive experiments and comprehensive analysis on synthetic and real-world HSIs confirm the strong denoising ability, strong learning capability, promising generalization ability, and high interpretability of SMDS-Net against the state-of-the-art HSI denoising methods. The source code and data of this article will be made publicly available at https://github.com/bearshng/smds-net for reproducible research.

Multi-Color Channels Based Group Sparse Model for Image Restoration

The group sparse representation (GSR) model combines local sparsity and nonlocal similarity in image processing, and achieves excellent results. However, the traditional GSR model and all subsequent improved GSR models convert the RGB space of the image to YCbCr space, and only extract the Y (luminance) channel of YCbCr space to change the color image to a gray image for processing. As a result, the image processing process cannot be loyal to each color channel, so the repair effect is not ideal. A new group sparse representation model based on multi-color channels is proposed in this paper. The model processes R, G and B color channels simultaneously when processing color images rather than processing a single color channel and then combining the results of different channels. The proposed multi-color-channels-based GSR model is compared with state-of-the-art methods. The experimental contrast results show that the proposed model is an effective method and can obtain good results in terms of objective quantitative metrics and subjective visual effects.

Image Denoising Using a Compressive Sensing Approach Based on Regularization Constraints

In remote sensing applications and medical imaging, one of the key points is the acquisition, real-time preprocessing and storage of information. Due to the large amount of information present in the form of images or videos, compression of these data is necessary. Compressed sensing is an efficient technique to meet this challenge. It consists in acquiring a signal, assuming that it can have a sparse representation, by using a minimum number of nonadaptive linear measurements. After this compressed sensing process, a reconstruction of the original signal must be performed at the receiver. Reconstruction techniques are often unable to preserve the texture of the image and tend to smooth out its details. To overcome this problem, we propose, in this work, a compressed sensing reconstruction method that combines the total variation regularization and the non-local self-similarity constraint. The optimization of this method is performed by using an augmented Lagrangian that avoids the difficult problem of nonlinearity and nondifferentiability of the regularization terms. The proposed algorithm, called denoising-compressed sensing by regularization (DCSR) terms, will not only perform image reconstruction but also denoising. To evaluate the performance of the proposed algorithm, we compare its performance with state-of-the-art methods, such as Nesterov’s algorithm, group-based sparse representation and wavelet-based methods, in terms of denoising and preservation of edges, texture and image details, as well as from the point of view of computational complexity. Our approach permits a gain up to 25% in terms of denoising efficiency and visual quality using two metrics: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

Motion Blur Kernel Rendering Using an Inertial Sensor: Interpreting the Mechanism of a Thermal Detector

Various types of motion blur are frequently observed in the images captured by sensors based on thermal and photon detectors. The difference in mechanisms between thermal and photon detectors directly results in different patterns of motion blur. Motivated by this observation, we propose a novel method to synthesize blurry images from sharp images by analyzing the mechanisms of the thermal detector. Further, we propose a novel blur kernel rendering method, which combines our proposed motion blur model with the inertial sensor in the thermal image domain. The accuracy of the blur kernel rendering method is evaluated by the task of thermal image deblurring. We construct a synthetic blurry image dataset based on acquired thermal images using an infrared camera for evaluation. This dataset is the first blurry thermal image dataset with ground-truth images in the thermal image domain. Qualitative and quantitative experiments are extensively carried out on our dataset, which show that our proposed method outperforms state-of-the-art methods.

Meta PID Attention Network for Flexible and Efficient Real-World Noisy Image Denoising

Recent deep convolutional neural networks for real-world noisy image denoising have shown a huge boost in performance by training a well-engineered network over external image pairs. However, most of these methods are generally trained with supervision. Once the testing data is no longer compatible with the training conditions, they can exhibit poor generalization and easily result in severe overfitting or degrading performances. To tackle this barrier, we propose a novel denoising algorithm, dubbed as Meta PID Attention Network (MPA-Net). Our MPA-Net is built based upon stacking Meta PID Attention Modules (MPAMs). In each MPAM, we utilize a second-order attention module (SAM) to exploit the channel-wise feature correlations with second-order statistics, which are then adaptively updated via a proportional-integral-derivative (PID) guided meta-learning framework. This learning framework exerts the unique property of the PID controller and meta-learning scheme to dynamically generate filter weights for beneficial update of the extracted features within a feedback control system. Moreover, the dynamic nature of the framework enables the generated weights to be flexibly tweaked according to the input at test time. Thus, MPAM not only achieves discriminative feature learning, but also facilitates a robust generalization ability on distinct noises for real images. Extensive experiments on ten datasets are conducted to inspect the effectiveness of the proposed MPA-Net quantitatively and qualitatively, which demonstrates both its superior denoising performance and promising generalization ability that goes beyond those of the state-of-the-art denoising methods.

Image Stripe Noise Removal Based on Compressed Sensing

The sensors or electronic components are vulnerable to interference in the camera’s imaging process, usually leading to random directional stripes. Therefore, a method of stripe noise removal based on compressed sensing is proposed. First, the measurement matrix of the image with stripe noise is established, which makes the stripe images equivalent to the observation of the original image. Second, the relationships between the corresponding coefficients of adjacent scales are defined. On this basis, the bivariate threshold function is set in the curvelet sparse domain to represent the features of images. Finally, the Landweber iteration algorithm of alternating convex projection and filtering operation is achieved. Furthermore, to accelerate the noise removal at the initial stage of iteration and preserve the image details later, the exponential threshold function is utilized. This method does not need many samples, which is different from the current deep learning method. The experimental results show that the proposed algorithm represents excellent performance in removing the stripes and preserving the texture details. In addition, the PSNR of the denoised image has been dramatically improved compared with similar algorithms.

A Hybrid Sparse Representation Model for Image Restoration

Group-based sparse representation (GSR) uses image nonlocal self-similarity (NSS) prior to grouping similar image patches, and then performs sparse representation. However, the traditional GSR model restores the image by training degraded images, which leads to the inevitable over-fitting of the data in the training model, resulting in poor image restoration results. In this paper, we propose a new hybrid sparse representation model (HSR) for image restoration. The proposed HSR model is improved in two aspects. On the one hand, the proposed HSR model exploits the NSS priors of both degraded images and external image datasets, making the model complementary in feature space and the plane. On the other hand, we introduce a joint sparse representation model to make better use of local sparsity and NSS characteristics of the images. This joint model integrates the patch-based sparse representation (PSR) model and GSR model, while retaining the advantages of the GSR model and the PSR model, so that the sparse representation model is unified. Extensive experimental results show that the proposed hybrid model outperforms several existing image recovery algorithms in both objective and subjective evaluations.

Group-Based Sparse Representation for Compressed Sensing Image Reconstruction with Joint Regularization

Achieving high-quality reconstructions of images is the focus of research in image compressed sensing. Group sparse representation improves the quality of reconstructed images by exploiting the non-local similarity of images; however, block-matching and dictionary learning in the image group construction process leads to a long reconstruction time and artifacts in the reconstructed images. To solve the above problems, a joint regularized image reconstruction model based on group sparse representation (GSR-JR) is proposed. A group sparse coefficients regularization term ensures the sparsity of the group coefficients and reduces the complexity of the model. The group sparse residual regularization term introduces the prior information of the image to improve the quality of the reconstructed image. The alternating direction multiplier method and iterative thresholding algorithm are applied to solve the optimization problem. Simulation experiments confirm that the optimized GSR-JR model is superior to other advanced image reconstruction models in reconstructed image quality and visual effects. When the sensing rate is 0.1, compared to the group sparse residual constraint with a nonlocal prior (GSRC-NLR) model, the gain of the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) is up to 4.86 dB and 0.1189, respectively.

Image Denoising Using Nonlocal Regularized Deep Image Prior

Deep neural networks have shown great potential in various low-level vision tasks, leading to several state-of-the-art image denoising techniques. Training a deep neural network in a supervised fashion usually requires the collection of a great number of examples and the consumption of a significant amount of time. However, the collection of training samples is very difficult for some application scenarios, such as the full-sampled data of magnetic resonance imaging and the data of satellite remote sensing imaging. In this paper, we overcome the problem of a lack of training data by using an unsupervised deep-learning-based method. Specifically, we propose a deep-learning-based method based on the deep image prior (DIP) method, which only requires a noisy image as training data, without any clean data. It infers the natural images with random inputs and the corrupted observation with the help of performing correction via a convolutional network. We improve the original DIP method as follows: Firstly, the original optimization objective function is modified by adding nonlocal regularizers, consisting of a spatial filter and a frequency domain filter, to promote the gradient sparsity of the solution. Secondly, we solve the optimization problem with the alternating direction method of multipliers (ADMM) framework, resulting in two separate optimization problems, including a symmetric U-Net training step and a plug-and-play proximal denoising step. As such, the proposed method exploits the powerful denoising ability of both deep neural networks and nonlocal regularizations. Experiments validate the effectiveness of leveraging a combination of DIP and nonlocal regularizers, and demonstrate the superior performance of the proposed method both quantitatively and visually compared with the original DIP method.

Simultaneous Patch-Group Sparse Coding with Dual-Weighted ℓp Minimization for Image Restoration

Sparse coding (SC) models have been proven as powerful tools applied in image restoration tasks, such as patch sparse coding (PSC) and group sparse coding (GSC). However, these two kinds of SC models have their respective drawbacks. PSC tends to generate visually annoying blocking artifacts, while GSC models usually produce over-smooth effects. Moreover, conventional ℓ1 minimization-based convex regularization was usually employed as a standard scheme for estimating sparse signals, but it cannot achieve an accurate sparse solution under many realistic situations. In this paper, we propose a novel approach for image restoration via simultaneous patch-group sparse coding (SPG-SC) with dual-weighted ℓp minimization. Specifically, in contrast to existing SC-based methods, the proposed SPG-SC conducts the local sparsity and nonlocal sparse representation simultaneously. A dual-weighted ℓp minimization-based non-convex regularization is proposed to improve the sparse representation capability of the proposed SPG-SC. To make the optimization tractable, a non-convex generalized iteration shrinkage algorithm based on the alternating direction method of multipliers (ADMM) framework is developed to solve the proposed SPG-SC model. Extensive experimental results on two image restoration tasks, including image inpainting and image deblurring, demonstrate that the proposed SPG-SC outperforms many state-of-the-art algorithms in terms of both objective and perceptual quality.

Deep-Learned Regularization and Proximal Operator for Image Compressive Sensing

Deep learning has recently been intensively studied in the context of image compressive sensing (CS) to discover and represent complicated image structures. These approaches, however, either suffer from nonflexibility for an arbitrary sampling ratio or lack an explicit deep-learned regularization term. This paper aims to solve the CS reconstruction problem by combining the deep-learned regularization term and proximal operator. We first introduce a regularization term using a carefully designed residual-regressive net, which can measure the distance between a corrupted image and a clean image set and accurately identify to which subspace the corrupted image belongs. We then address a proximal operator with a tailored dilated residual channel attention net, which enables the learned proximal operator to map the distorted image into the clean image set. We adopt an adaptive proximal selection strategy to embed the network into the loop of the CS image reconstruction algorithm. Moreover, a self-ensemble strategy is presented to improve CS recovery performance. We further utilize state evolution to analyze the effectiveness of the designed networks. Extensive experiments also demonstrate that our method can yield superior accurate reconstruction (PSNR gain over 1 dB) compared to other competing approaches while achieving the current state-of-the-art image CS reconstruction performance. The test code is available at https://github.com/zjut-gwl/CSDRCANet.

Image Restoration via Reconciliation of Group Sparsity and Low-Rank Models

Image nonlocal self-similarity (NSS) property has been widely exploited via various sparsity models such as joint sparsity (JS) and group sparse coding (GSC). However, the existing NSS-based sparsity models are either too restrictive, e.g., JS enforces the sparse codes to share the same support, or too general, e.g., GSC imposes only plain sparsity on the group coefficients, which limit their effectiveness for modeling real images. In this paper, we propose a novel NSS-based sparsity model, namely, low-rank regularized group sparse coding (LR-GSC), to bridge the gap between the popular GSC and JS. The proposed LR-GSC model simultaneously exploits the sparsity and low-rankness of the dictionary-domain coefficients for each group of similar patches. An alternating minimization with an adaptive adjusted parameter strategy is developed to solve the proposed optimization problem for different image restoration tasks, including image denoising, image deblocking, image inpainting, and image compressive sensing. Extensive experimental results demonstrate that the proposed LR-GSC algorithm outperforms many popular or state-of-the-art methods in terms of objective and perceptual metrics.