Image restoration using space-variant Gaussian scale mixtures in overcomplete pyramids

Bayesian Image Super-Resolution With Deep Modeling of Image Statistics

Modeling statistics of image priors is useful for image super-resolution, but little attention has been paid from the massive works of deep learning-based methods. In this work, we propose a Bayesian image restoration framework, where natural image statistics are modeled with the combination of smoothness and sparsity priors. Concretely, first we consider an ideal image as the sum of a smoothness component and a sparsity residual, and model real image degradation including blurring, downscaling, and noise corruption. Then, we develop a variational Bayesian approach to infer their posteriors. Finally, we implement the variational approach for single image super-resolution (SISR) using deep neural networks, and propose an unsupervised training strategy. The experiments on three image restoration tasks, i.e., ideal SISR, realistic SISR, and real-world SISR, demonstrate that our method has superior model generalizability against varying noise levels and degradation kernels and is effective in unsupervised SISR. The code and resulting models are released via https://zmiclab.github.io/projects.html.

Back-Projection based Fidelity Term for Ill-Posed Linear Inverse Problems

Ill-posed linear inverse problems appear in many image processing applications, such as deblurring, superresolution and compressed sensing. Many restoration strategies involve minimizing a cost function, which is composed of fidelity and prior terms, balanced by a regularization parameter. While a vast amount of research has been focused on different prior models, the fidelity term is almost always chosen to be the least squares (LS) objective, that encourages fitting the linearly transformed optimization variable to the observations. In this paper, we examine a different fidelity term, which has been implicitly used by the recently proposed iterative denoising and backward projections (IDBP) framework. This term encourages agreement between the projection of the optimization variable onto the row space of the linear operator and the pseudoinverse of the linear operator ("back-projection") applied on the observations. We analytically examine the difference between the two fidelity terms for Tikhonov regularization and identify cases (such as a badly conditioned linear operator) where the new term has an advantage over the standard LS one. Moreover, we demonstrate empirically that the behavior of the two induced cost functions for sophisticated convex and non-convex priors, such as total-variation, BM3D, and deep generative models, correlates with the obtained theoretical analysis.

An Entropy-Based Algorithm with Nonlocal Residual Learning for Image Compressive Sensing Recovery

Image recovery from compressive sensing (CS) measurement data, especially noisy data has always been challenging due to its implicit ill-posed nature, thus, to seek a domain where a signal can exhibit a high degree of sparsity and to design an effective algorithm have drawn increasingly more attention. Among various sparsity-based models, structured or group sparsity often leads to more powerful signal reconstruction techniques. In this paper, we propose a novel entropy-based algorithm for CS recovery to enhance image sparsity through learning the group sparsity of residual. To reduce the residual of similar packed patches, the group sparsity of residual is described by a Laplacian scale mixture (LSM) model, therefore, each singular value of the residual of similar packed patches is modeled as a Laplacian distribution with a variable scale parameter, to exploit the benefits of high-order dependency among sparse coefficients. Due to the latent variables, the maximum a posteriori (MAP) estimation of the sparse coefficients cannot be obtained, thus, we design a loss function for expectation–maximization (EM) method based on relative entropy. In the frame of EM iteration, the sparse coefficients can be estimated with the denoising-based approximate message passing (D-AMP) algorithm. Experimental results have shown that the proposed algorithm can significantly outperform existing CS techniques for image recovery.

Image Restoration by Iterative Denoising and Backward Projections

Inverse problems appear in many applications, such as image deblurring and inpainting. The common approach to address them is to design a specific algorithm for each problem. The Plug-and-Play (P&P) framework, which has been recently introduced, allows solving general inverse problems by leveraging the impressive capabilities of existing denoising algorithms. While this fresh strategy has found many applications, a burdensome parameter tuning is often required in order to obtain high-quality results. In this paper, we propose an alternative method for solving inverse problems using off-the-shelf denoisers, which requires less parameter tuning. First, we transform a typical cost function, composed of fidelity and prior terms, into a closely related, novel optimization problem. Then, we propose an efficient minimization scheme with a P&P property, i.e., the prior term is handled solely by a denoising operation. Finally, we present an automatic tuning mechanism to set the method's parameters. We provide a theoretical analysis of the method and empirically demonstrate its competitiveness with task-specific techniques and the P&P approach for image inpainting and deblurring.

Normalization and pooling in hierarchical models of natural images

Divisive normalization and subunit pooling are two canonical classes of computation that have become widely used in descriptive (what) models of visual cortical processing. Normative (why) models from natural image statistics can help constrain the form and parameters of such classes of models. We focus on recent advances in two particular directions, namely deriving richer forms of divisive normalization, and advances in learning pooling from image statistics. We discuss the incorporation of such components into hierarchical models. We consider both hierarchical unsupervised learning from image statistics, and discriminative supervised learning in deep convolutional neural networks (CNNs). We further discuss studies on the utility and extensions of the convolutional architecture, which has also been adopted by recent descriptive models. We review the recent literature and discuss the current promises and gaps of using such approaches to gain a better understanding of how cortical neurons represent and process complex visual stimuli.

PURE-LET Image Deconvolution

We propose a non-iterative image deconvolution algorithm for data corrupted by Poisson or mixed Poisson-Gaussian noise. Many applications involve such a problem, ranging from astronomical to biological imaging. We parameterize the deconvolution process as a linear combination of elementary functions, termed as linear expansion of thresholds. This parameterization is then optimized by minimizing a robust estimate of the true mean squared error, the Poisson unbiased risk estimate. Each elementary function consists of a Wiener filtering followed by a pointwise thresholding of undecimated Haar wavelet coefficients. In contrast to existing approaches, the proposed algorithm merely amounts to solving a linear system of equations, which has a fast and exact solution. Simulation experiments over different types of convolution kernels and various noise levels indicate that the proposed method outperforms the state-of-the-art techniques, in terms of both restoration quality and computational complexity. Finally, we present some results on real confocal fluorescence microscopy images and demonstrate the potential applicability of the proposed method for improving the quality of these images.We propose a non-iterative image deconvolution algorithm for data corrupted by Poisson or mixed Poisson-Gaussian noise. Many applications involve such a problem, ranging from astronomical to biological imaging. We parameterize the deconvolution process as a linear combination of elementary functions, termed as linear expansion of thresholds. This parameterization is then optimized by minimizing a robust estimate of the true mean squared error, the Poisson unbiased risk estimate. Each elementary function consists of a Wiener filtering followed by a pointwise thresholding of undecimated Haar wavelet coefficients. In contrast to existing approaches, the proposed algorithm merely amounts to solving a linear system of equations, which has a fast and exact solution. Simulation experiments over different types of convolution kernels and various noise levels indicate that the proposed method outperforms the state-of-the-art techniques, in terms of both restoration quality and computational complexity. Finally, we present some results on real confocal fluorescence microscopy images and demonstrate the potential applicability of the proposed method for improving the quality of these images.

A robust iterative algorithm for image restoration

We present a new image restoration method by combining iterative VanCittert algorithm with noise reduction modeling. Our approach enables decoupling between deblurring and denoising during the restoration process, so allows any well-established noise reduction operator to be implemented in our model, independent of the VanCittert deblurring operation. Such an approach has led to an analytic expression for error estimation of the restored images in our method as well as simple parameter setting for real applications, both of which are hard to attain in many regularization-based methods. Numerical experiments show that our method can achieve good balance between structure recovery and noise reduction, and perform close to the level of the state of the art method and favorably compared to many other methods.

Efficient and Robust Image Restoration Using Multiple-Feature L2-Relaxed Sparse Analysis Priors

We propose a novel formulation for relaxed analysis-based sparsity in multiple dictionaries as a general type of prior for images, and apply it for Bayesian estimation in image restoration problems. Our formulation of a ℓ2-relaxed ℓ0 pseudo-norm prior allows for an especially simple maximum a posteriori estimation iterative marginal optimization algorithm, whose convergence we prove. We achieve a significant speedup over the direct (static) solution by using dynamically evolving parameters through the estimation loop. As an added heuristic twist, we fix in advance the number of iterations, and then empirically optimize the involved parameters according to two performance benchmarks. The resulting constrained dynamic method is not just fast and effective, it is also highly robust and flexible. First, it is able to provide an outstanding tradeoff between computational load and performance, in visual and objective, mean square error and structural similarity terms, for a large variety of degradation tests, using the same set of parameter values for all tests. Second, the performance benchmark can be easily adapted to specific types of degradation, image classes, and even performance criteria. Third, it allows for using simultaneously several dictionaries with complementary features. This unique combination makes ours a highly practical deconvolution method.

Video compressive sensing using Gaussian mixture models

A Gaussian mixture model (GMM)-based algorithm is proposed for video reconstruction from temporally compressed video measurements. The GMM is used to model spatio-temporal video patches, and the reconstruction can be efficiently computed based on analytic expressions. The GMM-based inversion method benefits from online adaptive learning and parallel computation. We demonstrate the efficacy of the proposed inversion method with videos reconstructed from simulated compressive video measurements, and from a real compressive video camera. We also use the GMM as a tool to investigate adaptive video compressive sensing, i.e., adaptive rate of temporal compression.

A Framework for Analysis of Computational Imaging Systems: Role of Signal Prior, Sensor Noise and Multiplexing

Over the last decade, a number of computational imaging (CI) systems have been proposed for tasks such as motion deblurring, defocus deblurring and multispectral imaging. These techniques increase the amount of light reaching the sensor via multiplexing and then undo the deleterious effects of multiplexing by appropriate reconstruction algorithms. Given the widespread appeal and the considerable enthusiasm generated by these techniques, a detailed performance analysis of the benefits conferred by this approach is important. Unfortunately, a detailed analysis of CI has proven to be a challenging problem because performance depends equally on three components: (1) the optical multiplexing, (2) the noise characteristics of the sensor, and (3) the reconstruction algorithm which typically uses signal priors. A few recent papers [12], [30], [49] have performed analysis taking multiplexing and noise characteristics into account. However, analysis of CI systems under state-of-the-art reconstruction algorithms, most of which exploit signal prior models, has proven to be unwieldy. In this paper, we present a comprehensive analysis framework incorporating all three components. In order to perform this analysis, we model the signal priors using a Gaussian Mixture Model (GMM). A GMM prior confers two unique characteristics. First, GMM satisfies the universal approximation property which says that any prior density function can be approximated to any fidelity using a GMM with appropriate number of mixtures. Second, a GMM prior lends itself to analytical tractability allowing us to derive simple expressions for the `minimum mean square error' (MMSE) which we use as a metric to characterize the performance of CI systems. We use our framework to analyze several previously proposed CI techniques (focal sweep, flutter shutter, parabolic exposure, etc.), giving conclusive answer to the question: `How much performance gain is due to use of a signal prior and how much is due to multiplexing? Our analysis also clearly shows that multiplexing provides significant performance gains above and beyond the gains obtained due to use of signal priors.

Dictionary learning approach for image deconvolution with variance estimation

In this paper, we propose a new dictionary learning approach for image deconvolution, which effectively integrates the Fourier regularization and dictionary learning technique into the deconvolution framework. Specifically, we propose an iterative algorithm with the decoupling of the deblurring and denoising steps in the restoration process. In the deblurring step, we involve a regularized inversion of the blur in the Fourier domain. Then we remove the colored noise using a dictionary learning method in the denoising step. In the denoising step, we propose an approach to update the estimation of noise variance for dictionary learning. We will show that this approach outperforms several state-of-the-art image deconvolution methods in terms of improvement in signal-to-noise ratio and visual quality.

Noise-aware image deconvolution with multidirectional filters

In this paper we propose an approach for handling noise in deconvolution algorithm based on multidirectional filters. Most image deconvolution techniques are sensitive to the noise. Even a small amount of noise will degrade the quality of image estimation dramatically. We found that by applying a directional low-pass filter to the blurred image, we can reduce the noise level while preserving the blur information in the orthogonal direction to the filter. So we apply a series of directional filters at different orientations to the blurred image, and a guided filter based edge-preserving image deconvolution is used to estimate an accurate Radon transform of the clear image from each filtered image. Finally, we reconstruct the original image using the inverse Radon transform. We compare our deconvolution algorithm with many competitive deconvolution techniques in terms of the improvement in signal-to-noise ratio and visual quality.

Stochastic speckle noise compensation in optical coherence tomography using non-stationary spline-based speckle noise modelling

Optical coherence tomography (OCT) allows for non-invasive 3D visualization of biological tissue at cellular level resolution. Often hindered by speckle noise, the visualization of important biological tissue details in OCT that can aid disease diagnosis can be improved by speckle noise compensation. A challenge with handling speckle noise is its inherent non-stationary nature, where the underlying noise characteristics vary with the spatial location. In this study, an innovative speckle noise compensation method is presented for handling the non-stationary traits of speckle noise in OCT imagery. The proposed approach centers on a non-stationary spline-based speckle noise modeling strategy to characterize the speckle noise. The novel method was applied to ultra high-resolution OCT (UHROCT) images of the human retina and corneo-scleral limbus acquired in-vivo that vary in tissue structure and optical properties. Test results showed improved performance of the proposed novel algorithm compared to a number of previously published speckle noise compensation approaches in terms of higher signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR) and better overall visual assessment.

Multi-Wiener SURE-LET deconvolution

In this paper, we propose a novel deconvolution algorithm based on the minimization of a regularized Stein's unbiased risk estimate (SURE), which is a good estimate of the mean squared error. We linearly parametrize the deconvolution process by using multiple Wiener filters as elementary functions, followed by undecimated Haar-wavelet thresholding. Due to the quadratic nature of SURE and the linear parametrization, the deconvolution problem finally boils down to solving a linear system of equations, which is very fast and exact. The linear coefficients, i.e., the solution of the linear system of equations, constitute the best approximation of the optimal processing on the Wiener-Haar-threshold basis that we consider. In addition, the proposed multi-Wiener SURE-LET approach is applicable for both periodic and symmetric boundary conditions, and can thus be used in various practical scenarios. The very competitive (both in computation time and quality) results show that the proposed algorithm, which can be interpreted as a kind of nonlinear Wiener processing, can be used as a basic tool for building more sophisticated deconvolution algorithms.

Single-image super-resolution reconstruction via learned geometric dictionaries and clustered sparse coding

Recently, single image super-resolution reconstruction (SISR) via sparse coding has attracted increasing interest. In this paper, we proposed a multiple-geometric-dictionaries-based clustered sparse coding scheme for SISR. Firstly, a large number of high-resolution (HR) image patches are randomly extracted from a set of example training images and clustered into several groups of "geometric patches," from which the corresponding "geometric dictionaries" are learned to further sparsely code each local patch in a low-resolution image. A clustering aggregation is performed on the HR patches recovered by different dictionaries, followed by a subsequent patch aggregation to estimate the HR image. Considering that there are often many repetitive image structures in an image, we add a self-similarity constraint on the recovered image in patch aggregation to reveal new features and details. Finally, the HR residual image is estimated by the proposed recovery method and compensated to better preserve the subtle details of the images. Some experiments test the proposed method on natural images, and the results show that the proposed method outperforms its counterparts in both visual fidelity and numerical measures.

Solving inverse problems with piecewise linear estimators: from Gaussian mixture models to structured sparsity

A general framework for solving image inverse problems with piecewise linear estimations is introduced in this paper. The approach is based on Gaussian mixture models, which are estimated via a maximum a posteriori expectation-maximization algorithm. A dual mathematical interpretation of the proposed framework with a structured sparse estimation is described, which shows that the resulting piecewise linear estimate stabilizes the estimation when compared with traditional sparse inverse problem techniques. We demonstrate that, in a number of image inverse problems, including interpolation, zooming, and deblurring of narrow kernels, the same simple and computationally efficient algorithm yields results in the same ballpark as that of the state of the art.

BM3D frames and variational image deblurring

A family of the block matching 3-D (BM3D) algorithms for various imaging problems has been recently proposed within the framework of nonlocal patchwise image modeling , . In this paper, we construct analysis and synthesis frames, formalizing BM3D image modeling, and use these frames to develop novel iterative deblurring algorithms. We consider two different formulations of the deblurring problem, i.e., one given by the minimization of the single-objective function and another based on the generalized Nash equilibrium (GNE) balance of two objective functions. The latter results in the algorithm where deblurring and denoising operations are decoupled. The convergence of the developed algorithms is proved. Simulation experiments show that the decoupled algorithm derived from the GNE formulation demonstrates the best numerical and visual results and shows superiority with respect to the state of the art in the field, confirming a valuable potential of BM3D-frames as an advanced image modeling tool.

Statistical modeling of 3-D natural scenes with application to Bayesian stereopsis

We studied the empirical distributions of luminance, range and disparity wavelet coefficients using a coregistered database of luminance and range images. The marginal distributions of range and disparity are observed to have high peaks and heavy tails, similar to the well-known properties of luminance wavelet coefficients. However, we found that the kurtosis of range and disparity coefficients is significantly larger than that of luminance coefficients. We used generalized Gaussian models to fit the empirical marginal distributions. We found that the marginal distribution of luminance coefficients have a shape parameter p between 0.6 and 0.8, while range and disparity coefficients have much smaller parameters p < 0.32, corresponding to a much higher peak. We also examined the conditional distributions of luminance, range and disparity coefficients. The magnitudes of luminance and range (disparity) coefficients show a clear positive correlation, which means, at a location with larger luminance variation, there is a higher probability of a larger range (disparity) variation. We also used generalized Gaussians to model the conditional distributions of luminance and range (disparity) coefficients. The values of the two shape parameters (p,s) reflect the observed luminance-range (disparity) dependency. As an example of the usefulness of luminance statistics conditioned on range statistics, we modified a well-known Bayesian stereo ranging algorithm using our natural scene statistics models, which improved its performance.

Spatial sparsity-induced prediction (SIP) for images and video: a simple way to reject structured interference

We propose a prediction technique that is geared toward forming successful estimates of a signal based on a correlated anchor signal that is contaminated with complex interference. The corruption in the anchor signal involves intensity modulations, linear distortions, structured interference, clutter, and noise just to name a few. The proposed setup reflects nontrivial prediction scenarios involving images and video frames where statistically related data is rendered ineffective for traditional methods due to cross-fades, blends, clutter, brightness variations, focus changes, and other complex transitions. Rather than trying to solve a difficult estimation problem involving nonstationary signal statistics, we obtain simple predictors in linear transform domain where the underlying signals are assumed to be sparse. We show that these simple predictors achieve surprisingly good performance and seamlessly allow successful predictions even under complicated cases. None of the interference parameters are estimated as our algorithm provides completely blind and automated operation. We provide a general formulation that allows for nonlinearities in the prediction loop and we consider prediction optimal decompositions. Beyond an extensive set of results on prediction and registration, the proposed method is also implemented to operate inside a state-of-the-art compression codec and results show significant improvements on scenes that are difficult to encode using traditional prediction techniques.

Fuzzy logic-based approach to wavelet denoising of 3D images produced by time-of-flight cameras

In this paper we present a new denoising method for the depth images of a 3D imaging sensor, based on the time-of-flight principle. We propose novel ways to use luminance-like information produced by a time-of flight camera along with depth images. Firstly, we propose a wavelet-based method for estimating the noise level in depth images, using luminance information. The underlying idea is that luminance carries information about the power of the optical signal reflected from the scene and is hence related to the signal-to-noise ratio for every pixel within the depth image. In this way, we can efficiently solve the difficult problem of estimating the non-stationary noise within the depth images. Secondly, we use luminance information to better restore object boundaries masked with noise in the depth images. Information from luminance images is introduced into the estimation formula through the use of fuzzy membership functions. In particular, we take the correlation between the measured depth and luminance into account, and the fact that edges (object boundaries) present in the depth image are likely to occur in the luminance image as well. The results on real 3D images show a significant improvement over the state-of-the-art in the field.

Image denoising using mixtures of projected Gaussian Scale Mixtures

We propose a new statistical model for image restoration in which neighborhoods of wavelet subbands are modeled by a discrete mixture of linear projected Gaussian Scale Mixtures (MPGSM). In each projection, a lower dimensional approximation of the local neighborhood is obtained, thereby modeling the strongest correlations in that neighborhood. The model is a generalization of the recently developed Mixture of GSM (MGSM) model, that offers a significant improvement both in PSNR and visually compared to the current state-of-the-art wavelet techniques. However, the computation cost is very high which hampers its use for practical purposes. We present a fast EM algorithm that takes advantage of the projection bases to speed up the algorithm. The results show that, when projecting on a fixed data-independent basis, even computational advantages with a limited loss of PSNR can be obtained with respect to the BLS-GSM denoising method, while data-dependent bases of Principle Components offer a higher denoising performance, both visually and in PSNR compared to the current wavelet-based state-of-the-art denoising methods.

Modeling multiscale subbands of photographic images with fields of Gaussian scale mixtures

The local statistical properties of photographic images, when represented in a multi-scale basis, have been described using Gaussian scale mixtures. Here, we use this local description as a substrate for constructing a global field of Gaussian scale mixtures (FoGSMs). Specifically, we model multi-scale subbands as a product of an exponentiated homogeneous Gaussian Markov random field (hGMRF) and a second independent hGMRF. We show that parameter estimation for this model is feasible, and that samples drawn from a FoGSM model have marginal and joint statistics similar to subband coefficients of photographic images. We develop an algorithm for removing additive Gaussian white noise based on the FoGSM model, and demonstrate denoising performance comparable with state-of-the-art methods.

Image modeling and denoising with orientation-adapted Gaussian scale mixtures

We develop a statistical model to describe the spatially varying behavior of local neighborhoods of coefficients in a multiscale image representation. Neighborhoods are modeled as samples of a multivariate Gaussian density that are modulated and rotated according to the values of two hidden random variables, thus allowing the model to adapt to the local amplitude and orientation of the signal. A third hidden variable selects between this oriented process and a nonoriented scale mixture of Gaussians process, thus providing adaptability to the local orientedness of the signal. Based on this model, we develop an optimal Bayesian least squares estimator for denoising images and show through simulations that the resulting method exhibits significant improvement over previously published results obtained with Gaussian scale mixtures.