Variational bayesian image restoration with a product of spatially weighted total variation image priors

Dehazing and deblurring of underwater images with heavy-tailed priors

The common problems of underwater images include color cast, haze effect, and the motion blur effect caused by turbulence and camera shake. To address these problems, research on color cast and haze and blur effects is carried out in this paper. Because red light has significant attenuation underwater, which could cause color cast of images, this paper proposes a red channel compensation method to solve this problem. This approach adaptively compensates for the red channel according to the pixel value of the red channel, successfully preventing excessive compensation. To address the haze effect of underwater images, combined with the physical model of underwater images, a variational method is introduced in the paper. This method can not only recover clear underwater images, but also refine the transmission map at the same time. Furthermore, the blind deconvolution method is adopted to deblur underwater images. First, the blur kernel of an underwater image is estimated, and then a clear underwater image is recovered based on the obtained blur kernel. Finally, qualitative and quantitative comparisons of the underwater images recovered by different methods are also carried out. From the qualitative perspective, the images recovered by our method have higher image sharpness and more outstanding details. The quantitative comparison results show that the images recovered using our method have higher scores according to various criteria. Therefore, on the whole, our method presents great advantages in comparison with others.

Heavy-Tailed Self-Similarity Modeling for Single Image Super Resolution

Self-similarity is a prominent characteristic of natural images that can play a major role when it comes to their denoising, restoration or compression. In this paper, we propose a novel probabilistic model that is based on the concept of image patch similarity and applied to the problem of Single Image Super Resolution. Based on this model, we derive a Variational Bayes algorithm, which super resolves low-resolution images, where the assumed distribution for the quantified similarity between two image patches is heavy-tailed. Moreover, we prove mathematically that the proposed algorithm is both an extended and superior version of the probabilistic Non-Local Means (NLM). Its prime advantage remains though, which is that it requires no training. A comparison of the proposed approach with state-of-the-art methods, using various quantitative metrics shows that it is almost on par, for images depicting rural themes and in terms of the Structural Similarity Index (SSIM) with the best performing methods that rely on trained deep learning models. On the other hand, it is clearly inferior to them, for urban themed images and in terms of all metrics, especially for the Mean-Squared-Error (MSE). In addition, qualitative evaluation of the proposed approach is performed using the Perceptual Index metric, which has been introduced to better mimic the human perception of the image quality. This evaluation favors our approach when compared to the best performing method that requires no training, even if they perform equally in qualitative terms, reinforcing the argument that MSE is not always an accurate metric for image quality.

A New Design in Iterative Image Deblurring for Improved Robustness and Performance

In many applications, image deblurring is a pre-requisite to improve the sharpness of an image before it can be further processed. Iterative methods are widely used for deblurring images but care must be taken to ensure that the iterative process is robust, meaning that the process does not diverge and reaches the solution reasonably fast, two goals that sometimes compete against each other. In practice, it remains challenging to choose parameters for the iterative process to be robust. We propose a new approach consisting of relaxed initialization and pixel-wise updates of the step size for iterative methods to achieve robustness. The first novel design of the approach is to modify the initialization of existing iterative methods to stop a noise term from being propagated throughout the iterative process. The second novel design is the introduction of a vectorized step size that is adaptively determined through the iteration to achieve higher stability and accuracy in the whole iterative process. The vectorized step size aims to update each pixel of an image individually, instead of updating all the pixels by the same factor. In this work, we implemented the above designs based on the Landweber method to test and demonstrate the new approach. Test results showed that the new approach can deblur images from noisy observations and achieve a low mean squared error with a more robust performance.

Simultaneous recovery of both bright and dim structures from noisy fluorescence microscopy images using a modified TV constraint

The quality and information content of biological images can be significantly enhanced by postacquisition processing using deconvolution and denoising. However, when imaging complex biological samples, such as neurons, stained with fluorescence labels, the signal level of different structures can differ by several orders of magnitude. This poses a challenge as current image reconstruction algorithms are focused on recovering low signals and generally have sample-dependent performance, requiring tedious manual tuning. This is one of the main hurdles for their wide adoption by nonspecialists. In this work, we modify the general constrained reconstruction method (in our case utilizing a total variation constraint) so that both bright and dim structures can drive the deconvolution with equal force. In this way, we can simultaneously obtain high-quality reconstruction across a wide range of signals within a single image or image sequence. The algorithm is tested on both simulated and experimental data. When compared with current state-of-art algorithms, our algorithm outperforms others in terms of maintaining the resolution in the high-signal areas and reducing artefacts in the low-signal areas. The algorithm was also tested on image sequences where one set of parameters are used to reconstruct all images, with blind evaluation by a group of biologists demonstrating a marked preference for the images produced by our method. This means that our method is suitable for batch processing of image sequences obtained from either spatial or temporal scanning. LAY DESCRIPTION: Fluorescence microscopy images of complex biological samples contain a wide range of signal levels. This signal variation leads current reconstruction algorithms, which aim to enhance the quality of the raw images, to have sample-dependent performance. In this work, we design a new optimization that allows the reconstruction to "pay equal eqattention to" both bright and dim structures. In this way, we can simultaneously recover both bright and dim structures within a single image or image sequence, as validated when the algorithm was quantitatively tested on both simulated and experimental data. When our method was evaluated alongside current state of art algorithms by a group of biologists, our algorithm was considered qualitatively superior.

A robust iterative algorithm for image restoration

We present a new image restoration method by combining iterative VanCittert algorithm with noise reduction modeling. Our approach enables decoupling between deblurring and denoising during the restoration process, so allows any well-established noise reduction operator to be implemented in our model, independent of the VanCittert deblurring operation. Such an approach has led to an analytic expression for error estimation of the restored images in our method as well as simple parameter setting for real applications, both of which are hard to attain in many regularization-based methods. Numerical experiments show that our method can achieve good balance between structure recovery and noise reduction, and perform close to the level of the state of the art method and favorably compared to many other methods.

Variational Bayesian Approach to Multiframe Image Restoration

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.

Blind Deconvolution With Model Discrepancies

Blind deconvolution is a strongly ill-posed problem comprising of simultaneous blur and image estimation. Recent advances in prior modeling and/or inference methodology led to methods that started to perform reasonably well in real cases. However, as we show here, they tend to fail if the convolution model is violated even in a small part of the image. Methods based on variational Bayesian inference play a prominent role. In this paper, we use this inference in combination with the same prior for noise, image, and blur that belongs to the family of independent non-identical Gaussian distributions, known as the automatic relevance determination prior. We identify several important properties of this prior useful in blind deconvolution, namely, enforcing non-negativity of the blur kernel, favoring sharp images over blurred ones, and most importantly, handling non-Gaussian noise, which, as we demonstrate, is common in real scenarios. The presented method handles discrepancies in the convolution model, and thus extends applicability of blind deconvolution to real scenarios, such as photos blurred by camera motion and incorrect focus.

Super-Resolution of Multi-Observed RGB-D Images Based on Nonlocal Regression and Total Variation

There is growing demand for accuracy in image processing and visualization, and the super-resolution (SR) technique for multi-observed RGB-D images has become popular, because it provides space-redundant information and produces a detailed reconstruction even with a large magnification factor. This technique has been thoroughly investigated in recent years. Nevertheless, technical challenges remain, such as finding sub-pixel correspondences with low-resolution (LR) observations, exploiting space-redundant information, formulating space homogeneity constraints, and leveraging cross-image similarities in structures. To address these challenges, this paper proposes a unified optimization framework to estimate both the super-resolved RGB image and the super-resolved depth image from the multi-observed LR RGB-D images using their correlations. Using depth-assisted cross-image correspondences, the RGB image SR problem is formulated as an effective regularization function by incorporating the normalized bilateral total variation regularizer, and it is efficiently solved by a first-order primal-dual algorithm. The depth image SR estimate can be obtained by minimizing a nonlocal regression-based energy, which integrates the structural cues of the super-resolved RGB image in a detail-preserving fashion. Essentially, our unified optimization framework uses the RGB image and depth image as a priori knowledge that the SR process uses for better accuracy. Our extensive experiments on public RGB-D benchmarks and real data and our quantitative comparison with several state-of-the-art methods demonstrate the superiority of our method in terms of accuracy, versatility, and reliability of details and sharp feature preservation.

Efficient and Robust Image Restoration Using Multiple-Feature L2-Relaxed Sparse Analysis Priors

We propose a novel formulation for relaxed analysis-based sparsity in multiple dictionaries as a general type of prior for images, and apply it for Bayesian estimation in image restoration problems. Our formulation of a ℓ2-relaxed ℓ0 pseudo-norm prior allows for an especially simple maximum a posteriori estimation iterative marginal optimization algorithm, whose convergence we prove. We achieve a significant speedup over the direct (static) solution by using dynamically evolving parameters through the estimation loop. As an added heuristic twist, we fix in advance the number of iterations, and then empirically optimize the involved parameters according to two performance benchmarks. The resulting constrained dynamic method is not just fast and effective, it is also highly robust and flexible. First, it is able to provide an outstanding tradeoff between computational load and performance, in visual and objective, mean square error and structural similarity terms, for a large variety of degradation tests, using the same set of parameter values for all tests. Second, the performance benchmark can be easily adapted to specific types of degradation, image classes, and even performance criteria. Third, it allows for using simultaneously several dictionaries with complementary features. This unique combination makes ours a highly practical deconvolution method.

Improving thoracic four-dimensional cone-beam CT reconstruction with anatomical-adaptive image regularization (AAIR)

Total-variation (TV) minimization reconstructions can significantly reduce noise and streaks in thoracic four-dimensional cone-beam computed tomography (4D CBCT) images compared to the Feldkamp-Davis-Kress (FDK) algorithm currently used in practice. TV minimization reconstructions are, however, prone to over-smoothing anatomical details and are also computationally inefficient. The aim of this study is to demonstrate a proof of concept that these disadvantages can be overcome by incorporating the general knowledge of the thoracic anatomy via anatomy segmentation into the reconstruction. The proposed method, referred as the anatomical-adaptive image regularization (AAIR) method, utilizes the adaptive-steepest-descent projection-onto-convex-sets (ASD-POCS) framework, but introduces an additional anatomy segmentation step in every iteration. The anatomy segmentation information is implemented in the reconstruction using a heuristic approach to adaptively suppress over-smoothing at anatomical structures of interest. The performance of AAIR depends on parameters describing the weighting of the anatomy segmentation prior and segmentation threshold values. A sensitivity study revealed that the reconstruction outcome is not sensitive to these parameters as long as they are chosen within a suitable range. AAIR was validated using a digital phantom and a patient scan and was compared to FDK, ASD-POCS and the prior image constrained compressed sensing (PICCS) method. For the phantom case, AAIR reconstruction was quantitatively shown to be the most accurate as indicated by the mean absolute difference and the structural similarity index. For the patient case, AAIR resulted in the highest signal-to-noise ratio (i.e. the lowest level of noise and streaking) and the highest contrast-to-noise ratios for the tumor and the bony anatomy (i.e. the best visibility of anatomical details). Overall, AAIR was much less prone to over-smoothing anatomical details compared to ASD-POCS and did not suffer from residual noise/streaking and motion blur migrated from the prior image as in PICCS. AAIR was also found to be more computationally efficient than both ASD-POCS and PICCS, with a reduction in computation time of over 50% compared to ASD-POCS. The use of anatomy segmentation was, for the first time, demonstrated to significantly improve image quality and computational efficiency for thoracic 4D CBCT reconstruction. Further developments are required to facilitate AAIR for practical use.

A general framework for regularized, similarity-based image restoration

Any image can be represented as a function defined on a weighted graph, in which the underlying structure of the image is encoded in kernel similarity and associated Laplacian matrices. In this paper, we develop an iterative graph-based framework for image restoration based on a new definition of the normalized graph Laplacian. We propose a cost function, which consists of a new data fidelity term and regularization term derived from the specific definition of the normalized graph Laplacian. The normalizing coefficients used in the definition of the Laplacian and associated regularization term are obtained using fast symmetry preserving matrix balancing. This results in some desired spectral properties for the normalized Laplacian such as being symmetric, positive semidefinite, and returning zero vector when applied to a constant image. Our algorithm comprises of outer and inner iterations, where in each outer iteration, the similarity weights are recomputed using the previous estimate and the updated objective function is minimized using inner conjugate gradient iterations. This procedure improves the performance of the algorithm for image deblurring, where we do not have access to a good initial estimate of the underlying image. In addition, the specific form of the cost function allows us to render the spectral analysis for the solutions of the corresponding linear equations. In addition, the proposed approach is general in the sense that we have shown its effectiveness for different restoration problems, including deblurring, denoising, and sharpening. Experimental results verify the effectiveness of the proposed algorithm on both synthetic and real examples.

Adaptive dictionary learning in sparse gradient domain for image recovery

Image recovery from undersampled data has always been challenging due to its implicit ill-posed nature but becomes fascinating with the emerging compressed sensing (CS) theory. This paper proposes a novel gradient based dictionary learning method for image recovery, which effectively integrates the popular total variation (TV) and dictionary learning technique into the same framework. Specifically, we first train dictionaries from the horizontal and vertical gradients of the image and then reconstruct the desired image using the sparse representations of both derivatives. The proposed method enables local features in the gradient images to be captured effectively, and can be viewed as an adaptive extension of the TV regularization. The results of various experiments on MR images consistently demonstrate that the proposed algorithm efficiently recovers images and presents advantages over the current leading CS reconstruction approaches.

Parameter estimation for blind and non-blind deblurring using residual whiteness measures

Image deblurring (ID) is an ill-posed problem typically addressed by using regularization, or prior knowledge, on the unknown image (and also on the blur operator, in the blind case). ID is often formulated as an optimization problem, where the objective function includes a data term encouraging the estimated image (and blur, in blind ID) to explain the observed data well (typically, the squared norm of a residual) plus a regularizer that penalizes solutions deemed undesirable. The performance of this approach depends critically (among other things) on the relative weight of the regularizer (the regularization parameter) and on the number of iterations of the algorithm used to address the optimization problem. In this paper, we propose new criteria for adjusting the regularization parameter and/or the number of iterations of ID algorithms. The rationale is that if the recovered image (and blur, in blind ID) is well estimated, the residual image is spectrally white; contrarily, a poorly deblurred image typically exhibits structured artifacts (e.g., ringing, oversmoothness), yielding residuals that are not spectrally white. The proposed criterion is particularly well suited to a recent blind ID algorithm that uses continuation, i.e., slowly decreases the regularization parameter along the iterations; in this case, choosing this parameter and deciding when to stop are one and the same thing. Our experiments show that the proposed whiteness-based criteria yield improvements in SNR, on average, only 0.15 dB below those obtained by (clairvoyantly) stopping the algorithm at the best SNR. We also illustrate the proposed criteria on non-blind ID, reporting results that are competitive with state-of-the-art criteria (such as Monte Carlo-based GSURE and projected SURE), which, however, are not applicable for blind ID.

Iterative nonlocal total variation regularization method for image restoration

In this paper, a Bregman iteration based total variation image restoration algorithm is proposed. Based on the Bregman iteration, the algorithm splits the original total variation problem into sub-problems that are easy to solve. Moreover, non-local regularization is introduced into the proposed algorithm, and a method to choose the non-local filter parameter locally and adaptively is proposed. Experiment results show that the proposed algorithms outperform some other regularization methods.

Regional spatially adaptive total variation super-resolution with spatial information filtering and clustering

Total variation is used as a popular and effective image prior model in the regularization-based image processing fields. However, as the total variation model favors a piecewise constant solution, the processing result under high noise intensity in the flat regions of the image is often poor, and some pseudoedges are produced. In this paper, we develop a regional spatially adaptive total variation model. Initially, the spatial information is extracted based on each pixel, and then two filtering processes are added to suppress the effect of pseudoedges. In addition, the spatial information weight is constructed and classified with k-means clustering, and the regularization strength in each region is controlled by the clustering center value. The experimental results, on both simulated and real datasets, show that the proposed approach can effectively reduce the pseudoedges of the total variation regularization in the flat regions, and maintain the partial smoothness of the high-resolution image. More importantly, compared with the traditional pixel-based spatial information adaptive approach, the proposed region-based spatial information adaptive total variation model can better avoid the effect of noise on the spatial information extraction, and maintains robustness with changes in the noise intensity in the super-resolution process.

Iterative image restoration using nonstationary priors

In this paper, we propose an algorithm for image restoration based on fusing nonstationary edge-preserving priors. We develop a Bayesian modeling followed by an evidence approximation inference approach for deriving the analytic foundations of the proposed restoration method. Through a series of approximations, the final implementation of the proposed image restoration algorithm is iterative and takes advantage of the Fourier domain. Simulation results over a variety of blurred and noisy standard test images indicate that the presented method comfortably surpasses the current state-of-the-art image restoration for compactly supported degradations. We finally present experimental results by digitally refocusing images captured with controlled defocus, successfully confirming the ability of the proposed restoration algorithm in recovering extra features and rich details, while still preserving edges.

BM3D frames and variational image deblurring

A family of the block matching 3-D (BM3D) algorithms for various imaging problems has been recently proposed within the framework of nonlocal patchwise image modeling , . In this paper, we construct analysis and synthesis frames, formalizing BM3D image modeling, and use these frames to develop novel iterative deblurring algorithms. We consider two different formulations of the deblurring problem, i.e., one given by the minimization of the single-objective function and another based on the generalized Nash equilibrium (GNE) balance of two objective functions. The latter results in the algorithm where deblurring and denoising operations are decoupled. The convergence of the developed algorithms is proved. Simulation experiments show that the decoupled algorithm derived from the GNE formulation demonstrates the best numerical and visual results and shows superiority with respect to the state of the art in the field, confirming a valuable potential of BM3D-frames as an advanced image modeling tool.

Bayesian estimation for optimized structured illumination microscopy

Structured illumination microscopy is a recent imaging technique that aims at going beyond the classical optical resolution by reconstructing high-resolution (HR) images from low-resolution (LR) images acquired through modulation of the transfer function of the microscope. The classical implementation has a number of drawbacks, such as requiring a large number of images to be acquired and parameters to be manually set in an ad-hoc manner that have, until now, hampered its wide dissemination. Here, we present a new framework based on a Bayesian inverse problem formulation approach that enables the computation of one HR image from a reduced number of LR images and has no specific constraints on the modulation. Moreover, it permits to automatically estimate the optimal reconstruction hyperparameters and to compute an uncertainty bound on the estimated values. We demonstrate through numerical evaluations on simulated data and examples on real microscopy data that our approach represents a decisive advance for a wider use of HR microscopy through structured illumination.

Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization

As a powerful statistical image modeling technique, sparse representation has been successfully used in various image restoration applications. The success of sparse representation owes to the development of the l(1)-norm optimization techniques and the fact that natural images are intrinsically sparse in some domains. The image restoration quality largely depends on whether the employed sparse domain can represent well the underlying image. Considering that the contents can vary significantly across different images or different patches in a single image, we propose to learn various sets of bases from a precollected dataset of example image patches, and then, for a given patch to be processed, one set of bases are adaptively selected to characterize the local sparse domain. We further introduce two adaptive regularization terms into the sparse representation framework. First, a set of autoregressive (AR) models are learned from the dataset of example image patches. The best fitted AR models to a given patch are adaptively selected to regularize the image local structures. Second, the image nonlocal self-similarity is introduced as another regularization term. In addition, the sparsity regularization parameter is adaptively estimated for better image restoration performance. Extensive experiments on image deblurring and super-resolution validate that by using adaptive sparse domain selection and adaptive regularization, the proposed method achieves much better results than many state-of-the-art algorithms in terms of both PSNR and visual perception.