External Prior Guided Internal Prior Learning for Real-World Noisy Image Denoising

Overview of Research on Digital Image Denoising Methods

During image collection, images are often polluted by noise because of imaging conditions and equipment limitations. Images are also disturbed by external noise during compression and transmission, which adversely affects consequent processing, like image segmentation, target recognition, and text detection. A two-dimensional amplitude image is one of the most common image categories, which is widely used in people's daily life and work. Research on this kind of image-denoising algorithm is a hotspot in the field of image denoising. Conventional denoising methods mainly use the nonlocal self-similarity of images and sparser representatives in the converted domain for image denoising. In particular, the three-dimensional block matching filtering (BM3D) algorithm not only effectively removes the image noise but also better retains the detailed information in the image. As artificial intelligence develops, the deep learning-based image-denoising method has become an important research direction. This review provides a general overview and comparison of traditional image-denoising methods and deep neural network-based image-denoising methods. First, the essential framework of classic traditional denoising and deep neural network denoising approaches is presented, and the denoising approaches are classified and summarized. Then, existing denoising methods are compared with quantitative and qualitative analyses on a public denoising dataset. Finally, we point out some potential challenges and directions for future research in the field of image denoising. This review can help researchers clearly understand the differences between various image-denoising algorithms, which not only helps them to choose suitable algorithms or improve and innovate on this basis but also provides research ideas and directions for subsequent research in this field.

A two-stage HDR reconstruction pipeline for extreme dark-light RGGB images

RGGB sensor arrays are commonly used in digital cameras and mobile photography. However, images of extreme dark-light conditions often suffer from insufficient exposure because the sensor receives insufficient light. The existing methods mainly employ U-Net variants, multi-stage camera parameter simulation, or image parameter processing to address this issue. However, those methods usually apply color adjustments evenly across the entire image, which may cause extensive blue or green noise artifacts, especially in images with dark backgrounds. This study attacks the problem by proposing a novel multi-step process for image enhancement. The pipeline starts with a self-attention U-Net for initial color restoration and applies a Color Correction Matrix (CCM). Thereafter, High Dynamic Range (HDR) image reconstruction techniques are utilized to improve exposure using various Camera Response Functions (CRFs). After removing under- and over-exposed frames, pseudo-HDR images are created through multi-frame fusion. Also, a comparative analysis is conducted based on a standard dataset, and the results show that the proposed approach performs better in creating well-exposed images and improves the Peak-Signal-to-Noise Ratio (PSNR) by 0.16 dB compared to the benchmark methods.

Cross-Domain Denoising for Low-Dose Multi-Frame Spiral Computed Tomography

Computed tomography (CT) has been used worldwide as a non-invasive test to assist in diagnosis. However, the ionizing nature of X-ray exposure raises concerns about potential health risks such as cancer. The desire for lower radiation doses has driven researchers to improve reconstruction quality. Although previous studies on low-dose computed tomography (LDCT) denoising have demonstrated the effectiveness of learning-based methods, most were developed on the simulated data. However, the real-world scenario differs significantly from the simulation domain, especially when using the multi-slice spiral scanner geometry. This paper proposes a two-stage method for the commercially available multi-slice spiral CT scanners that better exploits the complete reconstruction pipeline for LDCT denoising across different domains. Our approach makes good use of the high redundancy of multi-slice projections and the volumetric reconstructions while leveraging the over-smoothing issue in conventional cascaded frameworks caused by aggressive denoising. The dedicated design also provides a more explicit interpretation of the data flow. Extensive experiments on various datasets showed that the proposed method could remove up to 70% of noise without compromised spatial resolution, while subjective evaluations by two experienced radiologists further supported its superior performance against state-of-the-art methods in clinical practice. Code is available at https://github.com/YCL92/TMD-LDCT.

Low-illumination and noisy bridge crack image restoration by deep CNN denoiser and normalized flow module

When applying deep learning and image processing techniques for bridge crack detection, the obtained images in real-world scenarios have severe image degradation problem. This study focuses on restoring low-illumination bridge crack images corrupted by noise to improve the accuracy of subsequent crack detection and semantic segmentation. The proposed algorithm consists of a deep CNN denoiser and a normalized flow-based brightness enhancement module. By taking the noise spectrum as an input, the deep CNN denoiser restores image at a broad range of noise levels. The normalized flow module, employs a conditional encoder and a reversible network to map the distribution of normally exposed images to a Gaussian distribution, effectively improving the image brightness. Extensive experiments have demonstrated the approach can usefully recover low-illumination images corrupted by noise compared to the state-of-the-art methods. Furthermore, the algorithm presented in this study can also be applied to other image quality restoration with high generalization and robust abilities. And the semantic segmentation accuracy of the restored image is significantly improved.

The Impact of Noise and Brightness on Object Detection Methods

The application of deep learning to image and video processing has become increasingly popular nowadays. Employing well-known pre-trained neural networks for detecting and classifying objects in images is beneficial in a wide range of application fields. However, diverse impediments may degrade the performance achieved by those neural networks. Particularly, Gaussian noise and brightness, among others, may be presented on images as sensor noise due to the limitations of image acquisition devices. In this work, we study the effect of the most representative noise types and brightness alterations on images in the performance of several state-of-the-art object detectors, such as YOLO or Faster-RCNN. Different experiments have been carried out and the results demonstrate how these adversities deteriorate their performance. Moreover, it is found that the size of objects to be detected is a factor that, together with noise and brightness factors, has a considerable impact on their performance.

Multichannel Image Completion With Mixture Noise: Adaptive Sparse Low-Rank Tensor Subspace Meets Nonlocal Self-Similarity

Multichannel image completion with mixture noise is a common but complex problem in the fields of machine learning, image processing, and computer vision. Most existing algorithms devote to explore global low-rank information and fail to optimize local and joint-mode structures, which may lead to oversmooth restoration results or lower quality restoration details. In this study, we propose a novel model to deal with multichannel image completion with mixture noise based on adaptive sparse low-rank tensor subspace and nonlocal self-similarity (ASLTS-NS). In the proposed model, a nonlocal similar patch matching framework cooperating with Tucker decomposition is used to explore information of global and joint modes and optimize the local structure for improving restoration quality. In order to enhance the robustness of low-rank decomposition to data missing and mixture noise, we present an adaptive sparse low-rank regularization to construct robust tensor subspace for self-weighing importance of different modes and capturing a stable inherent structure. In addition, joint tensor Frobenius and l1 regularizations are exploited to control two different types of noise. Based on alternating directions method of multipliers (ADMM), a convergent learning algorithm is designed to solve this model. Experimental results on three different types of multichannel image sets demonstrate the advantages of ASLTS-NS under five complex scenarios.

Flexible and Generalized Real Photograph Denoising Exploiting Dual Meta Attention

Supervised deep learning techniques have been widely explored in real photograph denoising and achieved noticeable performances. However, being subject to specific training data, most current image denoising algorithms can easily be restricted to certain noisy types and exhibit poor generalizability across testing sets. To address this issue, we propose a novel flexible and well-generalized approach, coined as dual meta attention network (DMANet). The DMANet is mainly composed of a cascade of the self-meta attention blocks (SMABs) and collaborative-meta attention blocks (CMABs). These two blocks have two forms of advantages. First, they simultaneously take both spatial and channel attention into account, allowing our model to better exploit more informative feature interdependencies. Second, the attention blocks are embedded with the meta-subnetwork, which is based on metalearning and supports dynamic weight generation. Such a scheme can provide a beneficial means for self and collaborative updating of the attention maps on-the-fly. Instead of directly stacking the SMABs and CMABs to form a deep network architecture, we further devise a three-stage learning framework, where different blocks are utilized for each feature extraction stage according to the individual characteristics of SMAB and CMAB. On five real datasets, we demonstrate the superiority of our approach against the state of the art. Unlike most existing image denoising algorithms, our DMANet not only possesses a good generalization capability but can also be flexibly used to cope with the unknown and complex real noises, making it highly competitive for practical applications.

A secondary EWMA-based dictionary learning algorithm for polynomial phase signal denoising

Under the influence of additive white Gaussian noise, sparse representation cannot effectively remove noise associated with the polynomial phase signal (PPS) via most dictionary learning algorithms whose training data come from the noisy signal, such as K-SVD and RLS-DLA. In this paper, we present a novel dictionary learning algorithm based on secondary exponentially weighted moving average (SEWMA) to denoise PPS. In the proposed algorithm, we first estimate the signal-to-noise (SNR) of the PPS to set the optimal rate of a weighted decline using covariance matrix model. Second we use RLS-DLA to train the dictionary. Thirdly, SEWMA is used to refine atoms in the learned dictionary. In this way, the SNR of the reconstructed signal obtained using the proposed algorithm is clearly higher than that of other algorithms, whereas the mean squared error is lower than that of other algorithms. To obtain the optimal denoising performance, the optimal rate of a weighted decline is set based on the estimated SNR. Simulation results show that the proposed method outperforms the K-SVD, RLS-DLA in mean square error and the SNR.

The Time-of-Arrival Offset Estimation in Neural Network Atomic Denoising in Wireless Location

With the increasing demand for wireless location services, it is of great interest to reduce the deployment cost of positioning systems. For this reason, indoor positioning based on WiFi has attracted great attention. Compared with the received signal strength indicator (RSSI), channel state information (CSI) captures the radio propagation environment more accurately. However, it is necessary to take signal bandwidth, interferences, noises, and other factors into account for accurate CSI-based positioning. In this paper, we propose a novel dictionary filtering method that uses the direct weight determination method of a neural network to denoise the dictionary and uses compressive sensing (CS) to extract the channel impulse response (CIR). A high-precision time-of-arrival (TOA) is then estimated by peak search. A median value filtering algorithm is used to locate target devices based on the time-difference-of-arrival (TDOA) technique. We demonstrate the superior performance of the proposed scheme experimentally, using data collected with a WiFi positioning testbed. Compared with the fingerprint location method, the proposed location method does not require a site survey in advance and therefore enables a fast system deployment.

PID Controller-Guided Attention Neural Network Learning for Fast and Effective Real Photographs Denoising

Real photograph denoising is extremely challenging in low-level computer vision since the noise is sophisticated and cannot be fully modeled by explicit distributions. Although deep-learning techniques have been actively explored for this issue and achieved convincing results, most of the networks may cause vanishing or exploding gradients, and usually entail more time and memory to obtain a remarkable performance. This article overcomes these challenges and presents a novel network, namely, PID controller guide attention neural network (PAN-Net), taking advantage of both the proportional-integral-derivative (PID) controller and attention neural network for real photograph denoising. First, a PID-attention network (PID-AN) is built to learn and exploit discriminative image features. Meanwhile, we devise a dynamic learning scheme by linking the neural network and control action, which significantly improves the robustness and adaptability of PID-AN. Second, we explore both the residual structure and share-source skip connections to stack the PID-ANs. Such a framework provides a flexible way to feature residual learning, enabling us to facilitate the network training and boost the denoising performance. Extensive experiments show that our PAN-Net achieves superior denoising results against the state-of-the-art in terms of image quality and efficiency.

A Nonlocal Self-Similarity-Based Weighted Tensor Low-Rank Decomposition for Multichannel Image Completion With Mixture Noise

Multichannel image completion with mixture noise is a challenging problem in the fields of machine learning, computer vision, image processing, and data mining. Traditional image completion models are not appropriate to deal with this problem directly since their reconstruction priors may mismatch corruption priors. To address this issue, we propose a novel nonlocal self-similarity-based weighted tensor low-rank decomposition (NSWTLD) model that can achieve global optimization and local enhancement. In the proposed model, based on the corruption priors and the reconstruction priors, a pixel weighting strategy is given to characterize the joint effects of missing data, the Gaussian noise, and the impulse noise. To discover and utilize the accurate nonlocal self-similarity information to enhance the restoration quality of the details, the traditional nonlocal learning framework is optimized by employing improved index determination of patch group and handling strip noise caused by patch overlapping. In addition, an efficient and convergent algorithm is presented to solve the NSWTLD model. Comprehensive experiments are conducted on four types of multichannel images under various corruption scenarios. The results demonstrate the efficiency and effectiveness of the proposed model.

Meta PID Attention Network for Flexible and Efficient Real-World Noisy Image Denoising

Recent deep convolutional neural networks for real-world noisy image denoising have shown a huge boost in performance by training a well-engineered network over external image pairs. However, most of these methods are generally trained with supervision. Once the testing data is no longer compatible with the training conditions, they can exhibit poor generalization and easily result in severe overfitting or degrading performances. To tackle this barrier, we propose a novel denoising algorithm, dubbed as Meta PID Attention Network (MPA-Net). Our MPA-Net is built based upon stacking Meta PID Attention Modules (MPAMs). In each MPAM, we utilize a second-order attention module (SAM) to exploit the channel-wise feature correlations with second-order statistics, which are then adaptively updated via a proportional-integral-derivative (PID) guided meta-learning framework. This learning framework exerts the unique property of the PID controller and meta-learning scheme to dynamically generate filter weights for beneficial update of the extracted features within a feedback control system. Moreover, the dynamic nature of the framework enables the generated weights to be flexibly tweaked according to the input at test time. Thus, MPAM not only achieves discriminative feature learning, but also facilitates a robust generalization ability on distinct noises for real images. Extensive experiments on ten datasets are conducted to inspect the effectiveness of the proposed MPA-Net quantitatively and qualitatively, which demonstrates both its superior denoising performance and promising generalization ability that goes beyond those of the state-of-the-art denoising methods.

Tensor Completion via Complementary Global, Local, and Nonlocal Priors

Completing missing entries in multidimensional visual data is a typical ill-posed problem that requires appropriate exploitation of prior information of the underlying data. Commonly used priors can be roughly categorized into three classes: global tensor low-rankness, local properties, and nonlocal self-similarity (NSS); most existing works utilize one or two of them to implement completion. Naturally, there arises an interesting question: can one concurrently make use of multiple priors in a unified way, such that they can collaborate with each other to achieve better performance? This work gives a positive answer by formulating a novel tensor completion framework which can simultaneously take advantage of the global-local-nonlocal priors. In the proposed framework, the tensor train (TT) rank is adopted to characterize the global correlation; meanwhile, two Plug-and-Play (PnP) denoisers, including a convolutional neural network (CNN) denoiser and the color block-matching and 3 D filtering (CBM3D) denoiser, are incorporated to preserve local details and exploit NSS, respectively. Then, we design a proximal alternating minimization algorithm to efficiently solve this model under the PnP framework. Under mild conditions, we establish the convergence guarantee of the proposed algorithm. Extensive experiments show that these priors organically benefit from each other to achieve state-of-the-art performance both quantitatively and qualitatively.

An Advanced Noise Reduction and Edge Enhancement Algorithm

Complementary metal-oxide-semiconductor (CMOS) image sensors can cause noise in images collected or transmitted in unfavorable environments, especially low-illumination scenarios. Numerous approaches have been developed to solve the problem of image noise removal. However, producing natural and high-quality denoised images remains a crucial challenge. To meet this challenge, we introduce a novel approach for image denoising with the following three main contributions. First, we devise a deep image prior-based module that can produce a noise-reduced image as well as a contrast-enhanced denoised one from a noisy input image. Second, the produced images are passed through a proposed image fusion (IF) module based on Laplacian pyramid decomposition to combine them and prevent noise amplification and color shift. Finally, we introduce a progressive refinement (PR) module, which adopts the summed-area tables to take advantage of spatially correlated information for edge and image quality enhancement. Qualitative and quantitative evaluations demonstrate the efficiency, superiority, and robustness of our proposed method.

Infrared image impulse noise suppression using tensor robust principal component analysis and truncated total variation

Infrared image denoising is an essential inverse problem that has been widely applied in many fields. However, when suppressing impulse noise, existing methods lead to blurred object details and loss of image information. Moreover, computational efficiency is another challenge for existing methods when processing infrared images with large resolution. An infrared image impulse-noise-suppression method is introduced based on tensor robust principal component analysis. Specifically, we propose a randomized tensor singular-value thresholding algorithm to solve the tensor kernel norm based on the matrix stochastic singular-value decomposition and tensor singular-value threshold. Combined with the image blocking, it can not only ensure the denoising performance but also greatly improve the algorithm's efficiency. Finally, truncated total variation is applied to improve the smoothness of the denoised image. Experimental results indicate that the proposed algorithm outperforms the state-of-the-art methods in computational efficiency, denoising effect, and detail feature preservation.

Scaled Simplex Representation for Subspace Clustering

The self-expressive property of data points, that is, each data point can be linearly represented by the other data points in the same subspace, has proven effective in leading subspace clustering (SC) methods. Most self-expressive methods usually construct a feasible affinity matrix from a coefficient matrix, obtained by solving an optimization problem. However, the negative entries in the coefficient matrix are forced to be positive when constructing the affinity matrix via exponentiation, absolute symmetrization, or squaring operations. This consequently damages the inherent correlations among the data. Besides, the affine constraint used in these methods is not flexible enough for practical applications. To overcome these problems, in this article, we introduce a scaled simplex representation (SSR) for the SC problem. Specifically, the non-negative constraint is used to make the coefficient matrix physically meaningful, and the coefficient vector is constrained to be summed up to a scalar to make it more discriminative. The proposed SSR-based SC (SSRSC) model is reformulated as a linear equality-constrained problem, which is solved efficiently under the alternating direction method of multipliers framework. Experiments on benchmark datasets demonstrate that the proposed SSRSC algorithm is very efficient and outperforms the state-of-the-art SC methods on accuracy. The code can be found at https://github.com/csjunxu/SSRSC.

CDAE: A Cascade of Denoising Autoencoders for Noise Reduction in the Clustering of Single-Particle Cryo-EM Images

As an emerging technology, cryo-electron microscopy (cryo-EM) has attracted more and more research interests from both structural biology and computer science, because many challenging computational tasks are involved in the processing of cryo-EM images. An important image processing step is to cluster the 2D cryo-EM images according to their projection angles, then the cluster mean images are used for the subsequent 3D reconstruction. However, cryo-EM images are quite noisy and denoising them is not easy, because the noise is a complicated mixture from samples and hardware. In this study, we design an effective cryo-EM image denoising model, CDAE, i.e., a cascade of denoising autoencoders. The new model comprises stacked blocks of deep neural networks to reduce noise in a progressive manner. Each block contains a convolutional autoencoder, pre-trained by simulated data of different SNRs and fine-tuned by target data set. We assess this new model on three simulated test sets and a real data set. CDAE achieves very competitive PSNR (peak signal-to-noise ratio) in the comparison of the state-of-the-art image denoising methods. Moreover, the denoised images have significantly enhanced clustering results compared to original image features or high-level abstraction features obtained by other deep neural networks. Both quantitative and visualized results demonstrate the good performance of CDAE for the noise reduction in clustering single-particle cryo-EM images.

Real-World Image Denoising with Deep Boosting

We propose a Deep Boosting Framework (DBF) for real-world image denoising by integrating the deep learning technique into the boosting algorithm. The DBF replaces conventional handcrafted boosting units by elaborate convolutional neural networks, which brings notable advantages in terms of both performance and speed. We design a lightweight Dense Dilated Fusion Network (DDFN) as an embodiment of the boosting unit, which addresses the vanishing of gradients during training due to the cascading of networks while promoting the efficiency of limited parameters. The capabilities of the proposed method are first validated on several representative simulation tasks including non-blind and blind Gaussian denoising and JPEG image deblocking. We then focus on a practical scenario to tackle with the complex and challenging real-world noise. To facilitate leaning-based methods including ours, we build a new Real-world Image Denoising (RID) dataset, which contains 200 pairs of high-resolution images with diverse scene content under various shooting conditions. Moreover, we conduct comprehensive analysis on the domain shift issue for real-world denoising and propose an effective one-shot domain transfer scheme to address this issue. Comprehensive experiments on widely used benchmarks demonstrate that the proposed method significantly surpasses existing methods on the task of real-world image denoising. Code and dataset are available at https://github.com/ngchc/deepBoosting.

Noisy-As-Clean: Learning Self-supervised Denoising from Corrupted Image

Supervised deep networks have achieved promising performance on image denoising, by learning image priors and noise statistics on plenty pairs of noisy and clean images. Unsupervised denoising networks are trained with only noisy images. However, for an unseen corrupted image, both supervised and unsupervised networks ignore either its particular image prior, the noise statistics, or both. That is, the networks learned from external images inherently suffer from a domain gap problem: the image priors and noise statistics are very different between the training and test images. This problem becomes more clear when dealing with the signal dependent realistic noise. To circumvent this problem, in this work, we propose a novel "Noisy-As-Clean" (NAC) strategy of training self-supervised denoising networks. Specifically, the corrupted test image is directly taken as the "clean" target, while the inputs are synthetic images consisted of this corrupted image and a second yet similar corruption. A simple but useful observation on our NAC is: as long as the noise is weak, it is feasible to learn a self-supervised network only with the corrupted image, approximating the optimal parameters of a supervised network learned with pairs of noisy and clean images. Experiments on synthetic and realistic noise removal demonstrate that, the DnCNN and ResNet networks trained with our self-supervised NAC strategy achieve comparable or better performance than the original ones and previous supervised/unsupervised/self-supervised networks. The code is publicly available at https://github.com/csjunxu/Noisy-As-Clean.

Deep learning on image denoising: An overview

Deep learning techniques have received much attention in the area of image denoising. However, there are substantial differences in the various types of deep learning methods dealing with image denoising. Specifically, discriminative learning based on deep learning can ably address the issue of Gaussian noise. Optimization models based on deep learning are effective in estimating the real noise. However, there has thus far been little related research to summarize the different deep learning techniques for image denoising. In this paper, we offer a comparative study of deep techniques in image denoising. We first classify the deep convolutional neural networks (CNNs) for additive white noisy images; the deep CNNs for real noisy images; the deep CNNs for blind denoising and the deep CNNs for hybrid noisy images, which represents the combination of noisy, blurred and low-resolution images. Then, we analyze the motivations and principles of the different types of deep learning methods. Next, we compare the state-of-the-art methods on public denoising datasets in terms of quantitative and qualitative analyses. Finally, we point out some potential challenges and directions of future research.

Conditional Variational Image Deraining

Image deraining is an important yet challenging image processing task. Though deterministic image deraining methods are developed with encouraging performance, they are infeasible to learn flexible representations for probabilistic inference and diverse predictions. Besides, rain intensity varies both in spatial locations and across color channels, making this task more difficult. In this paper, we propose a Conditional Variational Image Deraining (CVID) network for better deraining performance, leveraging the exclusive generative ability of Conditional Variational Auto-Encoder (CVAE) on providing diverse predictions for the rainy image. To perform spatially adaptive deraining, we propose a spatial density estimation (SDE) module to estimate a rain density map for each image. Since rain density varies across different color channels, we also propose a channel-wise (CW) deraining scheme. Experiments on synthesized and real-world datasets show that the proposed CVID network achieves much better performance than previous deterministic methods on image deraining. Extensive ablation studies validate the effectiveness of the proposed SDE module and CW scheme in our CVID network. The code is available at https://github.com/Yingjun-Du/VID.

High-ISO Long-Exposure Image Denoising Based on Quantitative Blob Characterization

Blob detection and image denoising are fundamental, sometimes related tasks in computer vision. In this paper, we present a computational method to quantitatively measure blob characteristics using normalized unilateral second-order Gaussian kernels. This method suppresses non-blob structures while yielding a quantitative measurement of the position, prominence and scale of blobs, which can facilitate the tasks of blob reconstruction and blob reduction. Subsequently, we propose a denoising scheme to address high-ISO long-exposure noise, which sometimes spatially shows a blob appearance, employing a blob reduction procedure as a cheap preprocessing for conventional denoising methods. We apply the proposed denoising methods to real-world noisy images as well as standard images that are corrupted by real noise. The experimental results demonstrate the superiority of the proposed methods over state-of-the-art denoising methods.

STAR: A Structure and Texture Aware Retinex Model

Retinex theory is developed mainly to decompose an image into the illumination and reflectance components by analyzing local image derivatives. In this theory, larger derivatives are attributed to the changes in reflectance, while smaller derivatives are emerged in the smooth illumination. In this paper, we utilize exponentiated local derivatives (with an exponent γ) of an observed image to generate its structure map and texture map. The structure map is produced by been amplified with γ > 1, while the texture map is generated by been shrank with γ < 1. To this end, we design exponential filters for the local derivatives, and present their capability on extracting accurate structure and texture maps, influenced by the choices of exponents γ. The extracted structure and texture maps are employed to regularize the illumination and reflectance components in Retinex decomposition. A novel Structure and Texture Aware Retinex (STAR) model is further proposed for illumination and reflectance decomposition of a single image. We solve the STAR model by an alternating optimization algorithm. Each sub-problem is transformed into a vectorized least squares regression, with closed-form solutions. Comprehensive experiments on commonly tested datasets demonstrate that, the proposed STAR model produce better quantitative and qualitative performance than previous competing methods, on illumination and reflectance decomposition, low-light image enhancement, and color correction. The code is publicly available at https://github.com/csjunxu/STAR.

A Weighted Fidelity and Regularization-Based Method for Mixed or Unknown Noise Removal from Images on Graphs

Image denoising technologies in a Euclidean domain have achieved good results and are becoming mature. However, in recent years, many real-world applications encountered in computer vision and geometric modeling involve image data defined in irregular domains modeled by huge graphs, which results in the problem on how to solve image denoising problems defined on graphs. In this paper, we propose a novel model for removing mixed or unknown noise in images on graphs. The objective is to minimize the sum of a weighted fidelity term and a sparse regularization term that additionally utilizes wavelet frame transform on graphs to retain feature details of images defined on graphs. Specifically, the weighted fidelity term with ℓ1-norm and ℓ2-norm is designed based on a analysis of the distribution of mixed noise. The augmented Lagrangian and accelerated proximal gradient methods are employed to achieve the optimal solution to the problem. Finally, some supporting numerical results and comparative analyses with other denoising algorithms are provided. It is noted that we investigate image denoising with unknown noise or a wide range of mixed noise, especially the mixture of Poisson, Gaussian, and impulse noise. Experimental results reported for synthetic and real images on graphs demonstrate that the proposed method is effective and efficient, and exhibits better performance for the removal of mixed or unknown noise in images on graphs than other denoising algorithms in the literature. The method can effectively remove mixed or unknown noise and retain feature details of images on graphs. It delivers a new avenue for denoising images in irregular domains.

Efficient and Fast Real-World Noisy Image Denoising by Combining Pyramid Neural Network and Two-Pathway Unscented Kalman Filter

Recently, image prior learning has emerged as an effective tool for image denoising, which exploits prior knowledge to obtain sparse coding models and utilize them to reconstruct the clean image from the noisy one. Albeit promising, these prior-learning based methods suffer from some limitations such as lack of adaptivity and failed attempts to improve performance and efficiency simultaneously. With the purpose of addressing these problems, in this paper, we propose a Pyramid Guided Filter Network (PGF-Net) integrated with pyramid-based neural network and Two-Pathway Unscented Kalman Filter (TP-UKF). The combination of pyramid network and TP-UKF is based on the consideration that the former enables our model to better exploit hierarchical and multi-scale features, while the latter can guide the network to produce an improved (a posteriori) estimation of the denoising results with fine-scale image details. Through synthesizing the respective advantages of pyramid network and TP-UKF, our proposed architecture, in stark contrast to prior learning methods, is able to decompose the image denoising task into a series of more manageable stages and adaptively eliminate the noise on real images in an efficient manner. We conduct extensive experiments and show that our PGF-Net achieves notable improvement on visual perceptual quality and higher computational efficiency compared to state-of-the-art methods.

Variational based Mixed Noise Removal with CNN Deep Learning Regularization

In this paper, the traditional model based variational methods and deep learning based algorithms are naturally integrated to address mixed noise removal, specially for Gaussian mixture noise and Gaussian-impulse noise removal problem. To be different from single type noise (e.g. Gaussian) removal, it is a challenge problem to accurately discriminate noise types and levels for each pixel. We propose a variational method to iteratively estimate the noise parameters, and then the algorithm can automatically classify the noise according to the different statistical parameters. The proposed variational problem can be separated into regularization, synthesis, parameters estimation and noise classification four steps with the operator splitting scheme. Each step is related to an optimization subproblem. To enforce the regularization, the deep learning method is employed to learn the natural images prior. Compared with some model based regularizations, the CNN regularizer can significantly improve the quality of the restored images. Compared with some learning based methods, the synthesis step can produce better reconstructions by analyzing the types and levels of the recognized noise. In our method, the convolution neutral network (CNN) can be regarded as an operator which associated to a variational functional. From this viewpoint, the proposed method can be extended to many image reconstruction and inverse problems. Numerical experiments in the paper show that our method can achieve some state-of-the-art results for Gaussian mixture noise and Gaussian-impulse noise removal.

Color Image and Multispectral Image Denoising Using Block Diagonal Representation

Filtering images of more than one channel are challenging in terms of both efficiency and effectiveness. By grouping similar patches to utilize the self-similarity and sparse linear approximation of natural images, recent nonlocal and transform-domain methods have been widely used in color and multispectral image (MSI) denoising. Many related methods focus on the modeling of group level correlation to enhance sparsity, which often resorts to a recursive strategy with a large number of similar patches. The importance of the patch level representation is understated. In this paper, we mainly investigate the influence and potential of representation at patch level by considering a general formulation with a block diagonal matrix. We further show that by training a proper global patch basis, along with a local principal component analysis transform in the grouping dimension, a simple transform-threshold-inverse method could produce very competitive results. Fast implementation is also developed to reduce the computational complexity. The extensive experiments on both the simulated and real datasets demonstrate its robustness, effectiveness, and efficiency.

Optimal Combination of Image Denoisers

Given a set of image denoisers, each having a different denoising capability, is there a provably optimal way of combining these denoisers to produce an overall better result? An answer to this question is fundamental to designing an ensemble of weak estimators for complex scenes. In this paper, we present an optimal combination scheme by leveraging the deep neural networks and the convex optimization. The proposed framework, called the Consensus Neural Network (CsNet), introduces three new concepts in image denoising: 1) a provably optimal procedure to combine the denoised outputs via convex optimization; 2) a deep neural network to estimate the mean squared error (MSE) of denoised images without needing the ground truths; and 3) an image boosting procedure using a deep neural network to improve the contrast and to recover the lost details of the combined images. Experimental results show that CsNet can consistently improve the denoising performance for both deterministic and neural network denoisers.

Content-Adaptive Noise Estimation for Color Images with Cross-Channel Noise Modeling

Noise estimation is crucial in many image processing tasks such as denoising. Most of the existing noise estimation methods are specially developed for grayscale images. For color images, these methods simply handle each color channel independently, without considering the correlation across channels. Moreover, these methods often assume a globally fixed noise model throughout the entire image, neglecting the adaptation to the local structures. In this work, we propose a contentadaptive multivariate Gaussian approach to model the noise in color images, in which we explicitly consider both the contentdependence and the inter-dependence among color channels. We design an effective method for estimating the noise covariance matrices within the proposed model. Specifically, a patch selection scheme is first introduced to select weakly textured patches via thresholding the texture strength indicators. Noticing that the patch selection actually depends on the unknown noise covariance, we present an iterative noise covariance estimation algorithm, where the patch selection and the covariance estimation are conducted alternately. For the remaining textured regions, we estimate a distinct covariance matrix associated with each pixel using a linear shrinkage estimator, which adaptively fuses the estimate coming from the weakly textured region and the sample covariance estimated from the local region. Experimental results show that our method can effectively estimate the noise covariance. The usefulness of our method is demonstrated with several image processing applications such as color image denoising and noise-robust superpixel.

Tensor regularized total variation for denoising of third harmonic generation images of brain tumors

Third harmonic generation (THG) microscopy shows great potential for instant pathology of brain tissue during surgery. However, the rich morphologies contained and the noise associated makes image restoration, necessary for quantification of the THG images, challenging. Anisotropic diffusion filtering (ADF) has been recently applied to restore THG images of normal brain, but ADF is hard-to-code, time-consuming and only reconstructs salient edges. This work overcomes these drawbacks by expressing ADF as a tensor regularized total variation model, which uses the Huber penalty and the L1 norm for tensor regularization and fidelity measurement, respectively. The diffusion tensor is constructed from the structure tensor of ADF yet the tensor decomposition is performed only in the non-flat areas. The resulting model is solved by an efficient and easy-to-code primal-dual algorithm. Tests on THG brain tumor images show that the proposed model has comparable denoising performance as ADF while it much better restores weak edges and it is up to 60% more time efficient.