Postprocessing of Compressed Images via Sequential Denoising

Multi-Modality Deep Restoration of Extremely Compressed Face Videos

Arguably the most common and salient object in daily video communications is the talking head, as encountered in social media, virtual classrooms, teleconferences, news broadcasting, talk shows, etc. When communication bandwidth is limited by network congestions or cost effectiveness, compression artifacts in talking head videos are inevitable. The resulting video quality degradation is highly visible and objectionable due to high acuity of human visual system to faces. To solve this problem, we develop a multi-modality deep convolutional neural network method for restoring face videos that are aggressively compressed. The main innovation is a new DCNN architecture that incorporates known priors of multiple modalities: the video-synchronized speech signal and semantic elements of the compression code stream, including motion vectors, code partition map and quantization parameters. These priors strongly correlate with the latent video and hence they are able to enhance the capability of deep learning to remove compression artifacts. Ample empirical evidences are presented to validate the superior performance of the proposed DCNN method on face videos over the existing state-of-the-art methods.

A Novel Reconstruction Algorithm with High Performance for Compressed Ultrafast Imaging

Compressed ultrafast photography (CUP) is a type of two-dimensional (2D) imaging technique to observe ultrafast processes. Intelligence reconstruction methods that influence the imaging quality are an essential part of a CUP system. However, existing reconstruction algorithms mostly rely on image priors and complex parameter spaces. Therefore, it usually takes a lot of time to obtain acceptable reconstruction results, which limits the practical application of the CUP. In this paper, we proposed a novel reconstruction algorithm named PnP-FFDNet, which can provide a high quality and high efficiency compared to previous methods. First, we built a forward model of the CUP and three sub-optimization problems were obtained using the alternating direction multiplier method (ADMM), and the closed-form solution of the first sub-optimization problem was derived. Secondly, inspired by the PnP-ADMM framework, we used an advanced denoising algorithm based on a neural network named FFDNet to solve the second sub-optimization problem. On the real CUP data, PSNR and SSIM are improved by an average of 3 dB and 0.06, respectively, compared with traditional algorithms. Both on the benchmark dataset and on the real CUP data, the proposed method reduces the running time by an average of about 96% over state-of-the-art algorithms, and show comparable visual results, but in a much shorter running time.

Better Compression With Deep Pre-Editing

Could we compress images via standard codecs while avoiding visible artifacts? The answer is obvious - this is doable as long as the bit budget is generous enough. What if the allocated bit-rate for compression is insufficient? Then unfortunately, artifacts are a fact of life. Many attempts were made over the years to fight this phenomenon, with various degrees of success. In this work we aim to break the unholy connection between bit-rate and image quality, and propose a way to circumvent compression artifacts by pre-editing the incoming image and modifying its content to fit the given bits. We design this editing operation as a learned convolutional neural network, and formulate an optimization problem for its training. Our loss takes into account a proximity between the original image and the edited one, a bit-budget penalty over the proposed image, and a no-reference image quality measure for forcing the outcome to be visually pleasing. The proposed approach is demonstrated on the popular JPEG compression, showing savings in bits and/or improvements in visual quality, obtained with intricate editing effects.

Dimensionality reduced plug and play priors for improving photoacoustic tomographic imaging with limited noisy data

The reconstruction methods for solving the ill-posed inverse problem of photoacoustic tomography with limited noisy data are iterative in nature to provide accurate solutions. These methods performance is highly affected by the noise level in the photoacoustic data. A singular value decomposition (SVD) based plug and play priors method for solving photoacoustic inverse problem was proposed in this work to provide robustness to noise in the data. The method was shown to be superior as compared to total variation regularization, basis pursuit deconvolution and Lanczos Tikhonov based regularization and provided improved performance in case of noisy data. The numerical and experimental cases show that the improvement can be as high as 8.1 dB in signal to noise ratio of the reconstructed image and 67.98% in root mean square error in comparison to the state of the art methods.

Image Restoration by Iterative Denoising and Backward Projections

Inverse problems appear in many applications, such as image deblurring and inpainting. The common approach to address them is to design a specific algorithm for each problem. The Plug-and-Play (P&P) framework, which has been recently introduced, allows solving general inverse problems by leveraging the impressive capabilities of existing denoising algorithms. While this fresh strategy has found many applications, a burdensome parameter tuning is often required in order to obtain high-quality results. In this paper, we propose an alternative method for solving inverse problems using off-the-shelf denoisers, which requires less parameter tuning. First, we transform a typical cost function, composed of fidelity and prior terms, into a closely related, novel optimization problem. Then, we propose an efficient minimization scheme with a P&P property, i.e., the prior term is handled solely by a denoising operation. Finally, we present an automatic tuning mechanism to set the method's parameters. We provide a theoretical analysis of the method and empirically demonstrate its competitiveness with task-specific techniques and the P&P approach for image inpainting and deblurring.

Motion Estimation-Assisted Denoising for an Efficient Combination with an HEVC Encoder

Noise, which is commonly generated in low-light environments or by low-performance cameras, is a major cause of the degradation of compression efficiency. In previous studies that attempted to combine a denoise algorithm and a video encoder, denoising was used independently of the code for pre-processing or post-processing. However, this process must be tightly coupled with encoding because noise affects the compression efficiency greatly. In addition, this represents a major opportunity to reduce the computational complexity, because the encoding process and a denoise algorithm have many similarities. In this paper, a simple, add-on denoising scheme is proposed through a combination of high-efficiency video coding (HEVC) and block matching three-dimensional collaborative filtering (BM3D) algorithms. It is known that BM3D has excellent denoise performance but that it is limited in its use due to its high computational complexity. This paper employs motion estimation in HEVC to replace the block matching of BM3D so that most of the time-consuming functions are shared. To overcome the challenging algorithmic differences, the hierarchical structure in HEVC is uniquely utilized. As a result, the computational complexity is drastically reduced while the competitive performance capabilities in terms of coding efficiency and denoising quality are maintained.

Optimized Pre-Compensating Compression

In imaging systems, following acquisition, an image/ video is transmitted or stored and eventually presented to human observers using different and often imperfect display devices. While the resulting quality of the output image may severely be affected by the display, this degradation is usually ignored in the preceding compression. In this paper we model the sub-optimality of the display device as a known degradation operator applied on the decompressed image/video. We assume the use of a standard compression path, and augment it with a suitable pre-processing procedure, providing a compressed signal intended to compensate the degradation without any post-filtering. Our approach originates from an intricate rate-distortion problem, optimizing the modifications to the input image/video for reaching best end-to-end performance. We address this seemingly computationally intractable problem using the alternating direction method of multipliers (ADMM) approach, leading to a procedure in which a standard compression technique is iteratively applied. We demonstrate the proposed method for adjusting HEVC image/video compression to compensate post-decompression visual effects due to a common type of displays. Particularly, we use our method to reduce motion-blur perceived while viewing video on LCD devices. The experiments establish our method as a leading approach for preprocessing high bit-rate compression to counterbalance a postdecompression degradation.

MuLoG, or How to Apply Gaussian Denoisers to Multi-Channel SAR Speckle Reduction?

Speckle reduction is a longstanding topic in synthetic aperture radar (SAR) imaging. Since most current and planned SAR imaging satellites operate in polarimetric, interferometric, or tomographic modes, SAR images are multi-channel and speckle reduction techniques must jointly process all channels to recover polarimetric and interferometric information. The distinctive nature of SAR signal (complex-valued, corrupted by multiplicative fluctuations) calls for the development of specialized methods for speckle reduction. Image denoising is a very active topic in image processing with a wide variety of approaches and many denoising algorithms available, almost always designed for additive Gaussian noise suppression. This paper proposes a general scheme, called MuLoG (MUlti-channel LOgarithm with Gaussian denoising), to include such Gaussian denoisers within a multi-channel SAR speckle reduction technique. A new family of speckle reduction algorithms can thus be obtained, benefiting from the ongoing progress in Gaussian denoising, and offering several speckle reduction results often displaying method-specific artifacts that can be dismissed by comparison between results.