Motion Blur Kernel Estimation via Deep Learning

Learning Frame-Event Fusion for Motion Deblurring

Motion deblurring is a highly ill-posed problem due to the significant loss of motion information in the blurring process. Complementary informative features from auxiliary sensors such as event cameras can be explored for guiding motion deblurring. The event camera can capture rich motion information asynchronously with microsecond accuracy. In this paper, a novel frame-event fusion framework is proposed for event-driven motion deblurring (FEF-Deblur), which can sufficiently explore long-range cross-modal information interactions. Firstly, different modalities are usually complementary and also redundant. Cross-modal fusion is modeled as complementary-unique features separation-and-aggregation, avoiding the modality redundancy. Unique features and complementary features are first inferred with parallel intra-modal self-attention and inter-modal cross-attention respectively. After that, a correlation-based constraint is designed to act between unique and complementary features to facilitate their differentiation, which assists in cross-modal redundancy suppression. Additionally, spatio-temporal dependencies among neighboring inputs are crucial for motion deblurring. A recurrent cross attention is introduced to preserve inter-input attention information, in which the current spatial features and aggregated temporal features are attending to each other by establishing the long-range interaction between them. Extensive experiments on both synthetic and real-world motion deblurring datasets demonstrate our method outperforms state-of-the-art event-based and image/video-based methods. The code will be made publicly available.

Event-Assisted Blurriness Representation Learning for Blurry Image Unfolding

The goal of blurry image deblurring and unfolding task is to recover a single sharp frame or a sequence from a blurry one. Recently, its performance is greatly improved with introduction of a bio-inspired visual sensor, event camera. Most existing event-assisted deblurring methods focus on the design of powerful network architectures and effective training strategy, while ignoring the role of blur modeling in removing various blur in dynamic scenes. In this work, we propose to implicitly model blur in an image by computing blurriness representation with an event-assisted blurriness encoder. The learning of blurriness representation is formulated as a ranking problem based on specially synthesized pairs. Blurriness-aware image unfolding is achieved by integrating blur relevant information contained in the representation into a base unfolding network. The integration is mainly realized by the proposed blurriness-guided modulation and multi-scale aggregation modules. Experiments on GOPRO and HQF datasets show favorable performance of the proposed method against state-of-the-art approaches. More results on real-world data validate its effectiveness in recovering a sequence of latent sharp frames from a blurry image.

Blind Deblurring Method for CASEarth Multispectral Images Based on Inter-Band Gradient Similarity Prior

Multispectral remote sensing images contain abundant information about the distribution and reflectance of ground objects, playing a crucial role in target detection, environmental monitoring, and resource exploration. However, due to the complexity of the imaging process in multispectral remote sensing, image blur is inevitable, and the blur kernel is typically unknown. In recent years, many researchers have focused on blind image deblurring, but most of these methods are based on single-band images. When applied to CASEarth satellite multispectral images, the spectral correlation is unutilized. To address this limitation, this paper proposes a novel approach that leverages the characteristics of multispectral data more effectively. We introduce an inter-band gradient similarity prior and incorporate it into the patch-wise minimal pixel (PMP)-based deblurring model. This approach aims to utilize the spectral correlation across bands to improve deblurring performance. A solution algorithm is established by combining the half-quadratic splitting method with alternating minimization. Subjectively, the final experiments on CASEarth multispectral images demonstrate that the proposed method offers good visual effects while enhancing edge sharpness. Objectively, our method leads to an average improvement in point sharpness by a factor of 1.6, an increase in edge strength level by a factor of 1.17, and an enhancement in RMS contrast by a factor of 1.11.

Estimation of non-uniform motion blur using a patch-based regression convolutional neural network

The non-uniform blur of atmospheric turbulence can be modeled as a superposition of linear motion blur kernels at a patch level. We propose a regression convolutional neural network (CNN) to predict angle and length of a linear motion blur kernel for varying sized patches. We analyze the robustness of the network for different patch sizes and the performance of the network in regions where the characteristics of the blur are transitioning. Alternating patch sizes per epoch in training, we find coefficient of determination scores across a range of patch sizes of R 2>0.78 for length and R 2>0.94 for angle prediction. We find that blur predictions in regions overlapping two blur characteristics transition between the two characteristics as overlap changes. These results validate the use of such a network for prediction of non-uniform blur characteristics at a patch level.

Blind Image Deconvolution Using Variational Deep Image Prior

Conventional deconvolution methods utilize hand-crafted image priors to constrain the optimization. While deep-learning-based methods have simplified the optimization by end-to-end training, they fail to generalize well to blurs unseen in the training dataset. Thus, training image-specific models is important for higher generalization. Deep image prior (DIP) provides an approach to optimize the weights of a randomly initialized network with a single degraded image by maximum a posteriori (MAP), which shows that the architecture of a network can serve as the hand-crafted image prior. Unlike conventional hand-crafted image priors, which are obtained through statistical methods, finding a suitable network architecture is challenging due to the unclear relationship between images and their corresponding architectures. As a result, the network architecture cannot provide enough constraint for the latent sharp image. This paper proposes a new variational deep image prior (VDIP) for blind image deconvolution, which exploits additive hand-crafted image priors on latent sharp images and approximates a distribution for each pixel to avoid suboptimal solutions. Our mathematical analysis shows that the proposed method can better constrain the optimization. The experimental results further demonstrate that the generated images have better quality than that of the original DIP on benchmark datasets.

A state-of-the-art review of image motion deblurring techniques in precision agriculture

Image motion deblurring is a crucial technology in computer vision that has gained significant attention attracted by its outstanding ability for accurate acquisition of motion image information, processing and intelligent decision making, etc. Motion blur has recently been considered as one of the major challenges for applications of computer vision in precision agriculture. Motion blurred images seriously affect the accuracy of information acquisition in precision agriculture scene image such as testing, tracking, and behavior analysis of animals, recognition of plant phenotype, critical characteristics of pests and diseases, etc. On the other hand, the fast motion and irregular deformation of agriculture livings, and motion of image capture device all introduce great challenges for image motion deblurring. Hence, the demand of more efficient image motion deblurring method is rapidly increasing and developing in the applications with dynamic scene. Up till now, some studies have been carried out to address this challenge, e.g., spatial motion blur, multi-scale blur and other types of blur. This paper starts with categorization of causes of image blur in precision agriculture. Then, it gives detail introduction of general-purpose motion deblurring methods and their the strengthen and weakness. Furthermore, these methods are compared for the specific applications in precision agriculture e.g., detection and tracking of livestock animal, harvest sorting and grading, and plant disease detection and phenotyping identification etc. Finally, future research directions are discussed to push forward the research and application of advancing in precision agriculture image motion deblurring field.

Reference-Based Multi-Level Features Fusion Deblurring Network for Optical Remote Sensing Images

Blind image deblurring is a long-standing challenge in remote sensing image restoration tasks. It aims to recover a latent sharp image from a blurry image while the blur kernel is unknown. To solve this problem, many image priors-based algorithms and learning-based algorithms have been proposed. However, most of these methods are based on a single blurry image. Due to the lack of high frequency information, the images restored by these algorithms still have some deficiencies in edge and texture details. In this work, we propose a novel deep learning model named Reference-Based Multi-Level Features Fusion Deblurring Network (Ref-MFFDN), which registers the reference image and the blurry image in the multi-level feature space and transfers the high-quality textures from registered reference features to assist image deblurring. Comparative experiments on the testing set prove that our Ref-MFFDN outperforms many state-of-the-art single image deblurring approaches in both quantitative evaluation and visual results, which indicates the effectiveness of using reference images in remote sensing image deblurring tasks. More ablation experiments demonstrates the robustness of Ref-MFFDN to the input image size, the effectiveness of multi-level features fusion network (MFFN) and the effect of different feature levels in multi-feature extractor (MFE) on algorithm performance.

Exploiting Raw Images for Real-Scene Super-Resolution

Super-resolution is a fundamental problem in computer vision which aims to overcome the spatial limitation of camera sensors. While significant progress has been made in single image super-resolution, most algorithms only perform well on synthetic data, which limits their applications in real scenarios. In this paper, we study the problem of real-scene single image super-resolution to bridge the gap between synthetic data and real captured images. We focus on two issues of existing super-resolution algorithms: lack of realistic training data and insufficient utilization of visual information obtained from cameras. To address the first issue, we propose a method to generate more realistic training data by mimicking the imaging process of digital cameras. For the second issue, we develop a two-branch convolutional neural network to exploit the radiance information originally-recorded in raw images. In addition, we propose a dense channel-attention block for better image restoration as well as a learning-based guided filter network for effective color correction. Our model is able to generalize to different cameras without deliberately training on images from specific camera types. Extensive experiments demonstrate that the proposed algorithm can recover fine details and clear structures, and achieve high-quality results for single image super-resolution in real scenes.

Blind Motion Deblurring Super-Resolution: When Dynamic Spatio-Temporal Learning Meets Static Image Understanding

Single-image super-resolution (SR) and multi-frame SR are two ways to super resolve low-resolution images. Single-Image SR generally handles each image independently, but ignores the temporal information implied in continuing frames. Multi-frame SR is able to model the temporal dependency via capturing motion information. However, it relies on neighbouring frames which are not always available in the real world. Meanwhile, slight camera shake easily causes heavy motion blur on long-distance-shot low-resolution images. To address these problems, a Blind Motion Deblurring Super-Reslution Networks, BMDSRNet, is proposed to learn dynamic spatio-temporal information from single static motion-blurred images. Motion-blurred images are the accumulation over time during the exposure of cameras, while the proposed BMDSRNet learns the reverse process and uses three-streams to learn Bidirectional spatio-temporal information based on well designed reconstruction loss functions to recover clean high-resolution images. Extensive experiments demonstrate that the proposed BMDSRNet outperforms recent state-of-the-art methods, and has the ability to simultaneously deal with image deblurring and SR.

Structure-Aware Motion Deblurring Using Multi-Adversarial Optimized CycleGAN

Recently, Convolutional Neural Networks (CNNs) have achieved great improvements in blind image motion deblurring. However, most existing image deblurring methods require a large amount of paired training data and fail to maintain satisfactory structural information, which greatly limits their application scope. In this paper, we present an unsupervised image deblurring method based on a multi-adversarial optimized cycle-consistent generative adversarial network (CycleGAN). Although original CycleGAN can handle unpaired training data well, the generated high-resolution images are probable to lose content and structure information. To solve this problem, we utilize a multi-adversarial mechanism based on CycleGAN for blind motion deblurring to generate high-resolution images iteratively. In this multi-adversarial manner, the hidden layers of the generator are gradually supervised, and the implicit refinement is carried out to generate high-resolution images continuously. Meanwhile, we also introduce the structure-aware mechanism to enhance the structure and detail retention ability of the multi-adversarial network for deblurring by taking the edge map as guidance information and adding multi-scale edge constraint functions. Our approach not only avoids the strict need for paired training data and the errors caused by blur kernel estimation, but also maintains the structural information better with multi-adversarial learning and structure-aware mechanism. Comprehensive experiments on several benchmarks have shown that our approach prevails the state-of-the-art methods for blind image motion deblurring.

Modeling and Enhancing Low-Quality Retinal Fundus Images

Retinal fundus images are widely used for the clinical screening and diagnosis of eye diseases. However, fundus images captured by operators with various levels of experience have a large variation in quality. Low-quality fundus images increase uncertainty in clinical observation and lead to the risk of misdiagnosis. However, due to the special optical beam of fundus imaging and structure of the retina, natural image enhancement methods cannot be utilized directly to address this. In this article, we first analyze the ophthalmoscope imaging system and simulate a reliable degradation of major inferior-quality factors, including uneven illumination, image blurring, and artifacts. Then, based on the degradation model, a clinically oriented fundus enhancement network (cofe-Net) is proposed to suppress global degradation factors, while simultaneously preserving anatomical retinal structures and pathological characteristics for clinical observation and analysis. Experiments on both synthetic and real images demonstrate that our algorithm effectively corrects low-quality fundus images without losing retinal details. Moreover, we also show that the fundus correction method can benefit medical image analysis applications, e.g., retinal vessel segmentation and optic disc/cup detection.

Blur Removal Via Blurred-Noisy Image Pair

Complex blur such as the mixup of space-variant and space-invariant blur, which is hard to model mathematically, widely exists in real images. In this article, we propose a novel image deblurring method that does not need to estimate blur kernels. We utilize a pair of images that can be easily acquired in low-light situations: (1) a blurred image taken with low shutter speed and low ISO noise; and (2) a noisy image captured with high shutter speed and high ISO noise. Slicing the blurred image into patches, we extend the Gaussian mixture model (GMM) to model the underlying intensity distribution of each patch using the corresponding patches in the noisy image. We compute patch correspondences by analyzing the optical flow between the two images. The Expectation Maximization (EM) algorithm is utilized to estimate the parameters of GMM. To preserve sharp features, we add an additional bilateral term to the objective function in the M-step. We eventually add a detail layer to the deblurred image for refinement. Extensive experiments on both synthetic and real-world data demonstrate that our method outperforms state-of-the-art techniques, in terms of robustness, visual quality, and quantitative metrics.

Invertibility-Driven Interpolation Filter for Video Coding

Motion compensation with fractional motion vector has been widely utilized in the video coding standards. The fractional samples are usually generated by fractional interpolation filters. Traditional interpolation filters are usually designed based on the signal processing theory with the assumption of band-limited signal, which cannot effectively capture the non-stationary property of video content and cannot adapt to the variety of video quality. In this paper, we reveal an intuitive property of the fractional interpolation problem, named invertibility. That is, the fractional interpolation filters should not only generate fractional samples from integer samples but also recover the integer samples from the fractional samples in an invertible manner. We prove in theory that the invertibility in the spatial domain is equivalent to the constant magnitude in the Fourier transform domain. Driven by the invertibility, we then develop a learning-based method to solve the fractional interpolation problem. Inspired by the advances of convolutional neural network (CNN), we propose to establish an end-to-end scheme using CNN to train invertibility-driven interpolation filter (InvIF). Different from the previous learning-based methods, the proposed training scheme does not need hand-crafted "ground truth" of fractional samples. The proposed InvIF is integrated into high efficiency video coding (HEVC), and extensive experiments are conducted to verify its effectiveness. The experimental results show that the proposed method can achieve on average 4.7% and 3.6% BD-rate reduction compared with the HEVC anchor, under low-delay-B and random-access configurations, respectively.

Blind Deblurring of Saturated Images Based on Optimization and Deep Learning for Dynamic Visual Inspection on the Assembly Line

Image deblurring can improve visual quality and mitigates motion blur for dynamic visual inspection. We propose a method to deblur saturated images for dynamic visual inspection by applying blur kernel estimation and deconvolution modeling. The blur kernel is estimated in a transform domain, whereas the deconvolution model is decoupled into deblurring and denoising stages via variable splitting. Deblurring predicts the mask specifying saturated pixels, which are then discarded, and denoising is learned via the fast and flexible denoising network (FFDNet) convolutional neural network (CNN) at a wide range of noise levels. Hence, the proposed deconvolution model provides the benefits of both model optimization and deep learning. Experiments demonstrate that the proposed method suitably restores visual quality and outperforms existing approaches with good score improvements.

Nonlocal Patch Tensor Sparse Representation for Hyperspectral Image Super-Resolution

This paper presents a hypserspectral image (HSI) super-resolution method which fuses a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) to get high-resolution HSI (HR-HSI). The proposed method first extracts the nonlocal similar patches to form a nonlocal patch tensor (NPT). A novel tensor-tensor product (t-product) based tensor sparse representation is proposed to model the extracted NPTs. Through the tensor sparse representation, both the spectral and spatial similarities between the nonlocal similar patches are well preserved. Then, the relationship between the HR-HSI and LR-HSI is built using t-product which allows us to design a unified objective function to incorporate the nonlocal similarity, tensor dictionary learning, and tensor sparse coding together. Finally, Alternating Direction Method of Multipliers (ADMM) is used to solve the optimization problem. Experimental results on three data sets and one real data set demonstrate that the proposed method substantially outperforms the existing state-of-the-art HSI super-resolution methods.