Deconvolving images with unknown boundaries using the alternating direction method of multipliers

ADMM-inspired image reconstruction for terahertz off-axis digital holography

Image reconstruction in off-axis terahertz digital holography is complicated due to the harsh recording conditions and the non-convexity form of the problem. In this paper, we propose an inverse problem-based reconstruction technique that jointly reconstructs the object field and the amplitude of the reference field. Regularization in the wavelet domain promotes a sparse object solution. A single objective function combining the data-fidelity and regularization terms is optimized with a dedicated algorithm based on an alternating direction method of multipliers framework. Each iteration alternates between two consecutive optimizations using projections operating on each solution and one soft thresholding operator applying to the object solution. The method is preceded by a windowing process to alleviate artifacts due to the mismatch between camera frame truncation and periodic boundary conditions assumed to implement convolution operators. Experiments demonstrate the effectiveness of the proposed method, in particular, improvements of reconstruction quality, compared to two other methods.

Learned lensless 3D camera

Single-shot three-dimensional (3D) imaging with compact device footprint, high imaging quality, and fast processing speed is challenging in computational imaging. Mask-based lensless imagers, which replace the bulky optics with customized thin optical masks, are portable and lightweight, and can recover 3D object from a snap-shot image. Existing lensless imaging typically requires extensive calibration of its point spread function and heavy computational resources to reconstruct the object. Here we overcome these challenges and demonstrate a compact and learnable lensless 3D camera for real-time photorealistic imaging. We custom designed and fabricated the optical phase mask with an optimized spatial frequency support and axial resolving ability. We developed a simple and robust physics-aware deep learning model with adversarial learning module for real-time depth-resolved photorealistic reconstructions. Our lensless imager does not require calibrating the point spread function and has the capability to resolve depth and "see-through" opaque obstacles to image features being blocked, enabling broad applications in computational imaging.

GEOMScope: Large Field-of-view 3D Lensless Microscopy with Low Computational Complexity

Imaging systems with miniaturized device footprint, real-time processing speed and high resolution three-dimensional (3D) visualization are critical to broad biomedical applications such as endoscopy. Most of existing imaging systems rely on bulky lenses and mechanically refocusing to perform 3D imaging. Here, we demonstrate GEOMScope, a lensless single-shot 3D microscope that forms image through a single layer of thin microlens array and reconstructs objects through an innovative algorithm combining geometrical-optics-based pixel back projection and background suppressions. We verify the effectiveness of GEOMScope on resolution target, fluorescent particles and volumetric objects. Comparing to other widefield lensless imaging devices, we significantly reduce the required computational resource and increase the reconstruction speed by orders of magnitude. This enables us to image and recover large volume 3D object in high resolution with near real-time processing speed. Such a low computational complexity is attributed to the joint design of imaging optics and reconstruction algorithms, and a joint application of geometrical optics and machine learning in the 3D reconstruction. More broadly, the excellent performance of GEOMScope in imaging resolution, volume, and reconstruction speed implicates that geometrical optics could greatly benefit and play an important role in computational imaging.

An Efficient FPGA Implementation of Richardson-Lucy Deconvolution Algorithm for Hyperspectral Images

This paper proposes an implementation of a Richardson-Lucy (RL) deconvolution method to reduce the spatial degradation in hyperspectral images during the image acquisition process. The degradation, modeled by convolution with a point spread function (PSF), is reduced by applying both standard and accelerated RLdeconvolution algorithms on the individual images in spectral bands. Boundary conditions are introduced to maintain a constant image size without distorting the estimated image boundaries. The RL deconvolution algorithm is implemented on a field-programmable gate array (FPGA)-based Xilinx Zynq-7020 System-on-Chip (SoC). The proposed architecture is parameterized with respect to the image size and configurable with respect to the algorithm variant, the number of iterations, and the kernel size by setting the dedicated configuration registers. A speed-up by factors of 61 and 21 are reported compared to software-only and FPGA-based state-of-the-art implementations, respectively.

ECG Signal Denoising and Reconstruction Based on Basis Pursuit

The electrocardiogram (ECG) is widely used for the diagnosis of heart diseases. However, ECG signals are easily contaminated by different noises. This paper presents efficient denoising and compressed sensing (CS) schemes for ECG signals based on basis pursuit (BP). In the process of signal denoising and reconstruction, the low-pass filtering method and alternating direction method of multipliers (ADMM) optimization algorithm are used. This method introduces dual variables, adds a secondary penalty term, and reduces constraint conditions through alternate optimization to optimize the original variable and the dual variable at the same time. This algorithm is able to remove both baseline wander and Gaussian white noise. The effectiveness of the algorithm is validated through the records of the MIT-BIH arrhythmia database. The simulations show that the proposed ADMM-based method performs better in ECG denoising. Furthermore, this algorithm keeps the details of the ECG signal in reconstruction and achieves higher signal-to-noise ratio (SNR) and smaller mean square error (MSE).

Single-shot 3D wide-field fluorescence imaging with a Computational Miniature Mesoscope

Fluorescence microscopes are indispensable to biology and neuroscience. The need for recording in freely behaving animals has further driven the development in miniaturized microscopes (miniscopes). However, conventional microscopes/miniscopes are inherently constrained by their limited space-bandwidth product, shallow depth of field (DOF), and inability to resolve three-dimensional (3D) distributed emitters. Here, we present a Computational Miniature Mesoscope (CM2) that overcomes these bottlenecks and enables single-shot 3D imaging across an 8 mm by 7 mm field of view and 2.5-mm DOF, achieving 7-μm lateral resolution and better than 200-μm axial resolution. The CM2 features a compact lightweight design that integrates a microlens array for imaging and a light-emitting diode array for excitation. Its expanded imaging capability is enabled by computational imaging that augments the optics by algorithms. We experimentally validate the mesoscopic imaging capability on 3D fluorescent samples. We further quantify the effects of scattering and background fluorescence on phantom experiments.

Application of Blind Deconvolution Based on the New Weighted L1-norm Regularization with Alternating Direction Method of Multipliers in Light Microscopy Images

This study aimed to develop and evaluate a blind-deconvolution framework using the alternating direction method of multipliers (ADMMs) incorporated with weighted L1-norm regularization for light microscopy (LM) images. A presimulation study was performed using the Siemens star phantom prior to conducting the actual experiments. Subsequently, the proposed algorithm and a total generalized variation-based (TGV-based) method were applied to cross-sectional images of a mouse molar captured at 40× and 400× on-microscope magnifications and the results compared, and the resulting images were compared. Both simulation and experimental results confirmed that the proposed deblurring algorithm effectively restored the LM images, as evidenced by the quantitative evaluation metrics. In conclusion, this study demonstrated that the proposed deblurring algorithm can efficiently improve the quality of LM images.

Collaborative Filtering of Correlated Noise: Exact Transform-Domain Variance for Improved Shrinkage and Patch Matching

Collaborative filters perform denoising through transform-domain shrinkage of a group of similar patches extracted from an image. Existing collaborative filters of stationary correlated noise have all used simple approximations of the transform noise power spectrum adopted from methods which do not employ patch grouping and instead operate on a single patch. We note the inaccuracies of these approximations and introduce a method for the exact computation of the noise power spectrum. Unlike earlier methods, the calculated noise variances are exact even when noise in one patch is correlated with noise in any of the other patches. We discuss the adoption of the exact noise power spectrum within shrinkage, in similarity testing (patch matching), and in aggregation. We also introduce effective approximations of the spectrum for faster computation. Extensive experiments support the proposed method over earlier crude approximations used by image denoising filters such as Block-Matching and 3D-filtering (BM3D), demonstrating dramatic improvement in many challenging conditions.

A Convergent Image Fusion Algorithm Using Scene-Adapted Gaussian-Mixture-Based Denoising

We propose a new approach to image fusion, inspired by the recent plug-and-play (PnP) framework. In PnP, a denoiser is treated as a black-box and plugged into an iterative algorithm, taking the place of the proximity operator of some convex regularizer, which is formally equivalent to a denoising operation. This approach offers flexibility and excellent performance, but convergence may be hard to analyze, as most state-of-the-art denoisers lack an explicit underlying objective function. Here, we propose using a scene-adapted denoiser (i.e., targeted to the specific scene being imaged) plugged into the iterations of the alternating direction method of multipliers (ADMM). This approach, which is a natural choice for image fusion problems, not only yields state-of-the-art results, but it also allows proving convergence of the resulting algorithm. The proposed method is tested on two different problems: hyperspectral fusion/sharpening and fusion of blurred-noisy image pairs.

A convex 3D deconvolution algorithm for low photon count fluorescence imaging

Deconvolution is widely used to improve the contrast and clarity of a 3D focal stack collected using a fluorescence microscope. But despite being extensively studied, deconvolution algorithms can introduce reconstruction artifacts when their underlying noise models or priors are violated, such as when imaging biological specimens at extremely low light levels. In this paper we propose a deconvolution method specifically designed for 3D fluorescence imaging of biological samples in the low-light regime. Our method utilizes a mixed Poisson-Gaussian model of photon shot noise and camera read noise, which are both present in low light imaging. We formulate a convex loss function and solve the resulting optimization problem using the alternating direction method of multipliers algorithm. Among several possible regularization strategies, we show that a Hessian-based regularizer is most effective for describing locally smooth features present in biological specimens. Our algorithm also estimates noise parameters on-the-fly, thereby eliminating a manual calibration step required by most deconvolution software. We demonstrate our algorithm on simulated images and experimentally-captured images with peak intensities of tens of photoelectrons per voxel. We also demonstrate its performance for live cell imaging, showing its applicability as a tool for biological research.

Convolutional Dictionary Learning: Acceleration and Convergence

Convolutional dictionary learning (CDL or sparsifying CDL) has many applications in image processing and computer vision. There has been growing interest in developing efficient algorithms for CDL, mostly relying on the augmented Lagrangian (AL) method or the variant alternating direction method of multipliers (ADMM). When their parameters are properly tuned, AL methods have shown fast convergence in CDL. However, the parameter tuning process is not trivial due to its data dependence and, in practice, the convergence of AL methods depends on the AL parameters for nonconvex CDL problems. To moderate these problems, this paper proposes a new practically feasible and convergent Block Proximal Gradient method using a Majorizer (BPG-M) for CDL. The BPG-M-based CDL is investigated with different block updating schemes and majorization matrix designs, and further accelerated by incorporating some momentum coefficient formulas and restarting techniques. All of the methods investigated incorporate a boundary artifacts removal (or, more generally, sampling) operator in the learning model. Numerical experiments show that, without needing any parameter tuning process, the proposed BPG-M approach converges more stably to desirable solutions of lower objective values than the existing state-of-the-art ADMM algorithm and its memory-efficient variant do. Compared with the ADMM approaches, the BPG-M method using a multi-block updating scheme is particularly useful in single-threaded CDL algorithm handling large data sets, due to its lower memory requirement and no polynomial computational complexity. Image denoising experiments show that, for relatively strong additive white Gaussian noise, the filters learned by BPG-M-based CDL outperform those trained by the ADMM approach.

Partial Deconvolution With Inaccurate Blur Kernel

Most non-blind deconvolution methods are developed under the error-free kernel assumption, and are not robust to inaccurate blur kernel. Unfortunately, despite the great progress in blind deconvolution, estimation error remains inevitable during blur kernel estimation. Consequently, severe artifacts such as ringing effects and distortions are likely to be introduced in the non-blind deconvolution stage. In this paper, we tackle this issue by suggesting: 1) a partial map in the Fourier domain for modeling kernel estimation error, and 2) a partial deconvolution model for robust deblurring with inaccurate blur kernel. The partial map is constructed by detecting the reliable Fourier entries of estimated blur kernel. And partial deconvolution is applied to wavelet-based and learning-based models to suppress the adverse effect of kernel estimation error. Furthermore, an E-M algorithm is developed for estimating the partial map and recovering the latent sharp image alternatively. Experimental results show that our partial deconvolution model is effective in relieving artifacts caused by inaccurate blur kernel, and can achieve favorable deblurring quality on synthetic and real blurry images.

Single Particle Tracking: From Theory to Biophysical Applications

After three decades of developments, single particle tracking (SPT) has become a powerful tool to interrogate dynamics in a range of materials including live cells and novel catalytic supports because of its ability to reveal dynamics in the structure-function relationships underlying the heterogeneous nature of such systems. In this review, we summarize the algorithms behind, and practical applications of, SPT. We first cover the theoretical background including particle identification, localization, and trajectory reconstruction. General instrumentation and recent developments to achieve two- and three-dimensional subdiffraction localization and SPT are discussed. We then highlight some applications of SPT to study various biological and synthetic materials systems. Finally, we provide our perspective regarding several directions for future advancements in the theory and application of SPT.

Blind Deconvolution With Model Discrepancies

Blind deconvolution is a strongly ill-posed problem comprising of simultaneous blur and image estimation. Recent advances in prior modeling and/or inference methodology led to methods that started to perform reasonably well in real cases. However, as we show here, they tend to fail if the convolution model is violated even in a small part of the image. Methods based on variational Bayesian inference play a prominent role. In this paper, we use this inference in combination with the same prior for noise, image, and blur that belongs to the family of independent non-identical Gaussian distributions, known as the automatic relevance determination prior. We identify several important properties of this prior useful in blind deconvolution, namely, enforcing non-negativity of the blur kernel, favoring sharp images over blurred ones, and most importantly, handling non-Gaussian noise, which, as we demonstrate, is common in real scenarios. The presented method handles discrepancies in the convolution model, and thus extends applicability of blind deconvolution to real scenarios, such as photos blurred by camera motion and incorrect focus.

Efficient, Convergent SENSE MRI Reconstruction for Nonperiodic Boundary Conditions via Tridiagonal Solvers

Undersampling is an effective method for reducing scan acquisition time for MRI. Strategies for accelerated MRI such as parallel MRI and Compressed Sensing MRI present challenging image reconstruction problems with non-differentiable cost functions and computationally demanding operations. Variable splitting (VS) can simplify implementation of difficult image reconstruction problems, such as the combination of parallel MRI and Compressed Sensing, CS-SENSE-MRI. Combined with augmented Lagrangian (AL) and alternating minimization strategies, variable splitting can yield iterative minimization algorithms with simpler auxiliary variable updates. However, arbitrary variable splitting schemes are not guaranteed to converge. Many variable splitting strategies are combined with periodic boundary conditions. The resultant circulant Hessians enable 𝒪(n log n) computation but may compromise image accuracy at the spatial boundaries. We propose two methods for CS-SENSE-MRI that use regularization with non-periodic boundary conditions to prevent wrap-around artifacts. Each algorithm computes one of the resulting variable updates efficiently in 𝒪(n) time using a parallelizable tridiagonal solver. AL-tridiag is a VS method designed to enable efficient computation for non-periodic boundary conditions. Another proposed algorithm, ADMM-tridiag, uses a similar VS scheme but also ensures convergence to a minimizer of the proposed cost function using the Alternating Direction Method of Multipliers (ADMM). AL-tridiag and ADMM-tridiag show speeds competitive with previous VS CS-SENSE-MRI reconstruction algorithm AL-P2. We also apply the tridiagonal VS approach to a simple image inpainting problem.

Super Temporal-Resolved Microscopy (STReM)

Super-resolution microscopy typically achieves high spatial resolution, but the temporal resolution remains low. We report super temporal-resolved microscopy (STReM) to improve the temporal resolution of 2D super-resolution microscopy by a factor of 20 compared to that of the traditional camera-limited frame rate. This is achieved by rotating a phase mask in the Fourier plane during data acquisition and then recovering the temporal information by fitting the point spread function (PSF) orientations. The feasibility of this technique is verified with both simulated and experimental 2D adsorption/desorption and 2D emitter transport. When STReM is applied to measure protein adsorption at a glass surface, previously unseen dynamics are revealed.

A Framework for Fast Image Deconvolution With Incomplete Observations

In image deconvolution problems, the diagonalization of the underlying operators by means of the fast Fourier transform (FFT) usually yields very large speedups. When there are incomplete observations (e.g., in the case of unknown boundaries), standard deconvolution techniques normally involve non-diagonalizable operators, resulting in rather slow methods or, otherwise, use inexact convolution models, resulting in the occurrence of artifacts in the enhanced images. In this paper, we propose a new deconvolution framework for images with incomplete observations that allows us to work with diagonalized convolution operators, and therefore is very fast. We iteratively alternate the estimation of the unknown pixels and of the deconvolved image, using, e.g., an FFT-based deconvolution method. This framework is an efficient, high-quality alternative to existing methods of dealing with the image boundaries, such as edge tapering. It can be used with any fast deconvolution method. We give an example in which a state-of-the-art method that assumes periodic boundary conditions is extended, using this framework, to unknown boundary conditions. Furthermore, we propose a specific implementation of this framework, based on the alternating direction method of multipliers (ADMM). We provide a proof of convergence for the resulting algorithm, which can be seen as a "partial" ADMM, in which not all variables are dualized. We report experimental comparisons with other primal-dual methods, where the proposed one performed at the level of the state of the art. Four different kinds of applications were tested in the experiments: deconvolution, deconvolution with inpainting, superresolution, and demosaicing, all with unknown boundaries.

Generalized recovery algorithm for 3D super-resolution microscopy using rotating point spread functions

Super-resolution microscopy with phase masks is a promising technique for 3D imaging and tracking. Due to the complexity of the resultant point spread functions, generalized recovery algorithms are still missing. We introduce a 3D super-resolution recovery algorithm that works for a variety of phase masks generating 3D point spread functions. A fast deconvolution process generates initial guesses, which are further refined by least squares fitting. Overfitting is suppressed using a machine learning determined threshold. Preliminary results on experimental data show that our algorithm can be used to super-localize 3D adsorption events within a porous polymer film and is useful for evaluating potential phase masks. Finally, we demonstrate that parallel computation on graphics processing units can reduce the processing time required for 3D recovery. Simulations reveal that, through desktop parallelization, the ultimate limit of real-time processing is possible. Our program is the first open source recovery program for generalized 3D recovery using rotating point spread functions.

Variational Dirichlet Blur Kernel Estimation

Blind image deconvolution involves two key objectives: 1) latent image and 2) blur estimation. For latent image estimation, we propose a fast deconvolution algorithm, which uses an image prior of nondimensional Gaussianity measure to enforce sparsity and an undetermined boundary condition methodology to reduce boundary artifacts. For blur estimation, a linear inverse problem with normalization and nonnegative constraints must be solved. However, the normalization constraint is ignored in many blind image deblurring methods, mainly because it makes the problem less tractable. In this paper, we show that the normalization constraint can be very naturally incorporated into the estimation process by using a Dirichlet distribution to approximate the posterior distribution of the blur. Making use of variational Dirichlet approximation, we provide a blur posterior approximation that considers the uncertainty of the estimate and removes noise in the estimated kernel. Experiments with synthetic and real data demonstrate that the proposed method is very competitive to the state-of-the-art blind image restoration methods.

Forward Looking Radar Imaging by Truncated Singular Value Decomposition and Its Application for Adverse Weather Aircraft Landing

The forward looking radar imaging task is a practical and challenging problem for adverse weather aircraft landing industry. Deconvolution method can realize the forward looking imaging but it often leads to the noise amplification in the radar image. In this paper, a forward looking radar imaging based on deconvolution method is presented for adverse weather aircraft landing. We first present the theoretical background of forward looking radar imaging task and its application for aircraft landing. Then, we convert the forward looking radar imaging task into a corresponding deconvolution problem, which is solved in the framework of algebraic theory using truncated singular decomposition method. The key issue regarding the selecting of the truncated parameter is addressed using generalized cross validation approach. Simulation and experimental results demonstrate that the proposed method is effective in achieving angular resolution enhancement with suppressing the noise amplification in forward looking radar imaging.

High-accuracy total variation with application to compressed video sensing

Numerous total variation (TV) regularizers, engaged in image restoration problem, encode the gradients by means of simple [-1, 1] finite-impulse-response (FIR) filter. Despite its low computational processing, this filter severely distorts signal's high-frequency components pertinent to edge/ discontinuous information and cause several deficiency issues known as texture and geometric loss. This paper addresses this problem by proposing an alternative model to the TV regularization problem via high-order accuracy differential FIR filters to preserve rapid transitions in signal recovery. A numerical encoding scheme is designed to extend the TV model into multidimensional representation (tensorial decomposition). We adopt this design to regulate the spatial and temporal redundancy in compressed video sensing problem to jointly recover frames from undersampled measurements. We then seek the solution via alternating direction methods of multipliers and find a unique solution to quadratic minimization step with capability of handling different boundary conditions. The resulting algorithm uses much lower sampling rate and highly outperforms alternative state-of-the-art methods. This is evaluated both in terms of restoration accuracy and visual quality of the recovered frames.

Large-margin Learning of Compact Binary Image Encodings

The use of high-dimensional features has become a normal practice in many computer vision applications. The large dimension of these features is a limiting factor upon the number of data points which may be effectively stored and processed, however. We address this problem by developing a novel approach to learning a compact binary encoding, which exploits both pair-wise proximity and class-label information on training data set. Exploiting this extra information allows the development of encodings which, although compact, outperform the original high-dimensional features in terms of final classification or retrieval performance. The method is general, in that it is applicable to both non-parametric and parametric learning methods. This generality means that the embedded features are suitable for a wide variety of computer vision tasks, such as image classification and content-based image retrieval. Experimental results demonstrate that the new compact descriptor achieves an accuracy comparable to, and in some cases better than, the visual descriptor in the original space despite being significantly more compact. Moreover, any convex loss function and convex regularization penalty (e.g., `p norm with p 1) can be incorporated into the framework, which provides future flexibility.

Accelerated edge-preserving image restoration without boundary artifacts

To reduce blur in noisy images, regularized image restoration methods have been proposed that use nonquadratic regularizers (like l1 regularization or total-variation) that suppress noise while preserving edges in the image. Most of these methods assume a circulant blur (periodic convolution with a blurring kernel) that can lead to wraparound artifacts along the boundaries of the image due to the implied periodicity of the circulant model. Using a noncirculant model could prevent these artifacts at the cost of increased computational complexity. In this paper, we propose to use a circulant blur model combined with a masking operator that prevents wraparound artifacts. The resulting model is noncirculant, so we propose an efficient algorithm using variable splitting and augmented Lagrangian (AL) strategies. Our variable splitting scheme, when combined with the AL framework and alternating minimization, leads to simple linear systems that can be solved noniteratively using fast Fourier transforms (FFTs), eliminating the need for more expensive conjugate gradient-type solvers. The proposed method can also efficiently tackle a variety of convex regularizers, including edge-preserving (e.g., total-variation) and sparsity promoting (e.g., l1-norm) regularizers. Simulation results show fast convergence of the proposed method, along with improved image quality at the boundaries where the circulant model is inaccurate.