Metamer-set-based approach to estimating surface reflectance from camera RGB

A Rehabilitation of Pixel-Based Spectral Reconstruction from RGB Images

Recently, many deep neural networks (DNN) have been proposed to solve the spectral reconstruction (SR) problem: recovering spectra from RGB measurements. Most DNNs seek to learn the relationship between an RGB viewed in a given spatial context and its corresponding spectra. Significantly, it is argued that the same RGB can map to different spectra depending on the context with respect to which it is seen and, more generally, that accounting for spatial context leads to improved SR. However, as it stands, DNN performance is only slightly better than the much simpler pixel-based methods where spatial context is not used. In this paper, we present a new pixel-based algorithm called A++ (an extension of the A+ sparse coding algorithm). In A+, RGBs are clustered, and within each cluster, a designated linear SR map is trained to recover spectra. In A++, we cluster the spectra instead in an attempt to ensure neighboring spectra (i.e., spectra in the same cluster) are recovered by the same SR map. A polynomial regression framework is developed to estimate the spectral neighborhoods given only the RGB values in testing, which in turn determines which mapping should be used to map each testing RGB to its reconstructed spectrum. Compared to the leading DNNs, not only does A++ deliver the best results, it is parameterized by orders of magnitude fewer parameters and has a significantly faster implementation. Moreover, in contradistinction to some DNN methods, A++ uses pixel-based processing, which is robust to image manipulations that alter the spatial context (e.g., blurring and rotations). Our demonstration on the scene relighting application also shows that, while SR methods, in general, provide more accurate relighting results compared to the traditional diagonal matrix correction, A++ provides superior color accuracy and robustness compared to the top DNN methods.

Toward non-metameric reflectance recovery by emulating the spectral neighborhood using corresponding color information

In learning-based reflectance reconstruction methods, usually localized training samples are used to reconstruct spectral curves. The state-of-the-art methods localize the training samples based on their colorimetric color differences with the test sample. This approach is dependent on the working color space, color difference equation, and/or illuminant used, and it may result in a metameric match. This issue can be resolved by localizing the training samples based on their spectral difference with the test sample; however, this would require an already unknown spectral curve of the test sample. In this paper, use of corresponding color information to emulate the spectral neighborhood of the test color for non-metameric reflectance recovery is proposed. The Wiener estimation method was extended by (1) using two thresholds, (i) on the color difference between the test sample and the training samples under the reference illuminant and (ii) on the color difference between the corresponding color of the test sample and the training samples under another illuminant, to mimic the spectral neighborhood of the test sample within the gamut of the training data, and (2) also using the tristimulus values of the corresponding color in the regression. Results showed that the proposed extension of the Wiener estimation method improved the reflectance recovery and hence reduced the metamerism.

Optimized light source spectral power distribution for RGB camera based spectral reflectance recovery

The accuracy of recovered spectra from camera responses mainly depends on the spectral estimation algorithm used, the camera and filters selected, and the light source used to illuminate the object. We present and compare different light source spectrum optimization methods together with different spectral estimation algorithms applied to reflectance recovery. These optimization methods include the Monte Carlo (MC) method, particle swarm optimization (PSO) and multi-population genetic algorithm (MPGA). Optimized SPDs are compared with D65, D50 A and three LED light sources in simulation and reality. Results obtained show us that MPGA has superior performance, and optimized light source spectra along with better spectral estimation algorithm can provide a more accurate spectral reflectance estimation of an object surface. Meanwhile, it is found that camera spectral sensitivities weighted by optimized SPDs tend to be mutually orthogonal.

Physically Plausible Spectral Reconstruction

Spectral reconstruction algorithms recover spectra from RGB sensor responses. Recent methods—with the very best algorithms using deep learning—can already solve this problem with good spectral accuracy. However, the recovered spectra are physically incorrect in that they do not induce the RGBs from which they are recovered. Moreover, if the exposure of the RGB image changes then the recovery performance often degrades significantly—i.e., most contemporary methods only work for a fixed exposure. In this paper, we develop a physically accurate recovery method: the spectra we recover provably induce the same RGBs. Key to our approach is the idea that the set of spectra that integrate to the same RGB can be expressed as the sum of a unique fundamental metamer (spanned by the camera’s spectral sensitivities and linearly related to the RGB) and a linear combination of a vector space of metameric blacks (orthogonal to the spectral sensitivities). Physically plausible spectral recovery resorts to finding a spectrum that adheres to the fundamental metamer plus metameric black decomposition. To further ensure spectral recovery that is robust to changes in exposure, we incorporate exposure changes in the training stage of the developed method. In experiments we evaluate how well the methods recover spectra and predict the actual RGBs and RGBs under different viewing conditions (changing illuminations and/or cameras). The results show that our method generally improves the state-of-the-art spectral recovery (with more stabilized performance when exposure varies) and provides zero colorimetric error. Moreover, our method significantly improves the color fidelity under different viewing conditions, with up to a 60% reduction in some cases.

Gamut expansion of consumer camera to the CIE XYZ color gamut using a specifically designed fourth sensor channel

This paper discusses the design of an additional spectral filter (i.e., a fourth channel) to be used with existing camera sensors such that the camera's modified color gamut overlaps almost the full CIE XYZ color gamut. The proposed approach leverages on the matrix-R theory that states that the space of metamerism of a sensor, known as the metameric black space, can be determined directly from the camera's spectral sensitivities. Using this metameric black space, a novel fourth channel has been designed on the sensor that can expand the camera's gamut. The effectiveness of this idea has been demonstrated for five commercial cameras, Munsell color chips, and images taken under various illuminations. It is shown that the designed fourth channel is very effective in fitting the camera's color gamuts to CIE XYZ color gamut, reducing CIE LAB colorimetric distances, as well as the color differences between the camera's XYZ images and the true CIE XYZ images.

Differentiating Biological Colours with Few and Many Sensors: Spectral Reconstruction with RGB and Hyperspectral Cameras

BACKGROUND

The ability to discriminate between two similar or progressively dissimilar colours is important for many animals as it allows for accurately interpreting visual signals produced by key target stimuli or distractor information. Spectrophotometry objectively measures the spectral characteristics of these signals, but is often limited to point samples that could underestimate spectral variability within a single sample. Algorithms for RGB images and digital imaging devices with many more than three channels, hyperspectral cameras, have been recently developed to produce image spectrophotometers to recover reflectance spectra at individual pixel locations. We compare a linearised RGB and a hyperspectral camera in terms of their individual capacities to discriminate between colour targets of varying perceptual similarity for a human observer.

MAIN FINDINGS

(1) The colour discrimination power of the RGB device is dependent on colour similarity between the samples whilst the hyperspectral device enables the reconstruction of a unique spectrum for each sampled pixel location independently from their chromatic appearance. (2) Uncertainty associated with spectral reconstruction from RGB responses results from the joint effect of metamerism and spectral variability within a single sample.

CONCLUSION

(1) RGB devices give a valuable insight into the limitations of colour discrimination with a low number of photoreceptors, as the principles involved in the interpretation of photoreceptor signals in trichromatic animals also apply to RGB camera responses. (2) The hyperspectral camera architecture provides means to explore other important aspects of colour vision like the perception of certain types of camouflage and colour constancy where multiple, narrow-band sensors increase resolution.

Linearisation of RGB camera responses for quantitative image analysis of visible and UV photography: a comparison of two techniques

Linear camera responses are required for recovering the total amount of incident irradiance, quantitative image analysis, spectral reconstruction from camera responses and characterisation of spectral sensitivity curves. Two commercially-available digital cameras equipped with Bayer filter arrays and sensitive to visible and near-UV radiation were characterised using biexponential and Bézier curves. Both methods successfully fitted the entire characteristic curve of the tested devices, allowing for an accurate recovery of linear camera responses, particularly those corresponding to the middle of the exposure range. Nevertheless the two methods differ in the nature of the required input parameters and the uncertainty associated with the recovered linear camera responses obtained at the extreme ends of the exposure range. Here we demonstrate the use of both methods for retrieving information about scene irradiance, describing and quantifying the uncertainty involved in the estimation of linear camera responses.

Generalized inverse-approach model for spectral-signal recovery

We have studied the transformation system of a spectral signal to the response of the system as a linear mapping from higher to lower dimensional space in order to look more closely at inverse-approach models. The problem of spectral-signal recovery from the response of a transformation system is generally stated on the basis of the generalized inverse-approach theorem, which provides a modular model for generating a spectral signal from a given response value. The controlling criteria, including the robustness of the inverse model to perturbations of the response caused by noise, and the condition number for matrix inversion, are proposed, together with the mean square error, so as to create an efficient model for spectral-signal recovery. The spectral-reflectance recovery and color correction of natural surface color are numerically investigated to appraise different illuminant-observer transformation matrices based on the proposed controlling criteria both in the absence and the presence of noise.

Spectral color constancy using a maximum entropy approach

This paper proposes a solution to the spectral color constancy problem. The method is based on a statistical model for the surface reflectance spectrum and applies a maximum entropy constraint. Unlike prior methods based on linear models, the solution process does not require a set of basis functions to be defined, nor does it require a database of spectra to be specified in advance. Experiments on simulated and real data show that spectral estimation using the maximum entropy approach is feasible and performs similarly to existing spectral methods in spite of the lower level of a priori information required.

A spectral theory of color perception

The paper adopts the philosophical stance that colors are real and can be identified with spectral models based on the photoreceptor signals. A statistical setting represents spectral profiles as probability density functions. This permits the use of analytic tools from the field of information geometry to determine a new kind of color space and structure deriving therefrom. In particular, the metric of the color space is shown to be the Fisher information matrix. A maximum entropy technique for spectral modeling is proposed that takes into account measurement noise. Theoretical predictions provided by our approach are compared with empirical colorfulness and color similarity data.

Spectral image reconstruction using an edge preserving spatio-spectral Wiener estimation

Reconstruction of spectral images from camera responses is investigated using an edge preserving spatio-spectral Wiener estimation. A Wiener denoising filter and a spectral reconstruction Wiener filter are combined into a single spatio-spectral filter using local propagation of the noise covariance matrix. To preserve edges the local mean and covariance matrix of camera responses is estimated by bilateral weighting of neighboring pixels. We derive the edge-preserving spatio-spectral Wiener estimation by means of Bayesian inference and show that it fades into the standard Wiener reflectance estimation shifted by a constant reflectance in case of vanishing noise. Simulation experiments conducted on a six-channel camera system and on multispectral test images show the performance of the filter, especially for edge regions. A test implementation of the method is provided as a MATLAB script at the first author's website.

Piecewise Wiener estimation for reconstruction of spectral reflectance image by multipoint spectral measurements

This study proposes a piecewise Wiener estimation method to reconstruct a spectral reflectance image from a three-band image by multipoint spectral information collected simultaneously with image acquisition. A three-band image is divided into several blocks and the spectral estimation is carried out using the Wiener estimation matrix assigned to each block. Each Wiener estimation matrix is constructed on the basis of spectral measurement data. The experimental results show that the proposed method reduces the average estimation error monotonically as the number of spectral measurements increases. In addition, the computational time of the piecewise Wiener estimation costs only severalfold of the computational time of the conventional single-matrix method.

Recovery of spectral reflectances of objects being imaged by multispectral cameras

Acquisition of spectral information of objects being imaged through the use of sensor responses is important to reproduce color images under various illuminations. In the past several models have been proposed to recover the spectral reflectances from sensor responses. The accuracy of the spectral reflectances recovered by five different models is compared by using multispectral cameras. It is shown that the Wiener estimation that uses the noise variance estimated as proposed in IEEE Trans. Image Process.15, 1848 (2006) recovers the spectral reflectances more accurately than the others when the test samples are different from learning samples.