CVCC Model: Learning-Based Computer Vision Color Constancy with RiR-DSN Architecture

To achieve computer vision color constancy (CVCC), it is vital but challenging to estimate scene illumination from a digital image, which distorts the true color of an object. Estimating illumination as accurately as possible is fundamental to improving the quality of the image processing pipeline. CVCC has a long history of research and has significantly advanced, but it has yet to overcome some limitations such as algorithm failure or accuracy decreasing under unusual circumstances. To cope with some of the bottlenecks, this article presents a novel CVCC approach that introduces a residual-in-residual dense selective kernel network (RiR-DSN). As its name implies, it has a residual network in a residual network (RiR) and the RiR houses a dense selective kernel network (DSN). A DSN is composed of selective kernel convolutional blocks (SKCBs). The SKCBs, or neurons herein, are interconnected in a feed-forward fashion. Every neuron receives input from all its preceding neurons and feeds the feature maps into all its subsequent neurons, which is how information flows in the proposed architecture. In addition, the architecture has incorporated a dynamic selection mechanism into each neuron to ensure that the neuron can modulate filter kernel sizes depending on varying intensities of stimuli. In a nutshell, the proposed RiR-DSN architecture features neurons called SKCBs and a residual block in a residual block, which brings several benefits such as alleviation of the vanishing gradients, enhancement of feature propagation, promotion of the reuse of features, modulation of receptive filter sizes depending on varying intensities of stimuli, and a dramatic drop in the number of parameters. Experimental results highlight that the RiR-DSN architecture performs well above its state-of-the-art counterparts, as well as proving to be camera- and illuminant-invariant.

Deep Dichromatic Model Estimation Under AC Light Sources

The dichromatic reflection model has been popularly exploited for computer vison tasks, such as color constancy and highlight removal. However, dichromatic model estimation is an severely ill-posed problem. Thus, several assumptions have been commonly made to estimate the dichromatic model, such as white-light (highlight removal) and the existence of highlight regions (color constancy). In this paper, we propose a spatio-temporal deep network to estimate the dichromatic parameters under AC light sources. The minute illumination variations can be captured with high-speed camera. The proposed network is composed of two sub-network branches. From high-speed video frames, each branch generates chromaticity and coefficient matrices, which correspond to the dichromatic image model. These two separate branches are jointly learned by spatio-temporal regularization. As far as we know, this is the first work that aims to estimate all dichromatic parameters in computer vision. To validate the model estimation accuracy, it is applied to color constancy and highlight removal. Both experimental results show that the dichromatic model can be estimated accurately via the proposed deep network.

CNN-Based Illumination Estimation with Semantic Information

For more than a decade, both academia and industry have focused attention on the computer vision and in particular the computational color constancy (CVCC). The CVCC is used as a fundamental preprocessing task in a wide range of computer vision applications. While our human visual system (HVS) has the innate ability to perceive constant surface colors of objects under varying illumination spectra, the computer vision is facing the color constancy challenge in nature. Accordingly, this article proposes novel convolutional neural network (CNN) architecture based on the residual neural network which consists of pre-activation, atrous or dilated convolution and batch normalization. The proposed network can automatically decide what to learn from input image data and how to pool without supervision. When receiving input image data, the proposed network crops each image into image patches prior to training. Once the network begins learning, local semantic information is automatically extracted from the image patches and fed to its novel pooling layer. As a result of the semantic pooling, a weighted map or a mask is generated. Simultaneously, the extracted information is estimated and combined to form global information during training. The use of the novel pooling layer enables the proposed network to distinguish between useful data and noisy data, and thus efficiently remove noisy data during learning and evaluating. The main contribution of the proposed network is taking CVCC to higher accuracy and efficiency by adopting the novel pooling method. The experimental results demonstrate that the proposed network outperforms its conventional counterparts in estimation accuracy.

Light source calibration for multispectral imaging in surgery

PURPOSE

Live intra-operative functional imaging has multiple potential clinical applications, such as localization of ischemia, assessment of organ transplantation success and perfusion monitoring. Recent research has shown that live monitoring of functional tissue properties, such as tissue oxygenation and blood volume fraction, is possible using multispectral imaging in laparoscopic surgery. While the illuminant spectrum is typically kept constant in laparoscopic surgery and can thus be estimated from preoperative calibration images, a key challenge in open surgery originates from the dynamic changes of lighting conditions.

METHODS

The present paper addresses this challenge with a novel approach to light source calibration based on specular highlight analysis. It involves the acquisition of low-exposure time images serving as a basis for recovering the illuminant spectrum from pixels that contain a dominant specular reflectance component.

RESULTS

Comprehensive in silico and in vivo experiments with a range of different light sources demonstrate that our approach enables an accurate and robust recovery of the illuminant spectrum in the field of view of the camera, which results in reduced errors with respect to the estimation of functional tissue properties. Our approach further outperforms state-of-the-art methods proposed in the field of computer vision.

CONCLUSION

Our results suggest that low-exposure multispectral images are well suited for light source calibration via specular highlight analysis. This work thus provides an important first step toward live functional imaging in open surgery.

Physically-Based Simulation of Cosmetics via Intrinsic Image Decomposition with Facial Priors

We present a physically-based approach for simulating makeup in face images. The key idea is to decompose the face image into intrinsic image layers - namely albedo, diffuse shading, and specular highlights - which are each differently affected by cosmetics, and then manipulate each layer according to corresponding models of reflectance. Accurate intrinsic image decompositions for faces are obtained with the help of human face priors, including statistics on skin reflectance and facial geometry. The intrinsic image layers are then transformed in appearance according to measured optical properties of cosmetics and proposed adaptations of physically-based reflectance models. With this approach, realistic results are generated in a manner that preserves the personal appearance features and lighting conditions of the target face while not requiring detailed geometric and reflectance measurements. We demonstrate this technique on various forms of cosmetics including foundation, blush, lipstick, and eye shadow. Results on both images and videos exhibit a close approximation to ground truth and compare favorably to existing techniques.

Illumination separation of non-Lambertian scenes from a single hyperspectral image

In this paper, we propose a general framework to estimate the spectrum of the illumination from global specular information in a single hyperspectral image. By utilizing the specular independent subspace, we iteratively separate the reflectance components and shape a weight scheme in order to find specular-contaminated pixels. After that, the illumination can be directly estimated by factorizing the weighted specular-contaminated pixels. The proposed method enables a direct and effective decomposition of the illumination and reflectance components from a single hyperspectral image. We demonstrate the robustness and accuracy of our method on simulation and real experiments. Moreover, we capture a hyperspectral image dataset with ground-truth illumination to quantitative compare the performance.

A Data Set for Camera-Independent Color Constancy

In this paper, we provide a novel data set designed for Camera-independent color constancy research. Camera independence corresponds to the robustness of an algorithm's performance when it runs on images of the same scene taken by different cameras. Accordingly, the images in our database correspond to several laboratory and field scenes each of which is captured by three different cameras with minimal registration errors. The laboratory scenes are also captured under five different illuminations. The spectral responses of cameras and the spectral power distributions of the laboratory light sources are also provided, as they may prove beneficial for training future algorithms to achieve color constancy. For a fair evaluation of future methods, we provide guidelines for supervised methods with indicated training, validation, and testing partitions. Accordingly, we evaluate two recently proposed convolutional neural network-based color constancy algorithms as baselines for future research. As a side contribution, this data set also includes images taken by a mobile camera with color shading corrected and uncorrected results. This allows research on the effect of color shading as well.

Depth Estimation and Specular Removal for Glossy Surfaces Using Point and Line Consistency with Light-Field Cameras

Light-field cameras have now become available in both consumer and industrial applications, and recent papers have demonstrated practical algorithms for depth recovery from a passive single-shot capture. However, current light-field depth estimation methods are designed for Lambertian objects and fail or degrade for glossy or specular surfaces. The standard Lambertian photoconsistency measure considers the variance of different views, effectively enforcing point-consistency, i.e., that all views map to the same point in RGB space. This variance or point-consistency condition is a poor metric for glossy surfaces. In this paper, we present a novel theory of the relationship between light-field data and reflectance from the dichromatic model. We present a physically-based and practical method to estimate the light source color and separate specularity. We present a new photo consistency metric, line-consistency, which represents how viewpoint changes affect specular points. We then show how the new metric can be used in combination with the standard Lambertian variance or point-consistency measure to give us results that are robust against scenes with glossy surfaces. With our analysis, we can also robustly estimate multiple light source colors and remove the specular component from glossy objects. We show that our method outperforms current state-of-the-art specular removal and depth estimation algorithms in multiple real world scenarios using the consumer Lytro and Lytro Illum light field cameras.

The challenge of objective scar colour assessment in a clinical setting: using digital photography

OBJECTIVE

Scar assessment in the clinical setting is typically impeded by a lack of quantitative data and most systems rely on subjective rating scales which are user dependant and show considerable variability between raters. The growing use of digital photography in medicine suggests a more objective approach to scar evaluation. Our objective was to determine if cameras could be of practical use for measuring colour in a clinical setting.

METHOD

The measurement of colour and reflectance spectra in photographs faces two difficulties: firstly the effects of variable illumination spectra, and secondly to recover accurate colour and spectral information from the sparse red, green and blue (RGB) camera signals. As a result the colour rendition is often inaccurate, and spectral information is lost. To deal with variable illumination and other factors that systematically affect all reflectance spectra ColourWorker (a method for image-based colour measurement implemented in software) calibrates the spectral responses of the camera's RGB sensors using a colour standard in the image. To make best use of the calibrated signals, it takes advantage of the fact that although a given RGB signal can be caused by an infinite number of spectra, most natural reflectance spectra vary smoothly and have predictable forms. This means given a set of examples of spectra produced by the materials of interest, it is possible to estimate the specific spectrum that produced a given RGB signal once corrected for the illumination. We describe a method for recovering spectral and chromatic information relating to surface reflectance from ordinary digital images and apply this to analyse photographs of surgical scars, taken as part of a clinical trial, in an attempt to better quantify clinical scar assessment. It should be noted the pre-existing trial protocol did not allow for a comprehensive evaluation of the accuracy of the method which would require the spectrophotometric measurement of skin regions corresponding to those in the photographs.

RESULTS

Scar colour was estimated reliably, and with simple image analysis we were able to record the change in colour across the skin. Furthermore, we describe a simple automated assessment procedure that enables scar severity to be quantified and defined using a single scalar value easily.

CONCLUSION

Such image-based colour measurement and assessment offers considerable advantages over other current methods, including spectrometers, which measure only a single point, or printed charts.

Yarbus's Conceptions on the General Mechanisms of Color Perception

In the last series of papers published during 1975 to 1980, Alfred Yarbus tried to formulate general conceptions concerning the basic principles of retinal image processing in the human visual system. The original ideas of Yarbus were based on the results of his numerous and various experiments carried out with extraordinary inventiveness and great skill. Being concentrated primarily on the problems of color vision, Alfred Yarbus dreamed of elaborating a comprehensive model that would simulate visual information processing at the monocular precognitive level in the visual system of humans with normal trichromatic color perception. In this article, the most important of Yarbus' experimental paradigms, findings, statements, and conclusions are systematized and considered in relation to the classical theories of color perception and, in particular, fundamental theses of the Nyberg school. The perceptual model developed by Alfred Yarbus remained incomplete. Nevertheless, it is already evident that some intrinsic contradictions make it inadequate in terms of comprehensive modeling. However, certain partial advantages deserve more thorough appreciation and further investigation.

Color Constancy Using Double-Opponency

The double-opponent (DO) color-sensitive cells in the primary visual cortex (V1) of the human visual system (HVS) have long been recognized as the physiological basis of color constancy. In this work we propose a new color constancy model by imitating the functional properties of the HVS from the single-opponent (SO) cells in the retina to the DO cells in V1 and the possible neurons in the higher visual cortexes. The idea behind the proposed double-opponency based color constancy (DOCC) model originates from the substantial observation that the color distribution of the responses of DO cells to the color-biased images coincides well with the vector denoting the light source color. Then the illuminant color is easily estimated by pooling the responses of DO cells in separate channels in LMS space with the pooling mechanism of sum or max. Extensive evaluations on three commonly used datasets, including the test with the dataset dependent optimal parameters, as well as the intra- and inter-dataset cross validation, show that our physiologically inspired DOCC model can produce quite competitive results in comparison to the state-of-the-art approaches, but with a relative simple implementation and without requiring fine-tuning of the method for each different dataset.

Estimation of illuminants from projections on the Planckian locus

This paper introduces a new approach for the automatic estimation of illuminants in a digital color image. The method relies on two assumptions. First, the image is supposed to contain at least a small set of achromatic pixels. The second assumption is physical and concerns the set of possible illuminants, assumed to be well approximated by black body radiators. The proposed scheme is based on a projection of selected pixels on the Planckian locus in a well chosen chromaticity space, followed by a voting procedure yielding the estimation of the illuminant. This approach is very simple and learning-free. The voting procedure can be extended for the detection of multiple illuminants when necessary. Experiments on various databases show that the performances of this approach are similar to those of the best learning-based state-of-the-art algorithms.

Effects of chromatic image statistics on illumination induced color differences

We measure the color fidelity of visual scenes that are rendered under different (simulated) illuminants and shown on a calibrated LCD display. Observers make triad illuminant comparisons involving the renderings from two chromatic test illuminants and one achromatic reference illuminant shown simultaneously. Four chromatic test illuminants are used: two along the daylight locus (yellow and blue), and two perpendicular to it (red and green). The observers select the rendering having the best color fidelity, thereby indirectly judging which of the two test illuminants induces the smallest color differences compared to the reference. Both multicolor test scenes and natural scenes are studied. The multicolor scenes are synthesized and represent ellipsoidal distributions in CIELAB chromaticity space having the same mean chromaticity but different chromatic orientations. We show that, for those distributions, color fidelity is best when the vector of the illuminant change (pointing from neutral to chromatic) is parallel to the major axis of the scene's chromatic distribution. For our selection of natural scenes, which generally have much broader chromatic distributions, we measure a higher color fidelity for the yellow and blue illuminants than for red and green. Scrambled versions of the natural images are also studied to exclude possible semantic effects. We quantitatively predict the average observer response (i.e., the illuminant probability) with four types of models, differing in the extent to which they incorporate information processing by the visual system. Results show different levels of performance for the models, and different levels for the multicolor scenes and the natural scenes. Overall, models based on the scene averaged color difference have the best performance. We discuss how color constancy algorithms may be improved by exploiting knowledge of the chromatic distribution of the visual scene.

Computational color constancy: survey and experiments

Computational color constancy is a fundamental prerequisite for many computer vision applications. This paper presents a survey of many recent developments and state-of-the-art methods. Several criteria are proposed that are used to assess the approaches. A taxonomy of existing algorithms is proposed and methods are separated in three groups: static methods, gamut-based methods, and learning-based methods. Further, the experimental setup is discussed including an overview of publicly available datasets. Finally, various freely available methods, of which some are considered to be state of the art, are evaluated on two datasets.

A uniform framework for estimating illumination chromaticity, correspondence, and specular reflection

Based upon a new correspondence matching invariant called illumination chromaticity constancy, we present a new solution for illumination chromaticity estimation, correspondence searching, and specularity removal. Using as few as two images, the core of our method is the computation of a vote distribution for a number of illumination chromaticity hypotheses via correspondence matching. The hypothesis with the highest vote is accepted as correct. The estimated illumination chromaticity is then used together with the new matching invariant to match highlights, which inherently provides solutions for correspondence searching and specularity removal. Our method differs from the previous approaches: those treat these vision problems separately and generally require that specular highlights be detected in a preprocessing step. Also, our method uses more images than previous illumination chromaticity estimation methods, which increases its robustness because more inputs/constraints are used. Experimental results on both synthetic and real images demonstrate the effectiveness of the proposed method.

Integrating color constancy into JPEG2000

The human visual system is able to perceive colors as approximately constant. This ability is known as color constancy. In contrast, the colors measured by a sensor vary with the type of illuminant used. Color constancy is very important for digital photography and automatic color-based object recognition. In digital photography, this ability is known under the name automatic white balance. A number of algorithms have been developed for color constancy. We review two well-known color constancy algorithms, the gray world assumption and the Retinex algorithm and show how a color constancy algorithm may be integrated into the JPEG2000 framework. Since computer images are usually stored in compressed form anyway, little overhead is required to add color constancy into the processing pipeline.

Color constancy from blackbody illumination

We present a theoretical analysis of what we believe to be a new color constancy method that inputs two chromaticities of an identical surface taken under two blackbody illuminations. By using the Planck formula for modeling spectra of outdoor illumination and by assuming that a narrowband camera sensitivity function is sufficiently narrow, surface colors can be estimated mathematically. Experiments with simulation and real data have been conducted to evaluate the effectiveness of the method. The results showed that although this method is a perfect vehicle for simulation data, it produces significant errors with real data. A thorough investigation of the cause of errors indicates how important the assumptions on both blackbody illuminations and narrowband camera sensitivities are to the method. Finally, we discuss the robustness of our method and the limitation of solving color constancy using the illumination constraint.

A multilinear constraint on dichromatic planes for illumination estimation

A new multilinear constraint on the color of the scene illuminant based on the dichromatic reflection model is proposed. The formulation avoids the problem, common to previous dichromatic methods, of having to first identify pixels corresponding to the same surface material. Once pixels from two or more materials have been identified, their corresponding dichromatic planes can be intersected to yield the illuminant color. However, it is not always easy to determine which pixels from an arbitrary region of an image belong to which dichromatic plane. The image region may cover an area of the scene encompassing several different materials and, hence, pixels from several different dichromatic planes. The new multilinear constraint accounts for this multiplicity of materials and provides a mechanism for choosing the most plausible illuminant from a finite set of candidate illuminants. The performance of this new method is tested on a database of real images.

Light source position and reflectance estimation from a single view without the distant illumination assumption

Several techniques have been developed for recovering reflectance properties of real surfaces under unknown illumination. However, in most cases, those techniques assume that the light sources are located at inifinity, which cannot be applied safely to, for example, reflectance modeling of indoor environments. In this paper, we propose two types of methods to estimate the surface reflectance property of an object, as well as the position of a light source from a single view without the distant illumination assumption, thus relaxing the conditions in the previous methods. Given a real image and a 3D geometric model of an object with specular reflection as inputs, the first method estimates the light source position by fitting to the Lambertian diffuse component, while separating the specular and diffuse components by using an iterative relaxation scheme. Our second method extends that first method by using as input a specular component image, which is acquired by analyzing multiple polarization images taken from a single view, thus removing its constraints on the diffuse reflectance property. This method simultaneously recovers the reflectance properties and the light source positions by optimizing the linearity of a log-transformed Torrance-Sparrow model. By estimating the object's reflectance property and the light source position, we can freely generate synthetic images of the target object under arbitrary lighting conditions with not only source direction modification but also source-surface distance modification. Experimental results show the accuracy of our estimation framework.

Separating reflection components based on chromaticity and noise analysis

Many algorithms in computer vision assume diffuse only reflections and deem specular reflections to be outliers. However, in the real world, the presence of specular reflections is inevitable since there are many dielectric inhomogeneous objects which have both diffuse and specular reflections. To resolve this problem, we present a method to separate the two reflection components. The method is principally based on the distribution of specular and diffuse points in a two-dimensional maximum chromaticity-intensity space. We found that, by utilizing the space and known illumination color, the problem of reflection component separation can be simplified into the problem of identifying diffuse maximum chromaticity. To be able to identify the diffuse maximum chromaticity correctly, an analysis of the noise is required since most real images suffer from it. Unlike existing methods, the proposed method can separate the reflection components robustly for any kind of surface roughness and light direction.

Spectral filter optimization for the recovery of parameters which describe human skin

This paper presents a method for finding spectral filters that minimize the error associated with histological parameters characterizing normal skin tissue. These parameters can be recovered from digital images of the skin using a physics-based model of skin coloration. The relationship between the image data and histological parameter values is defined as a mapping function from the image space to the parameter space. The accuracy of this function is determined by the choice of optical filters. An optimization criterion for finding the optimal filters is defined by combing methodology from differential geometry with statistical error analysis. It is shown that the magnitude of errors associated with the optimal filters is typically half of that for typical RGB filters on a three-parameter model of human skin coloration. Finally, other medical image applications are identified to which this generic methodology could be applied.

Color constancy through inverse-intensity chromaticity space

Existing color constancy methods cannot handle both uniformly colored surfaces and highly textured surfaces in a single integrated framework. Statistics-based methods require many surface colors and become error prone when there are only a few surface colors. In contrast, dichromatic-based methods can successfully handle uniformly colored surfaces but cannot be applied to highly textured surfaces, since they require precise color segmentation. We present a single integrated method to estimate illumination chromaticity from single-colored and multicolored surfaces. Unlike existing dichromatic-based methods, the proposed method requires only rough highlight regions without segmenting the colors inside them. We show that, by analyzing highlights, a direct correlation between illumination chromaticity and image chromaticity can be obtained. This correlation is clearly described in "inverse-intensity chromaticity space," a novel two-dimensional space that we introduce. In addition, when Hough transform and histogram analysis is utilized in this space, illumination chromaticity can be estimated robustly, even for a highly textured surface.