Hybrid-resolution spectral video system using low-resolution spectral sensor

Improving RGB illuminant estimation exploiting spectral average radiance

We introduce a method that enhances RGB color constancy accuracy by combining neural network and k-means clustering techniques. Our approach stands out from previous works because we combine multispectral and color information together to estimate illuminants. Furthermore, we investigate the combination of the illuminant estimation in the RGB color and in the spectral domains, as a strategy to provide a refined estimation in the RGB color domain. Our investigation can be divided into three main points: (1) identify the spatial resolution for sampling the input image in terms of RGB color and spectral information that brings the highest performance; (2) determine whether it is more effective to predict the illuminant in the spectral or in the RGB color domain, and finally, (3) assuming that the illuminant is in fact predicted in the spectral domain, investigate if it is better to have a loss function defined in the RGB color or spectral domain. Experimental results are carried out on NUS: a standard dataset of multispectral radiance images with an annotated spectral global illuminant. Among the several considered options, the best results are obtained with a model trained to predict the illuminant in the spectral domain using an RGB color loss function. In terms of comparison with the state of the art, this solution improves the recovery angular error metric by 66% compared to the best tested spectral method, and by 41% compared to the best tested RGB method.

Hyperspectral image super-resolution based on the transfer of both spectra and multi-level features

Existing hyperspectral image (HSI) super-resolution methods fusing a high-resolution RGB image (HR-RGB) and a low-resolution HSI (LR-HSI) always rely on spatial degradation and handcrafted priors, which hinders their practicality. To address these problems, we propose a novel, to the best of our knowledge, method with two transfer models: a window-based linear mixing (W-LM) model and a feature transfer model. Specifically, W-LM initializes a high-resolution HSI (HR-HSI) by transferring the spectra from the LR-HSI to the HR-RGB. By using the proposed feature transfer model, the HR-RGB multi-level features extracted by a pre-trained convolutional neural network (CNN) are then transferred to the initialized HR-HSI. The proposed method fully exploits spectra of LR-HSI and multi-level features of HR-RGB and achieves super-resolution without requiring the spatial degradation model and any handcrafted priors. The experimental results for 32 × super-resolution on two public datasets and our real image set demonstrate the proposed method outperforms eight state-of-the-art existing methods.

Ruggedized, field-ready snapshot light-guide-based imaging spectrometer for environmental and remote sensing applications

A field-ready, fiber-based high spatial sampling snapshot imaging spectrometer was developed for applications such as environmental monitoring and smart farming. The system achieves video rate frame transfer and exposure times down to a few hundred microseconds in typical daylight conditions with ∼63,000 spatial points and 32 spectral channels across the 470nm to 700nm wavelength range. We designed portable, ruggedized opto-mechanics to allow for imaging from an airborne platform. To ensure successful data collection prior to flight, imaging speed and signal-to-noise ratio was characterized for imaging a variety of land covers from the air. The system was validated by performing a series of observations including: Liriope Muscari plants under a range of water-stress conditions in a controlled laboratory experiment and field observations of sorghum plants in a variety of soil conditions. Finally, we collected data from a series of engineering flights and present reassembled images and spectral sampling of rural and urban landscapes.

Research on spectral reconstruction algorithm for snapshot microlens array micro-hyperspectral imaging system

Snapshot microlens array microscopic hyperspectral imaging systems do not require a scanning process and obtain (x,y,λ) three-dimensional data cubes in one shot. Currently, the three-dimensional spectra image data are interleaved on a charge-coupled device detector, which increases subsequent data processing difficulty. The optical design software cannot simulate actual engineering installation and adjustment results accurately and the tracking results cannot guide precise rapid online calibration of the snapshot microlens array microscopic hyperspectral imaging system. To solve these problems, we propose an accurate spectral image reconstruction model based on optical tracing, derive spatial dispersion equations for the prisms and gratings, establish an algorithm model for the correspondence between the microlens array's surface dispersion spectral distribution and its imaging position, and propose a three-dimensional spectral image reconstruction algorithm. Experimental results show that this algorithm's actual spectral calibration error is better than 0.2 nm. This meets the image processing requirements of snapshot microlens array microscopic hyperspectral systems.

Microlens array snapshot hyperspectral microscopy system for the biomedical domain

We propose a microlens array-type snapshot hyperspectral microscope system that can provide spatial spectrum sampling according to detector frame rates for the biomedical domain. The system uses a shared optical path design. One path is used to perform direct microscopic imaging with high spatial resolution, while the other is used to collect microscopic images through a microlens array; the images are then spatially cut and reimaged such that they are spaced simultaneously by the prism-grating type hyperspectral imager's dispersion. Rapid acquisition of a three-dimensional data cube measuring 28×14×180 (x×y×λ) can be performed at the detector's frame rate. The system has a spatial resolution of 2.5 µm and can achieve 180-channel sampling of a 100 nm spectrum in the 400-800 nm spectral range with spectral resolution of approximately 0.56 nm. Spectral imaging results from biological samples show that the microlens array-type snapshot hyperspectral microscope system may potentially be applied in real-time biological spectral imaging.

Light-guide snapshot imaging spectrometer for remote sensing applications

A fiber-based snapshot imaging spectrometer was developed with a maximum of 31853 (~188 x 170) spatial sampling and 61 spectral channels in the 450nm-750nm range. A compact, custom-fabricated fiber bundle was used to sample the object image at the input and create void spaces between rows at the output for dispersion. The bundle was built using multicore 6x6 fiber block ribbons. To avoid overlap between the cores in the direction of dispersion, we selected a subset of cores using two alternative approaches; a lenslet array and a photomask. To calibrate the >30000 spatial samples of the system, a rapid spatial calibration method was developed based on phase-shifting interferometry (PSI). System crosstalk and spectral resolution were also characterized. Preliminary hyperspectral imaging results of the Rice University campus landscape, obtained with the spectrometer, are presented to demonstrate the system's spectral imaging capability for distant scenes. The spectrum of different plant species with different health conditions, obtained with the spectrometer, was in accordance with reference instrument measurements. We also imaged Houston traffic to demonstrate the system's snapshot hyperspectral imaging capability. Potential applications of the system include terrestrial monitoring, land use, air pollution, water resources, and lightning spectroscopy. The fiber-based system design potentially allows tuning between spatial and spectral sampling to meet specific imaging requirements.

Three-area-array coherent-dispersion stereo-imaging spectrometer

A coherent-dispersion stereo-imaging spectrometer is presented, which combines three-view stereo imaging, interferometric spectroscopy and dispersive spectroscopy. Three area-array detectors record three spectral images of each scene unit from three views. For each of the three views, each scene unit is imaged on a given column of one area-array detector, and different wavelengths are dispersed across different rows of that column. For each scene unit, multiple interferograms are simultaneously generated at each view, each interferogram covering a separate wavelength range and located in a separate pixel. The orthographic view image is used to create a two-dimensional orthophoto image. The front view and back view images are used to reconstruct the three-dimensional stereoscopic image. Preliminary theoretical calculations are given. The instrument is a unique concept to obtain three-dimensional spatial information and one-dimensional spectral information while achieving high spectral resolution measurement of an ultraviolet-visible broadband spectral range (e.g., 0.05 nm at 450 nm together with 0.1 nm at 700 nm). It will be suitable for ultraviolet-visible hyperspectral remote sensing.

A High Throughput Integrated Hyperspectral Imaging and 3D Measurement System

Hyperspectral and three-dimensional measurements can obtain the intrinsic physicochemical properties and external geometrical characteristics of objects, respectively. The combination of these two kinds of data can provide new insights into objects, which has gained attention in the fields of agricultural management, plant phenotyping, cultural heritage conservation, and food production. Currently, a variety of sensors are integrated into a system to collect spectral and morphological information in agriculture. However, previous experiments were usually performed with several commercial devices on a single platform. Inadequate registration and synchronization among instruments often resulted in mismatch between spectral and 3D information of the same target. In addition, using slit-based spectrometers and point-based 3D sensors extends the working hours in farms due to the narrow field of view (FOV). Therefore, we propose a high throughput prototype that combines stereo vision and grating dispersion to simultaneously acquire hyperspectral and 3D information. Furthermore, fiber-reformatting imaging spectrometry (FRIS) is adopted to acquire the hyperspectral images. Test experiments are conducted for the verification of the system accuracy, and vegetation measurements are carried out to demonstrate its feasibility. The proposed system is an improvement in multiple data acquisition and has the potential to improve plant phenotyping.

Inverse conversion algorithm for an all-optical depth coloring camera

Three-dimensional (3D) metrology has received a lot of attention from academic and industrial communities due to its broad applications, such as 3D contents, 3D printing, and autonomous driving. The all-optical depth coloring (AODC) camera has some benefits in computation load since it extracts depth information of an object fully optically. The AODC camera represents the depth of the object as a variation of wavelength, and spectroscopy is generally required to measure the wavelength. However, in the AODC camera, the color vector in RGB color space is convertible inversely into the wavelength after projection on the normalized rgb plane because the detected spectrum through the gating part has a narrow bandwidth as a result of the width of the slit in the projection part. In this paper, we propose an inverse conversion algorithm from RGB color to depth without spectroscopy. Experimental results are presented to confirm its feasibility. Also, some practical limitations are discussed, resulting from the nonlinearity of the response of the image sensor and the widths of the slits in the projection part and the gating part.

Snapshot linear-Stokes imaging spectropolarimeter using division-of-focal-plane polarimetry and integral field spectroscopy

In this paper, the design and experimental demonstration of a snapshot linear-Stokes imaging spectropolarimeter (SLSIS) is presented. The SLSIS, which is based on division-of-focal-plane polarimetry with four parallel linear polarization channels and integral field spectroscopy with numerous slit dispersive paths, has no moving parts and provides video-rate Stokes-vector hyperspectral datacubes. It does not need any scanning in the spectral, spatial or polarization dimension and offers significant advantages of rapid reconstruction without heavy computation during post-processing. The principle and the experimental setup of the SLSIS are described in detail. The image registration, Stokes spectral reconstruction and calibration procedures are included, and the system is validated using measurements of tungsten light and a static scene. The SLSIS's snapshot ability to resolve polarization spectral signatures is demonstrated using measurements of a dynamic scene.

High resolution snapshot imaging spectrometer using a fusion algorithm based on grouping principal component analysis

We reported a high resolution snapshot imaging spectrometer (HR-SIS) and a fusion algorithm based on the properties of the HR-SIS. The system consists of an imaging branch and a spectral branch. The imaging branch captures a high spatial resolution panchromatic image with 680 × 680 pixels, while the spectral branch acquires a low spatial resolution spectral image with spectral resolution of 250 cm^-1. By using a fusion algorithm base on grouping principal component analysis, the spectral image is highly improved in spatial resolution. Experimental results demonstrated that the performance of the proposed algorithm is competitive with other state-of-the-art algorithms. The computing time for a single frame is less than 1 min with an Intel Core i5-4200H CPU, which can be further reduced by utilizing a graphics processing unit (GPU).

Spectral imaging using consumer-level devices and kernel-based regression

Hyperspectral reflectance factor image estimations were performed in the 400-700 nm wavelength range using a portable consumer-level laptop display as an adjustable light source for a trichromatic camera. Targets of interest were ColorChecker Classic samples, Munsell Matte samples, geometrically challenging tempera icon paintings from the turn of the 20th century, and human hands. Measurements and simulations were performed using Nikon D80 RGB camera and Dell Vostro 2520 laptop screen as a light source. Estimations were performed without spectral characteristics of the devices and by emphasizing simplicity for training sets and estimation model optimization. Spectral and color error images are shown for the estimations using line-scanned hyperspectral images as the ground truth. Estimations were performed using kernel-based regression models via a first-degree inhomogeneous polynomial kernel and a Matérn kernel, where in the latter case the median heuristic approach for model optimization and link function for bounded estimation were evaluated. Results suggest modest requirements for a training set and show that all estimation models have markedly improved accuracy with respect to the DE00 color distance (up to 99% for paintings and hands) and the Pearson distance (up to 98% for paintings and 99% for hands) from a weak training set (Digital ColorChecker SG) case when small representative training data were used in the estimation.

Digital holography based on multiwavelength spatial-bandwidth-extended capturing-technique using a reference arm (Multi-SPECTRA)

Single-shot digital holography based on multiwavelength spatial-bandwidth-extended capturing-technique using a reference arm (Multi-SPECTRA) is proposed. Both amplitude and quantitative phase distributions of waves containing multiple wavelengths are simultaneously recorded with a single reference arm in a single monochromatic image. Then, multiple wavelength information is separately extracted in the spatial frequency domain. The crosstalk between the object waves with different wavelengths is avoided and the number of wavelengths recorded with both a single-shot exposure and no crosstalk can be increased, by a large spatial carrier that causes the aliasing, and/or by use of a grating. The validity of Multi-SPECTRA is quantitatively, numerically, and experimentally confirmed.