Dimensionality Reduction for Hyperspectral Data Based on Class-Aware Tensor Neighborhood Graph and Patch Alignment

HSGAN: Hyperspectral Reconstruction From RGB Images With Generative Adversarial Network

Hyperspectral (HS) reconstruction from RGB images denotes the recovery of whole-scene HS information, which has attracted much attention recently. State-of-the-art approaches often adopt convolutional neural networks to learn the mapping for HS reconstruction from RGB images. However, they often do not achieve high HS reconstruction performance across different scenes consistently. In addition, their performance in recovering HS images from clean and real-world noisy RGB images is not consistent. To improve the HS reconstruction accuracy and robustness across different scenes and from different input images, we present an effective HSGAN framework with a two-stage adversarial training strategy. The generator is a four-level top-down architecture that extracts and combines features on multiple scales. To generalize well to real-world noisy images, we further propose a spatial-spectral attention block (SSAB) to learn both spatial-wise and channel-wise relations. We conduct the HS reconstruction experiments from both clean and real-world noisy RGB images on five well-known HS datasets. The results demonstrate that HSGAN achieves superior performance to existing methods. Please visit https://github.com/zhaoyuzhi/HSGAN to try our codes.

Deep Multiscale Detail Networks for Multiband Spectral Image Sharpening

We introduce a new deep detail network architecture with grouped multiscale dilated convolutions to sharpen images contain multiband spectral information. Specifically, our end-to-end network directly fuses low-resolution multispectral and panchromatic inputs to produce high-resolution multispectral results, which is the same goal of the pansharpening in remote sensing. The proposed network architecture is designed by utilizing our domain knowledge and considering the two aims of the pansharpening: spectral and spatial preservations. For spectral preservation, the up-sampled multispectral images are directly added to the output for lossless spectral information propagation. For spatial preservation, we train the proposed network in the high-frequency domain instead of the commonly used image domain. Different from conventional network structures, we remove pooling and batch normalization layers to preserve spatial information and improve generalization to new satellites, respectively. To effectively and efficiently obtain multiscale contextual features at a fine-grained level, we propose a grouped multiscale dilated network structure to enlarge the receptive fields for each network layer. This structure allows the network to capture multiscale representations without increasing the parameter burden and network complexity. These representations are finally utilized to reconstruct the residual images which contain spatial details of PAN. Our trained network is able to generalize different satellite images without the need for parameter tuning. Moreover, our model is a general framework, which can be directly used for other kinds of multiband spectral image sharpening, e.g., hyperspectral image sharpening. Experiments show that our model performs favorably against compared methods in terms of both qualitative and quantitative qualities.

Accurate Tensor Completion via Adaptive Low-Rank Representation

Low-rank representation-based approaches that assume low-rank tensors and exploit their low-rank structure with appropriate prior models have underpinned much of the recent progress in tensor completion. However, real tensor data only approximately comply with the low-rank requirement in most cases, viz., the tensor consists of low-rank (e.g., principle part) as well as non-low-rank (e.g., details) structures, which limit the completion accuracy of these approaches. To address this problem, we propose an adaptive low-rank representation model for tensor completion that represents low-rank and non-low-rank structures of a latent tensor separately in a Bayesian framework. Specifically, we reformulate the CANDECOMP/PARAFAC (CP) tensor rank and develop a sparsity-induced prior for the low-rank structure that can be used to determine tensor rank automatically. Then, the non-low-rank structure is modeled using a mixture of Gaussians prior that is shown to be sufficiently flexible and powerful to inform the completion process for a variety of real tensor data. With these two priors, we develop a Bayesian minimum mean-squared error estimate framework for inference. The developed framework can capture the important distinctions between low-rank and non-low-rank structures, thereby enabling more accurate model, and ultimately, completion. For various applications, compared with the state-of-the-art methods, the proposed model yields more accurate completion results.

Cascade Superpixel Regularized Gabor Feature Fusion for Hyperspectral Image Classification

A 3-D Gabor wavelet provides an effective way to obtain the spectral-spatial-fused features for hyperspectral image, which has shown advantageous performance for material classification and recognition. In this paper, instead of separately employing the Gabor magnitude and phase features, which, respectively, reflect the intensity and variation of surface materials in local area, a cascade superpixel regularized Gabor feature fusion (CSRGFF) approach has been proposed. First, the Gabor filters with particular orientation are utilized to obtain Gabor features (including magnitude and phase) from the original hyperspectral image. Second, a support vector machine (SVM)-based probability representation strategy is developed to fully exploit the decision information in SVM output, and the achieved confidence score can make the following fusion with Gabor phase more effective. Meanwhile, the quadrant bit coding and Hamming distance metric are applied to encode the Gabor phase features and measure sample similarity in sequence. Third, the carefully defined characteristics of two kinds of features are directly combined together without any weighting operation to describe the weight of samples belonging to each class. Finally, a series of superpixel graphs extracted from the raw hyperspectral image with different numbers of superpixels are employed to successively regularize the weighting cube from over-segmentation to under-segmentation, and the classification performance gradually improves with the decrease in the number of superpixels in the regularization procedure. Four widely used real hyperspectral images have been conducted, and the experimental results constantly demonstrate the superiority of our CSRGFF approach over several state-of-the-art methods.

Learning a Low Tensor-Train Rank Representation for Hyperspectral Image Super-Resolution

Hyperspectral images (HSIs) with high spectral resolution only have the low spatial resolution. On the contrary, multispectral images (MSIs) with much lower spectral resolution can be obtained with higher spatial resolution. Therefore, fusing the high-spatial-resolution MSI (HR-MSI) with low-spatial-resolution HSI of the same scene has become the very popular HSI super-resolution scheme. In this paper, a novel low tensor-train (TT) rank (LTTR)-based HSI super-resolution method is proposed, where an LTTR prior is designed to learn the correlations among the spatial, spectral, and nonlocal modes of the nonlocal similar high-spatial-resolution HSI (HR-HSI) cubes. First, we cluster the HR-MSI cubes as many groups based on their similarities, and the HR-HSI cubes are also clustered according to the learned cluster structure in the HR-MSI cubes. The HR-HSI cubes in each group are much similar to each other and can constitute a 4-D tensor, whose four modes are highly correlated. Therefore, we impose the LTTR constraint on these 4-D tensors, which can effectively learn the correlations among the spatial, spectral, and nonlocal modes because of the well-balanced matricization scheme of TT rank. We formulate the super-resolution problem as TT rank regularized optimization problem, which is solved via the scheme of alternating direction method of multipliers. Experiments on HSI data sets indicate the effectiveness of the LTTR-based method.

Tensor Based Multiscale Low Rank Decomposition for Hyperspectral Images Dimensionality Reduction

Dimensionality reduction is an essential and important issue in hyperspectral image processing. With the advantages of preserving the spatial neighborhood information and the global structure information, tensor analysis and low rank representation have been widely considered in this field and yielded satisfactory performance. In available tensor- and low rank-based methods, how to construct appropriate tensor samples and determine the optimal rank of hyperspectral images along each mode are still challenging issues. To address these drawbacks, an unsupervised tensor-based multiscale low rank decomposition (T-MLRD) method for hyperspectral images dimensionality reduction is proposed in this paper. By regarding the raw cube hyperspectral image as the only tensor sample, T-MLRD needs no labeled samples and avoids the processing of constructing tensor samples. In addition, a novel multiscale low rank estimating method is proposed to obtain the optimal rank along each mode of hyperspectral image which avoids the complicated rank computing. Finally, the multiscale low rank feature representation is fused to achieve dimensionality reduction. Experimental results on real hyperspectral datasets demonstrate the superiority of the proposed method over several state-of-the-art approaches.

A Sequentially Truncated Higher Order Singular Value Decomposition-Based Algorithm for Tensor Completion

The problem of recovering missing data of an incomplete tensor has drawn more and more attentions in the fields of pattern recognition, machine learning, data mining, computer vision, and signal processing. Researches on this problem usually share a common assumption that the original tensor is of low-rank. One of the important ways to capture the low-rank structure of the incomplete tensor is based on tensor factorization. For the traditional tensor factorization algorithms, the tensor ranks should be specified ahead, which is not reasonable in real applications. To overcome this drawback, an adaptive algorithm is first presented based on sequentially truncated higher order singular value decomposition (ST-HOSVD) for fast low-rank approximation of complete tensor, in which the tensor ranks can be obtained adaptively. Then for tensor with missing data, we use adaptive ST-HOSVD and the average operator of low-rank approximation to improve the accuracy of the fulfilled tensor. Convergence analysis of the proposed algorithm is also given in this paper. The experimental results on 14 image datasets and three video datasets show that the proposed method outperforms the state-of-the-art methods in terms of running time and the accuracy.

Self-Paced Learning-Based Probability Subspace Projection for Hyperspectral Image Classification

In this paper a self-paced learning-based probability subspace projection (SL-PSP) method is proposed for hyperspectral image classification. First, a probability label is assigned for each pixel, and a risk is assigned for each labeled pixel. Then, two regularizers are developed from a self-paced maximum margin and a probability label graph, respectively. The first regularizer can increase the discriminant ability of features by gradually involving the most confident pixels into the projection to simultaneously push away heterogeneous neighbors and pull inhomogeneous neighbors. The second regularizer adopts a relaxed clustering assumption to make avail of unlabeled samples, thus accurately revealing the affinity between mixed pixels and achieving accurate classification with very few labeled samples. Several hyperspectral data sets are used to verify the effectiveness of SL-PSP, and the experimental results show that it can achieve the state-of-the-art results in terms of accuracy and stability.

Deep Hyperspectral Image Sharpening

Hyperspectral image (HSI) sharpening, which aims at fusing an observable low spatial resolution (LR) HSI (LR-HSI) with a high spatial resolution (HR) multispectral image (HR-MSI) of the same scene to acquire an HR-HSI, has recently attracted much attention. Most of the recent HSI sharpening approaches are based on image priors modeling, which are usually sensitive to the parameters selection and time-consuming. This paper presents a deep HSI sharpening method (named DHSIS) for the fusion of an LR-HSI with an HR-MSI, which directly learns the image priors via deep convolutional neural network-based residual learning. The DHSIS method incorporates the learned deep priors into the LR-HSI and HR-MSI fusion framework. Specifically, we first initialize the HR-HSI from the fusion framework via solving a Sylvester equation. Then, we map the initialized HR-HSI to the reference HR-HSI via deep residual learning to learn the image priors. Finally, the learned image priors are returned to the fusion framework to reconstruct the final HR-HSI. Experimental results demonstrate the superiority of the DHSIS approach over existing state-of-the-art HSI sharpening approaches in terms of reconstruction accuracy and running time.

2D Feature Selection by Sparse Matrix Regression

For many image processing and computer vision problems, data points are in matrix form. Traditional methods often convert a matrix into a vector and then use vector-based approaches. They will ignore the location of matrix elements and the converted vector often has high dimensionality. How to select features for 2D matrix data directly is still an uninvestigated important issue. In this paper, we propose an algorithm named sparse matrix regression (SMR) for direct feature selection on matrix data. It employs the matrix regression model to accept matrix as input and bridges each matrix to its label. Based on the intrinsic property of regression coefficients, we design some sparse constraints on the coefficients to perform feature selection. An effective optimization method with provable convergence behavior is also proposed. We reveal that the number of regression vectors can be regarded as a tradeoff parameter to balance the capacity of learning and generalization in essence. To examine the effectiveness of SMR, we have compared it with several vector-based approaches on some benchmark data sets. Furthermore, we have also evaluated SMR in the application of scene classification. They all validate the effectiveness of our method.

Discovering Diverse Subset for Unsupervised Hyperspectral Band Selection

Band selection, as a special case of the feature selection problem, tries to remove redundant bands and select a few important bands to represent the whole image cube. This has attracted much attention, since the selected bands provide discriminative information for further applications and reduce the computational burden. Though hyperspectral band selection has gained rapid development in recent years, it is still a challenging task because of the following requirements: 1) an effective model can capture the underlying relations between different high-dimensional spectral bands; 2) a fast and robust measure function can adapt to general hyperspectral tasks; and 3) an efficient search strategy can find the desired selected bands in reasonable computational time. To satisfy these requirements, a multigraph determinantal point process (MDPP) model is proposed to capture the full structure between different bands and efficiently find the optimal band subset in extensive hyperspectral applications. There are three main contributions: 1) graphical model is naturally transferred to address band selection problem by the proposed MDPP; 2) multiple graphs are designed to capture the intrinsic relationships between hyperspectral bands; and 3) mixture DPP is proposed to model the multiple dependencies in the proposed multiple graphs, and offers an efficient search strategy to select the optimal bands. To verify the superiority of the proposed method, experiments have been conducted on three hyperspectral applications, such as hyperspectral classification, anomaly detection, and target detection. The reliability of the proposed method in generic hyperspectral tasks is experimentally proved on four real-world hyperspectral data sets.