Salient Band Selection for Hyperspectral Image Classification via Manifold Ranking

HSGAN: Hyperspectral Reconstruction From RGB Images With Generative Adversarial Network

Hyperspectral (HS) reconstruction from RGB images denotes the recovery of whole-scene HS information, which has attracted much attention recently. State-of-the-art approaches often adopt convolutional neural networks to learn the mapping for HS reconstruction from RGB images. However, they often do not achieve high HS reconstruction performance across different scenes consistently. In addition, their performance in recovering HS images from clean and real-world noisy RGB images is not consistent. To improve the HS reconstruction accuracy and robustness across different scenes and from different input images, we present an effective HSGAN framework with a two-stage adversarial training strategy. The generator is a four-level top-down architecture that extracts and combines features on multiple scales. To generalize well to real-world noisy images, we further propose a spatial-spectral attention block (SSAB) to learn both spatial-wise and channel-wise relations. We conduct the HS reconstruction experiments from both clean and real-world noisy RGB images on five well-known HS datasets. The results demonstrate that HSGAN achieves superior performance to existing methods. Please visit https://github.com/zhaoyuzhi/HSGAN to try our codes.

AUD-Net: A Unified Deep Detector for Multiple Hyperspectral Image Anomaly Detection via Relation and Few-Shot Learning

This article addresses the problem of the building an out-of-the-box deep detector, motivated by the need to perform anomaly detection across multiple hyperspectral images (HSIs) without repeated training. To solve this challenging task, we propose a unified detector [anomaly detection network (AUD-Net)] inspired by few-shot learning. The crucial issues solved by AUD-Net include: how to improve the generalization of the model on various HSIs that contain different categories of land cover; and how to unify the different spectral sizes between HSIs. To achieve this, we first build a series of subtasks to classify the relations between the center and its surroundings in the dual window. Through relation learning, AUD-Net can be more easily generalized to unseen HSIs, as the relations of the pixel pairs are shared among different HSIs. Secondly, to handle different HSIs with various spectral sizes, we propose a pooling layer based on the vector of local aggregated descriptors, which maps the variable-sized features to the same space and acquires the fixed-sized relation embeddings. To determine whether the center of the dual window is an anomaly, we build a memory model by the transformer, which integrates the contextual relation embeddings in the dual window and estimates the relation embeddings of the center. By computing the feature difference between the estimated relation embeddings of the centers and the corresponding real ones, the centers with large differences will be detected as anomalies, as they are more difficult to be estimated by the corresponding surroundings. Extensive experiments on both the simulation dataset and 13 real HSIs demonstrate that this proposed AUD-Net has strong generalization for various HSIs and achieves significant advantages over the specific-trained detectors for each HSI.

Heterogeneous Regularization-Based Tensor Subspace Clustering for Hyperspectral Band Selection

Band selection (BS) reduces effectively the spectral dimension of a hyperspectral image (HSI) by selecting relatively few representative bands, which allows efficient processing in subsequent tasks. Existing unsupervised BS methods based on subspace clustering are built on matrix-based models, where each band is reshaped as a vector. They encode the correlation of data only in the spectral mode (dimension) and neglect strong correlations between different modes, i.e., spatial modes and spectral mode. Another issue is that the subspace representation of bands is performed in the raw data space, where the dimension is often excessively high, resulting in a less efficient and less robust performance. To address these issues, in this article, we propose a tensor-based subspace clustering model for hyperspectral BS. Our model is developed on the well-known Tucker decomposition. The three factor matrices and a core tensor in our model encode jointly the multimode correlations of HSI, avoiding effectively to destroy the tensor structure and information loss. In addition, we propose well-motivated heterogeneous regularizations (HRs) on the factor matrices by taking into account the important local and global properties of HSI along three dimensions, which facilitates the learning of the intrinsic cluster structure of bands in the low-dimensional subspaces. Instead of learning the correlations of bands in the original domain, a common way for the matrix-based models, our model learns naturally the band correlations in a low-dimensional latent feature space, which is derived by the projections of two factor matrices associated with spatial dimensions, leading to a computationally efficient model. More importantly, the latent feature space is learned in a unified framework. We also develop an efficient algorithm to solve the resulting model. Experimental results on benchmark datasets demonstrate that our model yields improved performance compared to the state-of-the-art.

Weighted Sparseness-Based Anomaly Detection for Hyperspectral Imagery

Anomaly detection of hyperspectral remote sensing data has recently become more attractive in hyperspectral image processing. The low-rank and sparse matrix decomposition-based anomaly detection algorithm (LRaSMD) exhibits poor detection performance in complex scenes with multiple background edges and noise. Therefore, this study proposes a weighted sparse hyperspectral anomaly detection method. First, using the idea of matrix decomposition in mathematics, the original hyperspectral data matrix is reconstructed into three sub-matrices with low rank, small sparsity and representing noise, respectively. Second, to suppress the noise interference in the complex background, we employed the low-rank, background image as a reference, built a local spectral and spatial dictionary through the sliding window strategy, reconstructed the HSI pixels of the original data, and extracted the sparse coefficient. We proposed the sparse coefficient divergence evaluation index (SCDI) as a weighting factor to weight the sparse anomaly map to obtain a significant anomaly map to suppress the background edge, noise, and other residues caused by decomposition, and enhance the abnormal target. Finally, abnormal pixels are segmented based on the adaptive threshold. The experimental results demonstrate that, on a real-scene hyperspectral dataset with a complicated background, the proposed method outperforms the existing representative algorithms in terms of detection performance.

Learning Salient Feature for Salient Object Detection Without Labels

Supervised salient object detection (SOD) methods achieve state-of-the-art performance by relying on human-annotated saliency maps, while unsupervised methods attempt to achieve SOD by not using any annotations. In unsupervised SOD, how to obtain saliency in a completely unsupervised manner is a huge challenge. Existing unsupervised methods usually gain saliency by introducing other handcrafted feature-based saliency methods. In general, the location information of salient objects is included in the feature maps. If the features belonging to salient objects are called salient features and the features that do not belong to salient objects, such as background, are called nonsalient features, by dividing the feature maps into salient features and nonsalient features in an unsupervised way, then the object at the location of the salient feature is the salient object. Based on the above motivation, a novel method called learning salient feature (LSF) is proposed, which achieves unsupervised SOD by LSF from the data itself. This method takes enhancing salient feature and suppressing nonsalient features as the objective. Furthermore, a salient object localization method is proposed to roughly locate objects where the salient feature is located, so as to obtain the salient activation map. Usually, the object in the salient activation map is incomplete and contains a lot of noise. To address this issue, a saliency map update strategy is introduced to gradually remove noise and strengthen boundaries. The visualization of images and their salient activation maps show that our method can effectively learn salient visual objects. Experiments show that we achieve superior unsupervised performance on a series of datasets.

Dimensionality reduction strategies for land use land cover classification based on airborne hyperspectral imagery: a survey

Hyperspectral image (HSI) contains hundreds of adjacent spectral bands, which can effectively differentiate the region of interest. Nevertheless, many irrelevant and highly correlated spectral bands lead to the Hughes phenomenon. Consequently, hyperspectral image dimensionality reduction is necessary to select the most informative and significant spectral band and eliminate the redundant spectral band. To this end, this paper represents an extensive and systematic survey of hyperspectral dimensionality reduction approaches for land use land cover (LULC) classification. Moreover, this paper reviewed the following important points: (1) hyperspectral imaging data acquisition methods, (2) the difference between hyperspectral and multispectral images, (3) hyperspectral image dimensionality reduction based on machine learning (ML) and deep learning (DL) techniques, (4) the popular benchmark hyperspectral datasets with the performance metrics for LULC classification, and (5) the significant challenges with the future trends for hyperspectral dimensionality reduction.

PANet: Patch-Aware Network for Light Field Salient Object Detection

Most existing light field saliency detection methods have achieved great success by exploiting unique light field data-focus information in focal slices. However, they process light field data in a slicewise way, leading to suboptimal results because the relative contribution of different regions in focal slices is ignored. How we can comprehensively explore and integrate focused saliency regions that would positively contribute to accurate saliency detection. Answering this question inspires us to develop a new insight. In this article, we propose a patch-aware network to explore light field data in a regionwise way. First, we excavate focused salient regions with a proposed multisource learning module (MSLM), which generates a filtering strategy for integration followed by three guidances based on saliency, boundary, and position. Second, we design a sharpness recognition module (SRM) to refine and update this strategy and perform feature integration. With our proposed MSLM and SRM, we can obtain more accurate and complete saliency maps. Comprehensive experiments on three benchmark datasets prove that our proposed method achieves competitive performance over 2-D, 3-D, and 4-D salient object detection methods. The code and results of our method are available at https://github.com/OIPLab-DUT/IEEE-TCYB-PANet.

Adjacent Context Coordination Network for Salient Object Detection in Optical Remote Sensing Images

Salient object detection (SOD) in optical remote sensing images (RSIs), or RSI-SOD, is an emerging topic in understanding optical RSIs. However, due to the difference between optical RSIs and natural scene images (NSIs), directly applying NSI-SOD methods to optical RSIs fails to achieve satisfactory results. In this article, we propose a novel adjacent context coordination network (ACCoNet) to explore the coordination of adjacent features in an encoder-decoder architecture for RSI-SOD. Specifically, ACCoNet consists of three parts: 1) an encoder; 2) adjacent context coordination modules (ACCoMs); and 3) a decoder. As the key component of ACCoNet, ACCoM activates the salient regions of output features of the encoder and transmits them to the decoder. ACCoM contains a local branch and two adjacent branches to coordinate the multilevel features simultaneously. The local branch highlights the salient regions in an adaptive way, while the adjacent branches introduce global information of adjacent levels to enhance salient regions. In addition, to extend the capabilities of the classic decoder block (i.e., several cascaded convolutional layers), we extend it with two bifurcations and propose a bifurcation-aggregation block (BAB) to capture the contextual information in the decoder. Extensive experiments on two benchmark datasets demonstrate that the proposed ACCoNet outperforms 22 state-of-the-art methods under nine evaluation metrics, and runs up to 81 fps on a single NVIDIA Titan X GPU. The code and results of our method are available at https://github.com/MathLee/ACCoNet.

Weakly Supervised Discriminative Learning With Spectral Constrained Generative Adversarial Network for Hyperspectral Anomaly Detection

Anomaly detection (AD) using hyperspectral images (HSIs) is of great interest for deep space exploration and Earth observations. This article proposes a weakly supervised discriminative learning with a spectral constrained generative adversarial network (GAN) for hyperspectral anomaly detection (HAD), called weaklyAD. It can enhance the discrimination between anomaly and background with background homogenization and anomaly saliency in cases where anomalous samples are limited and sensitive to the background. A novel probability-based category thresholding is first proposed to label coarse samples in preparation for weakly supervised learning. Subsequently, a discriminative reconstruction model is learned by the proposed network in a weakly supervised fashion. The proposed network has an end-to-end architecture, which not only includes an encoder, a decoder, a latent layer discriminator, and a spectral discriminator competitively but also contains a novel Kullback-Leibler (KL) divergence-based orthogonal projection divergence (OPD) spectral constraint. Finally, the well-learned network is used to reconstruct HSIs captured by the same sensor. Our work paves a new weakly supervised way for HAD, which intends to match the performance of supervised methods without the prerequisite of manually labeled data. Assessments and generalization experiments over real HSIs demonstrate the unique promise of such a proposed approach.

Deep Manifold Embedding for Hyperspectral Image Classification

Deep learning methods have played a more important role in hyperspectral image classification. However, general deep learning methods mainly take advantage of the samplewise information to formulate the training loss while ignoring the intrinsic data structure of each class. Due to the high spectral dimension and great redundancy between different spectral channels in the hyperspectral image, these former training losses usually cannot work so well for the deep representation of the image. To tackle this problem, this work develops a novel deep manifold embedding method (DMEM) for deep learning in hyperspectral image classification. First, each class in the image is modeled as a specific nonlinear manifold, and the geodesic distance is used to measure the correlation between the samples. Then, based on the hierarchical clustering, the manifold structure of the data can be captured and each nonlinear data manifold can be divided into several subclasses. Finally, considering the distribution of each subclass and the correlation between different subclasses under data manifold, DMEM is constructed as the novel training loss to incorporate the special classwise information in the training process and obtain discriminative representation for the hyperspectral image. Experiments over four real-world hyperspectral image datasets have demonstrated the effectiveness of the proposed method when compared with general sample-based losses and showed superiority when compared with state-of-the-art methods.

Hyperspectral Image Classification via Discriminant Gabor Ensemble Filter

For a broad range of applications, hyperspectral image (HSI) classification is a hot topic in remote sensing, and convolutional neural network (CNN)-based methods are drawing increasing attention. However, to train millions of parameters in CNN requires a large number of labeled training samples, which are difficult to collect. A conventional Gabor filter can effectively extract spatial information with different scales and orientations without training, but it may be missing some important discriminative information. In this article, we propose the Gabor ensemble filter (GEF), a new convolutional filter to extract deep features for HSI with fewer trainable parameters. GEF filters each input channel by some fixed Gabor filters and learnable filters simultaneously, then reduces the dimensions by some learnable 1×1 filters to generate the output channels. The fixed Gabor filters can extract common features with different scales and orientations, while the learnable filters can learn some complementary features that Gabor filters cannot extract. Based on GEF, we design a network architecture for HSI classification, which extracts deep features and can learn from limited training samples. In order to simultaneously learn more discriminative features and an end-to-end system, we propose to introduce the local discriminant structure for cross-entropy loss by combining the triplet hard loss. Results of experiments on three HSI datasets show that the proposed method has significantly higher classification accuracy than other state-of-the-art methods. Moreover, the proposed method is speedy for both training and testing.

A Novel Method for Fast Kernel Minimum Noise Fraction Transformation in Hyperspectral Image Dimensionality Reduction

Feature extraction, aiming to simplify and optimize data features, is a typical hyperspectral image dimensionality reduction technique. As a kernel-based method, kernel minimum noise fraction (KMNF) transformation is excellent at handling the nonlinear features within HSIs. It adopts the kernel function to ensure data linear separability by transforming the original data to a higher feature space, following which a linear analysis can be performed in this space. However, KMNF transformation has the problem of high computational complexity and low execution efficiency. It is not suitable for the processing of large-scale datasets. In terms of this problem, this paper proposes a graphics processing unit (GPU) and Nyström method-based algorithm for Fast KMNF transformation (GNKMNF). First, the Nyström method estimates the eigenvector of the entire kernel matrix in KMNF transformation by the decomposition and extrapolation of the sub-kernel matrix to reduce the computational complexity. Then, the sample size in the Nyström method is determined utilizing a proportional gradient selection strategy. Finally, GPU parallel computing is employed to further improve the execution efficiency. Experimental results show that compared with KMNF transformation, improvements of up to 1.94% and 2.04% are achieved by GNKMNF in overall classification accuracy and Kappa, respectively. Moreover, with a data size of 64 × 64 × 250, the execution efficiency of GNKMNF speeds up by about 80x. The outcome demonstrates the significant performance of GNKMNF in feature extraction and execution efficiency.

Hyperspectral Image Classification Based on 3D Asymmetric Inception Network with Data Fusion Transfer Learning

Hyperspectral image (HSI) classification has been marked by exceptional progress in recent years. Much of this progess has come from advances in convolutional neural networks (CNNs). Different from the RGB images, HSI images are captured by various remote sensors with different spectral configurations. Moreover, each HSI dataset only contains very limited training samples and thus the model is prone to overfitting when using deep CNNs. In this paper, we first propose a 3D asymmetric inception network, AINet, to overcome the overfitting problem. With the emphasis on spectral signatures over spatial contexts of HSI data, the 3D convolution layer of AINet is replaced with two asymmetric inception units, i.e., a space inception unit and spectrum inception unit, to convey and classify the features effectively. In addition, we exploited a data-fusion transfer learning strategy to improve model initialization and classification performance. Extensive experiments show that the proposed approach outperforms all of the state-of-the-art methods via several HSI benchmarks, including Pavia University, Indian Pines and Kennedy Space Center (KSC).

Highlight Every Step: Knowledge Distillation via Collaborative Teaching

High storage and computational costs obstruct deep neural networks to be deployed on resource-constrained devices. Knowledge distillation (KD) aims to train a compact student network by transferring knowledge from a larger pretrained teacher model. However, most existing methods on KD ignore the valuable information among the training process associated with training results. In this article, we provide a new collaborative teaching KD (CTKD) strategy which employs two special teachers. Specifically, one teacher trained from scratch (i.e., scratch teacher) assists the student step by step using its temporary outputs. It forces the student to approach the optimal path toward the final logits with high accuracy. The other pretrained teacher (i.e., expert teacher) guides the student to focus on a critical region that is more useful for the task. The combination of the knowledge from two special teachers can significantly improve the performance of the student network in KD. The results of experiments on CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, and ImageNet datasets verify that the proposed KD method is efficient and achieves state-of-the-art performance.

Time Series Surface Temperature Prediction Based on Cyclic Evolutionary Network Model for Complex Sea Area

The prediction of marine elements has become increasingly important in the field of marine research. However, time series data in a complex environment vary significantly because they are composed of dynamic changes with multiple mechanisms, causes, and laws. For example, sea surface temperature (SST) can be influenced by ocean currents. Conventional models often focus on capturing the impact of historical data but ignore the spatio–temporal relationships in sea areas, and they cannot predict such widely varying data effectively. In this work, we propose a cyclic evolutionary network model (CENS), an error-driven network group, which is composed of multiple network node units. Different regions of data can be automatically matched to a suitable network node unit for prediction so that the model can cluster the data based on their characteristics and, therefore, be more practical. Experiments were performed on the Bohai Sea and the South China Sea. Firstly, we performed an ablation experiment to verify the effectiveness of the framework of the model. Secondly, we tested the model to predict sea surface temperature, and the results verified the accuracy of CENS. Lastly, there was a meaningful finding that the clustering results of the model in the South China Sea matched the actual characteristics of the continental shelf of the South China Sea, and the cluster had spatial continuity.

Correlation-Guided Ensemble Clustering for Hyperspectral Band Selection

Hyperspectral band selection is a commonly used technique to alleviate the curse of dimensionality. Recently, clustering-based methods have attracted much attention for their effectiveness in selecting informative and representative bands. However, the single clustering algorithm is used in most of the clustering-based methods, and the neglect of the correlation among adjacent bands in their clustering procedure is prone to resulting in the degradation of the representativeness of the selected band set. This may, consequently, adversely impact hyperspectral classification performance. To tackle such issues, in this paper, we propose a correlation-guided ensemble clustering approach for hyperspectral band selection. By exploiting ensemble clustering, more effective clustering results are expected based on multiple band partitions given by base clustering with different parameters. In addition, given that adjacent bands are most probably located in the same cluster, a novel consensus function is designed to construct the final clustering partition by performing an agglomerative clustering. Thus, the performance of our addressed task (band selection) is further improved. The experimental results on three real-world datasets demonstrate that the performance of our proposed method is superior compared with those of state-of-the-art methods.

Learning Static-Adaptive Graphs for RGB-T Image Saliency Detection

Many works have been proposed on image saliency detection to handle challenging issues including low illumination, cluttered background, low contrast, and so on. Although good performance has been achieved by these algorithms, detection results are still poor based on RGB modality. Inspired by the recent progress of multi-modality fusion, we propose a novel RGB-thermal saliency detection algorithm through learning static-adaptive graphs. Specifically, we first extract superpixels from the two modalities and calculate their affinity matrix. Then, we learn the affinity matrix dynamically and construct a static-adaptive graph. Finally, the saliency maps can be obtained by a two-stage ranking algorithm. Our method is evaluated on RGBT-Saliency Dataset with eleven kinds of challenging subsets. Experimental results show that the proposed method has better generalization performance. The complementary benefits of RGB and thermal images and the more robust feature expression of learning static-adaptive graphs create an effective way to improve the detection effectiveness of image saliency in complex scenes.

Multiscale Feature Fusion Network Incorporating 3D Self-Attention for Hyperspectral Image Classification

In recent years, the deep learning-based hyperspectral image (HSI) classification method has achieved great success, and the convolutional neural network (CNN) method has achieved good classification performance in the HSI classification task. However, the convolutional operation only works with local neighborhoods, and is effective in extracting local features. It is difficult to capture interactive features over long distances, which affects the accuracy of classification to some extent. At the same time, the data from HSI have the characteristics of three-dimensionality, redundancy, and noise. To solve these problems, we propose a 3D self-attention multiscale feature fusion network (3DSA-MFN) that integrates 3D multi-head self-attention. 3DSA-MFN first uses different sized convolution kernels to extract multiscale features, samples the different granularities of the feature map, and effectively fuses the spatial and spectral features of the feature map. Then, we propose an improved 3D multi-head self-attention mechanism that provides local feature details for the self-attention branch, and fully exploits the context of the input matrix. To verify the performance of the proposed method, we compare it with six current methods on three public datasets. The experimental results show that the proposed 3DSA-MFN achieves competitive classification and highlights the HSI classification task.

Multilayer Spectral-Spatial Graphs for Label Noisy Robust Hyperspectral Image Classification

In hyperspectral image (HSI) analysis, label information is a scarce resource and it is unavoidably affected by human and nonhuman factors, resulting in a large amount of label noise. Although most of the recent supervised HSI classification methods have achieved good classification results, their performance drastically decreases when the training samples contain label noise. To address this issue, we propose a label noise cleansing method based on spectral-spatial graphs (SSGs). In particular, an affinity graph is constructed based on spectral and spatial similarity, in which pixels in a superpixel segmentation-based homogeneous region are connected, and their similarities are measured by spectral feature vectors. Then, we use the constructed affinity graph to regularize the process of label noise cleansing. In this manner, we transform label noise cleansing to an optimization problem with a graph constraint. To fully utilize spatial information, we further develop multiscale segmentation-based multilayer SSGs (MSSGs). It can efficiently merge the complementary information of multilayer graphs and thus provides richer spatial information compared with any single-layer graph obtained from isolation segmentation. Experimental results show that MSSG reduces the level of label noise. Compared with the state of the art, the proposed MSSG method exhibits significantly enhanced classification accuracy toward the training data with noisy labels. The significant advantages of the proposed method over four major classifiers are also demonstrated. The source code is available at https://github.com/junjun-jiang/MSSG.

A Novel 2D-3D CNN with Spectral-Spatial Multi-Scale Feature Fusion for Hyperspectral Image Classification

Multifarious hyperspectral image (HSI) classification methods based on convolutional neural networks (CNN) have been gradually proposed and achieve a promising classification performance. However, hyperspectral image classification still suffers from various challenges, including abundant redundant information, insufficient spectral-spatial representation, irregular class distribution, and so forth. To address these issues, we propose a novel 2D-3D CNN with spectral-spatial multi-scale feature fusion for hyperspectral image classification, which consists of two feature extraction streams, a feature fusion module as well as a classification scheme. First, we employ two diverse backbone modules for feature representation, that is, the spectral feature and the spatial feature extraction streams. The former utilizes a hierarchical feature extraction module to capture multi-scale spectral features, while the latter extracts multi-stage spatial features by introducing a multi-level fusion structure. With these network units, the category attribute information of HSI can be fully excavated. Then, to output more complete and robust information for classification, a multi-scale spectral-spatial-semantic feature fusion module is presented based on a Decomposition-Reconstruction structure. Last of all, we innovate a classification scheme to lift the classification accuracy. Experimental results on three public datasets demonstrate that the proposed method outperforms the state-of-the-art methods.

EBARec-BS: Effective Band Attention Reconstruction Network for Hyperspectral Imagery Band Selection

Hyperspectral band selection (BS) is an effective means to avoid the Hughes phenomenon and heavy computational burden in hyperspectral image processing. However, most of the existing BS methods fail to fully consider the interaction between spectral bands and cannot comprehensively consider the representativeness and redundancy of the selected band subset. To solve these problems, we propose an unsupervised effective band attention reconstruction framework for band selection (EBARec-BS) in this article. The framework utilizes the EBARec network to learn the representativeness of each band to the original band set and measures the redundancy between the bands by calculating the distance of each unselected band to the selected band subset. Subsequently, by designing an adaptive weight to balance the influence of the representativeness metric and redundancy metric on the band evaluation, a final band scoring function is obtained to select a band subset that well represents the original hyperspectral image and has low redundancy. Experiments on three well-known hyperspectral data sets indicate that compared with the existing BS methods, the proposed EBARec-BS is robust to noise bands and can effectively select the band subset with higher classification accuracy and less redundant information.

Mixed Noise Estimation Model for Optimized Kernel Minimum Noise Fraction Transformation in Hyperspectral Image Dimensionality Reduction

Dimensionality reduction (DR) is of great significance for simplifying and optimizing hyperspectral image (HSI) features. As a widely used DR method, kernel minimum noise fraction (KMNF) transformation preserves the high-order structures of the original data perfectly. However, the conventional KMNF noise estimation (KMNF-NE) uses the local regression residual of neighbourhood pixels, which depends heavily on spatial information. Due to the limited spatial resolution, there are many mixed pixels in HSI, making KMNF-NE unreliable for noise estimation and leading to poor performance in KMNF for classification on HSIs with low spatial resolution. In order to overcome this problem, a mixed noise estimation model (MNEM) is proposed in this paper for optimized KMNF (OP-KMNF). The MNEM adopts the sequential and linear combination of the Gaussian prior denoising model, median filter, and Sobel operator to estimate noise. It retains more details and edge features, making it more suitable for noise estimation in KMNF. Experiments using several HSI datasets with different spatial and spectral resolutions are conducted. The results show that, compared with some other DR methods, the improvement of OP-KMNF in average classification accuracy is up to 4%. To improve the efficiency, the OP-KMNF was implemented on graphics processing units (GPU) and sped up by about 60× compared to the central processing unit (CPU) implementation. The outcome demonstrates the significant performance of OP-KMNF in terms of classification ability and execution efficiency.

Joint and Progressive Subspace Analysis (JPSA) With Spatial-Spectral Manifold Alignment for Semisupervised Hyperspectral Dimensionality Reduction

Conventional nonlinear subspace learning techniques (e.g., manifold learning) usually introduce some drawbacks in explainability (explicit mapping) and cost effectiveness (linearization), generalization capability (out-of-sample), and representability (spatial-spectral discrimination). To overcome these shortcomings, a novel linearized subspace analysis technique with spatial-spectral manifold alignment is developed for a semisupervised hyperspectral dimensionality reduction (HDR), called joint and progressive subspace analysis (JPSA). The JPSA learns a high-level, semantically meaningful, joint spatial-spectral feature representation from hyperspectral (HS) data by: 1) jointly learning latent subspaces and a linear classifier to find an effective projection direction favorable for classification; 2) progressively searching several intermediate states of subspaces to approach an optimal mapping from the original space to a potential more discriminative subspace; and 3) spatially and spectrally aligning a manifold structure in each learned latent subspace in order to preserve the same or similar topological property between the compressed data and the original data. A simple but effective classifier, that is, nearest neighbor (NN), is explored as a potential application for validating the algorithm performance of different HDR approaches. Extensive experiments are conducted to demonstrate the superiority and effectiveness of the proposed JPSA on two widely used HS datasets: 1) Indian Pines (92.98%) and 2) the University of Houston (86.09%) in comparison with previous state-of-the-art HDR methods. The demo of this basic work (i.e., ECCV2018) is openly available at https://github.com/danfenghong/ECCV2018_J-Play.

Model-Guided Deep Hyperspectral Image Super-Resolution

The trade-off between spatial and spectral resolution is one of the fundamental issues in hyperspectral images (HSI). Given the challenges of directly acquiring high-resolution hyperspectral images (HR-HSI), a compromised solution is to fuse a pair of images: one has high-resolution (HR) in the spatial domain but low-resolution (LR) in spectral-domain and the other vice versa. Model-based image fusion methods including pan-sharpening aim at reconstructing HR-HSI by solving manually designed objective functions. However, such hand-crafted prior often leads to inevitable performance degradation due to a lack of end-to-end optimization. Although several deep learning-based methods have been proposed for hyperspectral pan-sharpening, HR-HSI related domain knowledge has not been fully exploited, leaving room for further improvement. In this paper, we propose an iterative Hyperspectral Image Super-Resolution (HSISR) algorithm based on a deep HSI denoiser to leverage both domain knowledge likelihood and deep image prior. By taking the observation matrix of HSI into account during the end-to-end optimization, we show how to unfold an iterative HSISR algorithm into a novel model-guided deep convolutional network (MoG-DCN). The representation of the observation matrix by subnetworks also allows the unfolded deep HSISR network to work with different HSI situations, which enhances the flexibility of MoG-DCN. Extensive experimental results are reported to demonstrate that the proposed MoG-DCN outperforms several leading HSISR methods in terms of both implementation cost and visual quality. The code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/MoG-DCN.htm.

Deep Blind Hyperspectral Image Super-Resolution

The production of a high spatial resolution (HR) hyperspectral image (HSI) through the fusion of a low spatial resolution (LR) HSI with an HR multispectral image (MSI) has underpinned much of the recent progress in HSI super-resolution. The premise of these signs of progress is that both the degeneration from the HR HSI to LR HSI in the spatial domain and the degeneration from the HR HSI to HR MSI in the spectral domain are assumed to be known in advance. However, such a premise is difficult to achieve in practice. To address this problem, we propose to incorporate degeneration estimation into HSI super-resolution and present an unsupervised deep framework for "blind" HSIs super-resolution where the degenerations in both domains are unknown. In this framework, we model the latent HR HSI and the unknown degenerations with deep network structures to regularize them instead of using handcrafted (or shallow) priors. Specifically, we generate the latent HR HSI with an image-specific generator network and structure the degenerations in spatial and spectral domains through a convolution layer and a fully connected layer, respectively. By doing this, the proposed framework can be formulated as an end-to-end deep network learning problem, which is purely supervised by those two input images (i.e., LR HSI and HR MSI) and can be effectively solved by the backpropagation algorithm. Experiments on both natural scene and remote sensing HSI data sets show the superior performance of the proposed method in coping with unknown degeneration either in the spatial domain, spectral domain, or even both of them.

Deep Multiscale Detail Networks for Multiband Spectral Image Sharpening

We introduce a new deep detail network architecture with grouped multiscale dilated convolutions to sharpen images contain multiband spectral information. Specifically, our end-to-end network directly fuses low-resolution multispectral and panchromatic inputs to produce high-resolution multispectral results, which is the same goal of the pansharpening in remote sensing. The proposed network architecture is designed by utilizing our domain knowledge and considering the two aims of the pansharpening: spectral and spatial preservations. For spectral preservation, the up-sampled multispectral images are directly added to the output for lossless spectral information propagation. For spatial preservation, we train the proposed network in the high-frequency domain instead of the commonly used image domain. Different from conventional network structures, we remove pooling and batch normalization layers to preserve spatial information and improve generalization to new satellites, respectively. To effectively and efficiently obtain multiscale contextual features at a fine-grained level, we propose a grouped multiscale dilated network structure to enlarge the receptive fields for each network layer. This structure allows the network to capture multiscale representations without increasing the parameter burden and network complexity. These representations are finally utilized to reconstruct the residual images which contain spatial details of PAN. Our trained network is able to generalize different satellite images without the need for parameter tuning. Moreover, our model is a general framework, which can be directly used for other kinds of multiband spectral image sharpening, e.g., hyperspectral image sharpening. Experiments show that our model performs favorably against compared methods in terms of both qualitative and quantitative qualities.

Implementation of IoT Framework with Data Analysis Using Deep Learning Methods for Occupancy Prediction in a Building

Many countries worldwide face challenges in controlling building incidence prevention measures for fire disasters. The most critical issues are the localization, identification, detection of the room occupant. Internet of Things (IoT) along with machine learning proved the increase of the smartness of the building by providing real-time data acquisition using sensors and actuators for prediction mechanisms. This paper proposes the implementation of an IoT framework to capture indoor environmental parameters for occupancy multivariate time-series data. The application of the Long Short Term Memory (LSTM) Deep Learning algorithm is used to infer the knowledge of the presence of human beings. An experiment is conducted in an office room using multivariate time-series as predictors in the regression forecasting problem. The results obtained demonstrate that with the developed system it is possible to obtain, process, and store environmental information. The information collected was applied to the LSTM algorithm and compared with other machine learning algorithms. The compared algorithms are Support Vector Machine, Naïve Bayes Network, and Multilayer Perceptron Feed-Forward Network. The outcomes based on the parametric calibrations demonstrate that LSTM performs better in the context of the proposed application.

Regularizing Hyperspectral and Multispectral Image Fusion by CNN Denoiser

Hyperspectral image (HSI) and multispectral image (MSI) fusion, which fuses a low-spatial-resolution HSI (LR-HSI) with a higher resolution multispectral image (MSI), has become a common scheme to obtain high-resolution HSI (HR-HSI). This article presents a novel HSI and MSI fusion method (called as CNN-Fus), which is based on the subspace representation and convolutional neural network (CNN) denoiser, i.e., a well-trained CNN for gray image denoising. Our method only needs to train the CNN on the more accessible gray images and can be directly used for any HSI and MSI data sets without retraining. First, to exploit the high correlations among the spectral bands, we approximate the desired HR-HSI with the low-dimensional subspace multiplied by the coefficients, which can not only speed up the algorithm but also lead to more accurate recovery. Since the spectral information mainly exists in the LR-HSI, we learn the subspace from it via singular value decomposition. Due to the powerful learning performance and high speed of CNN, we use the well-trained CNN for gray image denoising to regularize the estimation of coefficients. Specifically, we plug the CNN denoiser into the alternating direction method of multipliers (ADMM) algorithm to estimate the coefficients. Experiments demonstrate that our method has superior performance over the state-of-the-art fusion methods.

Random Collective Representation-Based Detector with Multiple Features for Hyperspectral Images

Collaborative representation-based detector (CRD), as the most representative anomaly detection method, has been widely applied in the field of hyperspectral anomaly detection (HAD). However, the sliding dual window of the original CRD introduces high computational complexity. Moreover, most HAD models only consider a single spectral or spatial feature of the hyperspectral image (HSI), which is unhelpful for improving detection accuracy. To solve these problems, in terms of speed and accuracy, we propose a novel anomaly detection approach, named Random Collective Representation-based Detector with Multiple Feature (RCRDMF). This method includes the following steps. This method first extract the different features include spectral feature, Gabor feature, extended multiattribute profile (EMAP) feature, and extended morphological profile (EMP) feature matrix from the HSI image, which enables us to improve the accuracy of HAD by combining the multiple spectral and spatial features. The ensemble and random collaborative representation detector (ERCRD) method is then applied, which can improve the anomaly detection speed. Finally, an adaptive weight approach is proposed to calculate the weight for each feature. Experimental results on six hyperspectral datasets demonstrate that the proposed approach has the superiority over accuracy and speed.

3-D Quasi-Recurrent Neural Network for Hyperspectral Image Denoising

In this article, we propose an alternating directional 3-D quasi-recurrent neural network for hyperspectral image (HSI) denoising, which can effectively embed the domain knowledge-structural spatiospectral correlation and global correlation along spectrum (GCS). Specifically, 3-D convolution is utilized to extract structural spatiospectral correlation in an HSI, while a quasi-recurrent pooling function is employed to capture the GCS. Moreover, the alternating directional structure is introduced to eliminate the causal dependence with no additional computation cost. The proposed model is capable of modeling spatiospectral dependence while preserving the flexibility toward HSIs with an arbitrary number of bands. Extensive experiments on HSI denoising demonstrate significant improvement over the state-of-the-art under various noise settings, in terms of both restoration accuracy and computation time. Our code is available at https://github.com/Vandermode/QRNN3D.

Statistical Loss and Analysis for Deep Learning in Hyperspectral Image Classification

Nowadays, deep learning methods, especially the convolutional neural networks (CNNs), have shown impressive performance on extracting abstract and high-level features from the hyperspectral image. However, the general training process of CNNs mainly considers the pixelwise information or the samples' correlation to formulate the penalization while ignores the statistical properties especially the spectral variability of each class in the hyperspectral image. These sample-based penalizations would lead to the uncertainty of the training process due to the imbalanced and limited number of training samples. To overcome this problem, this article characterizes each class from the hyperspectral image as a statistical distribution and further develops a novel statistical loss with the distributions, not directly with samples for deep learning. Based on the Fisher discrimination criterion, the loss penalizes the sample variance of each class distribution to decrease the intraclass variance of the training samples. Moreover, an additional diversity-promoting condition is added to enlarge the interclass variance between different class distributions, and this could better discriminate samples from different classes in the hyperspectral image. Finally, the statistical estimation form of the statistical loss is developed with the training samples through multivariant statistical analysis. Experiments over the real-world hyperspectral images show the effectiveness of the developed statistical loss for deep learning.

Hyperspectral Images Super-Resolution via Learning High-Order Coupled Tensor Ring Representation

Hyperspectral image (HSI) super-resolution is a hot topic in remote sensing and computer vision. Recently, tensor analysis has been proven to be an efficient technology for HSI image processing. However, the existing tensor-based methods of HSI super-resolution are not able to capture the high-order correlations in HSI. In this article, we propose to learn a high-order coupled tensor ring (TR) representation for HSI super-resolution. The proposed method first tensorizes the HSI to be estimated into a high-order tensor in which multiscale spatial structures and the original spectral structure are represented. Then, a coupled TR representation model is proposed to fuse the low-resolution HSI (LR-HSI) and high-resolution multispectral image (HR-MSI). In the proposed model, some latent core tensors in TR of the LR-HSI and the HR-MSI are shared, and we use the relationship between the spectral core tensors to reconstruct the HSI. In addition, the graph-Laplacian regularization is introduced to the spectral core tensors to preserve the spectral information. To enhance the robustness of the proposed model, Frobenius norm regularizations are introduced to the other core tensors. Experimental results on both synthetic and real data sets show that the proposed method achieves the state-of-the-art super-resolution performance.

A Pseudo-Label Guided Artificial Bee Colony Algorithm for Hyperspectral Band Selection

Hyperspectral remote sensing images have characteristics such as high dimensionality and high redundancy. This paper proposes a pseudo-label guided artificial bee colony band selection algorithm with hypergraph clustering (HC-ABC) to remove redundant and noise bands. Firstly, replacing traditional pixel points by super-pixel centers, a hypergraph evolutionary clustering method with low computational cost is developed to generate high-quality pseudo-labels; Then, on the basis of these pseudo-labels, taking classification accuracy as the optimized objective, a supervised band selection algorithm based on artificial bee colony is proposed. Moreover, a noise filtering mechanism based on grid division is designed to ensure the accuracy of pseudo-labels. Finally, the proposed algorithm is applied in 3 real datasets and compared with 6 classical band selection algorithms. Experimental results show that the proposed algorithm can obtain a band subset with high classification accuracy for all the three classifiers, KNN, Random Forest, and SVM.

Band Ranking via Extended Coefficient of Variation for Hyperspectral Band Selection

Hundreds of narrow bands over a continuous spectral range make hyperspectral imagery rich in information about objects, while at the same time causing the neighboring bands to be highly correlated. Band selection is a technique that provides clear physical-meaning results for hyperspectral dimensional reduction, alleviating the difficulty for transferring and processing hyperspectral images caused by a property of hyperspectral images: large data volumes. In this study, a simple and efficient band ranking via extended coefficient of variation (BRECV) is proposed for unsupervised hyperspectral band selection. The naive idea of the BRECV algorithm is to select bands with relatively smaller means and lager standard deviations compared to their adjacent bands. To make this simple idea into an algorithm, and inspired by coefficient of variation (CV), we constructed an extended CV matrix for every three adjacent bands to study the changes of means and standard deviations, and accordingly propose a criterion to allocate values to each band for ranking. A derived unsupervised band selection based on the same idea while using entropy is also presented. Though the underlying idea is quite simple, and both cluster and optimization methods are not used, the BRECV method acquires qualitatively the same level of classification accuracy, compared with some state-of-the-art band selection methods

Recent Advances of Hyperspectral Imaging Technology and Applications in Agriculture

Remote sensing is a useful tool for monitoring spatio-temporal variations of crop morphological and physiological status and supporting practices in precision farming. In comparison with multispectral imaging, hyperspectral imaging is a more advanced technique that is capable of acquiring a detailed spectral response of target features. Due to limited accessibility outside of the scientific community, hyperspectral images have not been widely used in precision agriculture. In recent years, different mini-sized and low-cost airborne hyperspectral sensors (e.g., Headwall Micro-Hyperspec, Cubert UHD 185-Firefly) have been developed, and advanced spaceborne hyperspectral sensors have also been or will be launched (e.g., PRISMA, DESIS, EnMAP, HyspIRI). Hyperspectral imaging is becoming more widely available to agricultural applications. Meanwhile, the acquisition, processing, and analysis of hyperspectral imagery still remain a challenging research topic (e.g., large data volume, high data dimensionality, and complex information analysis). It is hence beneficial to conduct a thorough and in-depth review of the hyperspectral imaging technology (e.g., different platforms and sensors), methods available for processing and analyzing hyperspectral information, and recent advances of hyperspectral imaging in agricultural applications. Publications over the past 30 years in hyperspectral imaging technology and applications in agriculture were thus reviewed. The imaging platforms and sensors, together with analytic methods used in the literature, were discussed. Performances of hyperspectral imaging for different applications (e.g., crop biophysical and biochemical properties’ mapping, soil characteristics, and crop classification) were also evaluated. This review is intended to assist agricultural researchers and practitioners to better understand the strengths and limitations of hyperspectral imaging to agricultural applications and promote the adoption of this valuable technology. Recommendations for future hyperspectral imaging research for precision agriculture are also presented.

Recent Advances in Saliency Estimation for Omnidirectional Images, Image Groups, and Video Sequences

We present a review of methods for automatic estimation of visual saliency: the perceptual property that makes specific elements in a scene stand out and grab the attention of the viewer. We focus on domains that are especially recent and relevant, as they make saliency estimation particularly useful and/or effective: omnidirectional images, image groups for co-saliency, and video sequences. For each domain, we perform a selection of recent methods, we highlight their commonalities and differences, and describe their unique approaches. We also report and analyze the datasets involved in the development of such methods, in order to reveal additional peculiarities of each domain, such as the representation used for the ground truth saliency information (scanpaths, saliency maps, or salient object regions). We define domain-specific evaluation measures, and provide quantitative comparisons on the basis of common datasets and evaluation criteria, highlighting the different impact of existing approaches on each domain. We conclude by synthesizing the emerging directions for research in the specialized literature, which include novel representations for omnidirectional images, inter- and intra- image saliency decomposition for co-saliency, and saliency shift for video saliency estimation.

Residual Group Channel and Space Attention Network for Hyperspectral Image Classification

Recently, deep learning methods based on three-dimensional (3-D) convolution have been widely used in the hyperspectral image (HSI) classification tasks and shown good classification performance. However, affected by the irregular distribution of various classes in HSI datasets, most previous 3-D convolutional neural network (CNN)-based models require more training samples to obtain better classification accuracies. In addition, as the network deepens, which leads to the spatial resolution of feature maps gradually decreasing, much useful information may be lost during the training process. Therefore, how to ensure efficient network training is key to the HSI classification tasks. To address the issue mentioned above, in this paper, we proposed a 3-DCNN-based residual group channel and space attention network (RGCSA) for HSI classification. Firstly, the proposed bottom-up top-down attention structure with the residual connection can improve network training efficiency by optimizing channel-wise and spatial-wise features throughout the whole training process. Secondly, the proposed residual group channel-wise attention module can reduce the possibility of losing useful information, and the novel spatial-wise attention module can extract context information to strengthen the spatial features. Furthermore, our proposed RGCSA network only needs few training samples to achieve higher classification accuracies than previous 3-D-CNN-based networks. The experimental results on three commonly used HSI datasets demonstrate the superiority of our proposed network based on the attention mechanism and the effectiveness of the proposed channel-wise and spatial-wise attention modules for HSI classification. The code and configurations are released at Github.com.

Synergistic 2D/3D Convolutional Neural Network for Hyperspectral Image Classification

Accurate hyperspectral image classification has been an important yet challenging task for years. With the recent success of deep learning in various tasks, 2-dimensional (2D)/3-dimensional (3D) convolutional neural networks (CNNs) have been exploited to capture spectral or spatial information in hyperspectral images. On the other hand, few approaches make use of both spectral and spatial information simultaneously, which is critical to accurate hyperspectral image classification. This paper presents a novel Synergistic Convolutional Neural Network (SyCNN) for accurate hyperspectral image classification. The SyCNN consists of a hybrid module that combines 2D and 3D CNNs in feature learning and a data interaction module that fuses spectral and spatial hyperspectral information. Additionally, it introduces a 3D attention mechanism before the fully-connected layer which helps filter out interfering features and information effectively. Extensive experiments over three public benchmarking datasets show that our proposed SyCNNs clearly outperform state-of-the-art techniques that use 2D/3D CNNs.

Hyperspectral remote sensing image classification based on random average band selection and an ensemble kernel extreme learning machine

Hyperspectral remote sensing technology can explore a lot of information about ground objects, and the information is not explored in multispectral technology. This study proposes a hyperspectral remote sensing image classification method. First, we preprocess the hyperspectral data to obtain the average spectral information of the pixels; the average spectral information contains spectral-spatial features. Second, the average spectral information is randomly band selected to obtain multiple different datasets. Third, based on ensemble learning and a kernel extreme learning machine, an ensemble kernel extreme learning machine is proposed. Finally, a hyperspectral remote sensing image classification model based on the ensemble kernel extreme learning machine is established. Experiments with two common hyperspectral remote sensing image datasets demonstrate the effectiveness of the proposed method.

Hyperspectral Pansharpening With Deep Priors

Hyperspectral (HS) image can describe subtle differences in the spectral signatures of materials, but it has low spatial resolution limited by the existing technical and budget constraints. In this paper, we propose a promising HS pansharpening method with deep priors (HPDP) to fuse a low-resolution (LR) HS image with a high-resolution (HR) panchromatic (PAN) image. Different from the existing methods, we redefine the spectral response function (SRF) based on the larger eigenvalue of structure tensor (ST) matrix for the first time that is more in line with the characteristics of HS imaging. Then, we introduce HFNet to capture deep residual mapping of high frequency across the upsampled HS image and the PAN image in a band-by-band manner. Specifically, the learned residual mapping of high frequency is injected into the structural transformed HS images, which are the extracted deep priors served as additional constraint in a Sylvester equation to estimate the final HR HS image. Comparative analyses validate that the proposed HPDP method presents the superior pansharpening performance by ensuring higher quality both in spatial and spectral domains for all types of data sets. In addition, the HFNet is trained in the high-frequency domain based on multispectral (MS) images, which overcomes the sensitivity of deep neural network (DNN) to data sets acquired by different sensors and the difficulty of insufficient training samples for HS pansharpening.

Cascade Superpixel Regularized Gabor Feature Fusion for Hyperspectral Image Classification

A 3-D Gabor wavelet provides an effective way to obtain the spectral-spatial-fused features for hyperspectral image, which has shown advantageous performance for material classification and recognition. In this paper, instead of separately employing the Gabor magnitude and phase features, which, respectively, reflect the intensity and variation of surface materials in local area, a cascade superpixel regularized Gabor feature fusion (CSRGFF) approach has been proposed. First, the Gabor filters with particular orientation are utilized to obtain Gabor features (including magnitude and phase) from the original hyperspectral image. Second, a support vector machine (SVM)-based probability representation strategy is developed to fully exploit the decision information in SVM output, and the achieved confidence score can make the following fusion with Gabor phase more effective. Meanwhile, the quadrant bit coding and Hamming distance metric are applied to encode the Gabor phase features and measure sample similarity in sequence. Third, the carefully defined characteristics of two kinds of features are directly combined together without any weighting operation to describe the weight of samples belonging to each class. Finally, a series of superpixel graphs extracted from the raw hyperspectral image with different numbers of superpixels are employed to successively regularize the weighting cube from over-segmentation to under-segmentation, and the classification performance gradually improves with the decrease in the number of superpixels in the regularization procedure. Four widely used real hyperspectral images have been conducted, and the experimental results constantly demonstrate the superiority of our CSRGFF approach over several state-of-the-art methods.