Detecting Coherent Groups in Crowd Scenes by Multiview Clustering

Deep Spectral Representation Learning From Multi-View Data

Multi-view representation learning (MvRL) aims to learn a consensus representation from diverse sources or domains to facilitate downstream tasks such as clustering, retrieval, and classification. Due to the limited representative capacity of the adopted shallow models, most existing MvRL methods may yield unsatisfactory results, especially when the labels of data are unavailable. To enjoy the representative capacity of deep learning, this paper proposes a novel multi-view unsupervised representation learning method, termed as Multi-view Laplacian Network (MvLNet), which could be the first deep version of the multi-view spectral representation learning method. Note that, such an attempt is nontrivial because simply combining Laplacian embedding (i.e., spectral representation) with neural networks will lead to trivial solutions. To solve this problem, MvLNet enforces an orthogonal constraint and reformulates it as a layer with the help of Cholesky decomposition. The orthogonal layer is stacked on the embedding network so that a common space could be learned for consensus representation. Compared with numerous recent-proposed approaches, extensive experiments on seven challenging datasets demonstrate the effectiveness of our method in three multi-view tasks including clustering, recognition, and retrieval. The source code could be found at www.pengxi.me.

Consensus guided incomplete multi-view spectral clustering

Incomplete multi-view clustering which aims to solve the difficult clustering challenge on incomplete multi-view data collected from diverse domains with missing views has drawn considerable attention in recent years. In this paper, we propose a novel method, called consensus guided incomplete multi-view spectral clustering (CGIMVSC), to address the incomplete clustering problem. Specifically, CGIMVSC seeks to explore the local information within every single-view and the semantic consistent information shared by all views in a unified framework simultaneously, where the local structure is adaptively obtained from the incomplete data rather than pre-constructed via a k-nearest neighbor approach in the existing methods. Considering the semantic consistency of multiple views, CGIMVSC introduces a co-regularization constraint to minimize the disagreement between the common representation and the individual representations with respect to different views, such that all views will obtain a consensus clustering result. Experimental comparisons with some state-of-the-art methods on seven datasets validate the effectiveness of the proposed method on incomplete multi-view clustering.

A Multitask Cascading CNN with MultiScale Infrared Optical Flow Feature Fusion-Based Abnormal Crowd Behavior Monitoring UAV

Visual-based object detection and understanding is an important problem in computer vision and signal processing. Due to their advantages of high mobility and easy deployment, unmanned aerial vehicles (UAV) have become a flexible monitoring platform in recent years. However, visible-light-based methods are often greatly influenced by the environment. As a result, a single type of feature derived from aerial monitoring videos is often insufficient to characterize variations among different abnormal crowd behaviors. To address this, we propose combining two types of features to better represent behavior, namely, multitask cascading CNN (MC-CNN) and multiscale infrared optical flow (MIR-OF), capturing both crowd density and average speed and the appearances of the crowd behaviors, respectively. First, an infrared (IR) camera and Nvidia Jetson TX1 were chosen as an infrared vision system. Since there are no published infrared-based aerial abnormal-behavior datasets, we provide a new infrared aerial dataset named the IR-flying dataset, which includes sample pictures and videos in different scenes of public areas. Second, MC-CNN was used to estimate the crowd density. Third, MIR-OF was designed to characterize the average speed of crowd. Finally, considering two typical abnormal crowd behaviors of crowd aggregating and crowd escaping, the experimental results show that the monitoring UAV system can detect abnormal crowd behaviors in public areas effectively.

Submanifold-Preserving Discriminant Analysis With an Auto-Optimized Graph

Due to the multimodality of non-Gaussian data, traditional globality-preserved dimensionality reduction (DR) methods, such as linear discriminant analysis (LDA) and principal component analysis (PCA) are difficult to deal with. In this paper, we present a novel local DR framework via auto-optimized graph embedding to extract the intrinsic submanifold structure of multimodal data. Specifically, the proposed model seeks to learn an embedding space which can preserve the local neighborhood structure by constructing a k -nearest neighbors ( k NNs) graph on data points. Different than previous works, our model employs the l₀ -norm constraint and binary constraint on the similarity matrix to impose that there only be a k nonzero value in each row of the similarity matrix, which can ensure the k -connectivity in graph. More important, as the high-dimensional data probably contains some noises and redundant features, calculating the similarity matrix in the original space by using a kernel function is inaccurate. As a result, a mechanism of an auto-optimized graph is derived in the proposed model. Concretely, we learn the embedding space and similarity matrix simultaneously. In other words, the selection of neighbors is automatically executed in the optimal subspace rather than in the original space when the algorithm reaches convergence, which can alleviate the affect of noises and improve the robustness of the proposed model. In addition, four supervised and semisupervised local DR methods are derived by the proposed framework which can extract the discriminative features while preserving the submanifold structure of data. Last but not least, since two variables need to be optimized simultaneously in the proposed methods, and the constraints on the similarity matrix are difficult to satisfy, which is an NP-hard problem. Consequently, an efficient iterative optimization algorithm is introduced to solve the proposed problems. Extensive experiments conducted on synthetic data and several real-world datasets have demonstrated the advantages of the proposed methods in robustness and recognition accuracy.

RoSeq: Robust Sequence Labeling

In this paper, we mainly investigate two issues for sequence labeling, namely, label imbalance and noisy data that are commonly seen in the scenario of named entity recognition (NER) and are largely ignored in the existing works. To address these two issues, a new method termed robust sequence labeling (RoSeq) is proposed. Specifically, to handle the label imbalance issue, we first incorporate label statistics in a novel conditional random field (CRF) loss. In addition, we design an additional loss to reduce the weights of overwhelming easy tokens for augmenting the CRF loss. To address the noisy training data, we adopt an adversarial training strategy to improve model generalization. In experiments, the proposed RoSeq achieves the state-of-the-art performances on CoNLL and English Twitter NER-88.07% on CoNLL-2002 Dutch, 87.33% on CoNLL-2002 Spanish, 52.94% on WNUT-2016 Twitter, and 43.03% on WNUT-2017 Twitter without using the additional data.

Micro-cracks detection of solar cells surface via combining short-term and long-term deep features

The machine vision based methods for micro-cracks detection of solar cells surface have become one of the main research directions with its efficiency and convenience. The existed methods are roughly classified into two categories: current viewing information based methods, prior knowledge based methods, however, the former usually adopt hand-designed features with poor generality and lacks the guidance of prior knowledge, the latter are usually implemented through the machine learning, and the generalization ability is also limited since the large-scale annotation dataset is scarce. To resolve above problems, a novel micro-cracks detection method via combining short-term and long-term deep features is proposed in this paper. The short-term deep features which represent the current viewing information are learned from the input image itself through stacked denoising auto encoder (SDAE), the long-term deep features which represent the prior knowledge are learned from a large number of natural scene images that people often see through convolutional neural networks (CNNs). The subjective and objective evaluations demonstrate that: 1) the performance of combining the short-term and long-term deep features is better than any of them alone, 2) the performance of proposed method is superior to the shallow learning based methods, 3) the proposed method can effectively detect various kinds of micro-cracks.

Quantifying and Detecting Collective Motion in Crowd Scenes

People in crowd scenes always exhibit consistent behaviors and form collective motions. The analysis of collective motion has motivated a surge of interest in computer vision. Nevertheless, the effort is hampered by the complex nature of collective motions. Considering the fact that collective motions are formed by individuals, this paper proposes a new framework for both quantifying and detecting collective motion by investigating the spatio-temporal behavior of individuals. The main contributions of this work are threefold: 1) an intention-aware model is built to fully capture the intrinsic dynamics of individuals; 2) a structure-based collectiveness measurement is developed to accurately quantify the collective properties of crowds; 3) a multistage clustering strategy is formulated to detect both the local and global behavior consistency in crowd scenes. Experiments on real world data sets show that our method is able to handle crowds with various structures and time-varying dynamics. Especially, the proposed method shows nearly 10% improvement over the competitors in terms of NMI, Purity and RI. Its applicability is illustrated in the context of anomaly detection and semantic scene segmentation.

Multiview Consensus Graph Learning for lncRNA-Disease Association Prediction

Long noncoding RNAs (lncRNAs) are a class of noncoding RNA molecules longer than 200 nucleotides. Recent studies have uncovered their functional roles in diverse cellular processes and tumorigenesis. Therefore, identifying novel disease-related lncRNAs might deepen our understanding of disease etiology. However, due to the relatively small number of verified associations between lncRNAs and diseases, it remains a challenging task to reliably and effectively predict the associated lncRNAs for given diseases. In this paper, we propose a novel multiview consensus graph learning method to infer potential disease-related lncRNAs. Specifically, we first construct a set of similarity matrices for lncRNAs and diseases by taking advantage of the known associations. We then iteratively learn a consensus graph from the multiple input matrices and simultaneously optimize the predicted association probability based on a multi-label learning framework. To convey the utility of our method, three state-of-the-art methods are compared with our method on three widely used datasets. The experiment results illustrate that our method could obtain the best prediction performance under different cross validation schemes. The case study analysis implemented for uterine cervical neoplasms further confirmed the utility of our method in identifying lncRNAs as potential prognostic biomarkers in practice.

Hyperspectral Anomaly Detection Based on Separability-Aware Sample Cascade

A hyperspectral image usually covers a large scale of ground scene, which contains various materials with different spectral properties. When directly exploring the background information using all the image pixels, complex spectral interactions and inter-/intra-difference of different samples will significantly reduce the accuracy of background evaluation and further affect the detection performance. To address this problem, this paper proposes a novel hyperspectral anomaly detection method based on separability-aware sample cascade model. Through identifying separability of hyperspectral pixels, background samples are sifted out layer-by-layer according to their separable degrees from anomalies, which can ensure the accuracy and distinctiveness of background representation. First, as spatial structure is beneficial for recognizing target, a new spectral–spatial feature extraction technique is used in this work based on the PCA technique and edge-preserving filtering. Second, depending on different separability computed by sparse representation, samples are separated into different sets which can effectively and completely reflect various characteristics of background across all the cascade layers. Meanwhile, some potential abnormal targets are removed at each selection step to avoid their effects on subsequent layers. Finally, comprehensively taking different good properties of all the separability-aware layers into consideration, a simple multilayer anomaly detection strategy is adopted to obtain the final detection map. Extensive experimental results on five real-world hyperspectral images demonstrate our method’s superior performance. Compared with seven representative anomaly detection methods, our method improves the average detection accuracy with great advantages.

Large-Scale Remote Sensing Image Retrieval Based on Semi-Supervised Adversarial Hashing

Remote sensing image retrieval (RSIR), a superior content organization technique, plays an important role in the remote sensing (RS) community. With the number of RS images increases explosively, not only the retrieval precision but also the retrieval efficiency is emphasized in the large-scale RSIR scenario. Therefore, the approximate nearest neighborhood (ANN) search attracts the researchers’ attention increasingly. In this paper, we propose a new hash learning method, named semi-supervised deep adversarial hashing (SDAH), to accomplish the ANN for the large-scale RSIR task. The assumption of our model is that the RS images have been represented by the proper visual features. First, a residual auto-encoder (RAE) is developed to generate the class variable and hash code. Second, two multi-layer networks are constructed to regularize the obtained latent vectors using the prior distribution. These two modules mentioned are integrated under the generator adversarial framework. Through the minimax learning, the class variable would be a one-hot-like vector while the hash code would be the binary-like vector. Finally, a specific hashing function is formulated to enhance the quality of the generated hash code. The effectiveness of the hash codes learned by our SDAH model was proved by the positive experimental results counted on three public RS image archives. Compared with the existing hash learning methods, the proposed method reaches improved performance.

VSSA-NET: Vertical Spatial Sequence Attention Network for Traffic Sign Detection

Although traffic sign detection has been studied for years and great progress has been made with the rise of deep learning technique, there are still many problems remaining to be addressed. For complicated real-world traffic scenes, there are two main challenges. First, traffic signs are usually small-sized objects, which makes them more difficult to detect than large ones; second, it is hard to distinguish false targets which resemble real traffic signs in complex street scenes without context information. To handle these problems, we propose a novel end-to-end deep learning method for traffic sign detection in complex environments. Our contributions are as follows: 1) we propose a multi-resolution feature fusion network architecture which exploits densely connected deconvolution layers with skip connections, and can learn more effective features for a small-size object and 2) we frame the traffic sign detection as a spatial sequence classification and regression task, and propose a vertical spatial sequence attention module to gain more context information for better detection performance. To comprehensively evaluate the proposed method, we experiment on several traffic sign datasets as well as the general object detection dataset, and the results have shown the effectiveness of our proposed method.

Hyperspectral and LiDAR Data Fusion Classification Using Superpixel Segmentation-Based Local Pixel Neighborhood Preserving Embedding

A new method of superpixel segmentation-based local pixel neighborhood preserving embedding (SSLPNPE) is proposed for the fusion of hyperspectral and light detection and ranging (LiDAR) data based on the extinction profiles (EPs), superpixel segmentation and local pixel neighborhood preserving embedding (LPNPE). A new workflow is proposed to calibrate the Goddard’s LiDAR, hyperspectral and thermal (G-LiHT) data, which allows our method to be applied to actual data. Specifically, EP features are extracted from both sources. Then, the derived features of each source are fused by the SSLPNPE. Using the labeled samples, the final label assignment is produced by a classifier. For the open standard experimental data and the actual data, experimental results prove that the proposed method is fast and effective in hyperspectral and LiDAR data fusion.

Fast Spectral Clustering for Unsupervised Hyperspectral Image Classification

Hyperspectral image classification is a challenging and significant domain in the field of remote sensing with numerous applications in agriculture, environmental science, mineralogy, and surveillance. In the past years, a growing number of advanced hyperspectral remote sensing image classification techniques based on manifold learning, sparse representation and deep learning have been proposed and reported a good performance in accuracy and efficiency on state-of-the-art public datasets. However, most existing methods still face challenges in dealing with large-scale hyperspectral image datasets due to their high computational complexity. In this work, we propose an improved spectral clustering method for large-scale hyperspectral image classification without any prior information. The proposed algorithm introduces two efficient approximation techniques based on Nyström extension and anchor-based graph to construct the affinity matrix. We also propose an effective solution to solve the eigenvalue decomposition problem by multiplicative update optimization. Experiments on both the synthetic datasets and the hyperspectral image datasets were conducted to demonstrate the efficiency and effectiveness of the proposed algorithm.

An Efficient Clustering Method for Hyperspectral Optimal Band Selection via Shared Nearest Neighbor

A hyperspectral image (HSI) has many bands, which leads to high correlation between adjacent bands, so it is necessary to find representative subsets before further analysis. To address this issue, band selection is considered as an effective approach that removes redundant bands for HSI. Recently, many band selection methods have been proposed, but the majority of them have extremely poor accuracy in a small number of bands and require multiple iterations, which does not meet the purpose of band selection. Therefore, we propose an efficient clustering method based on shared nearest neighbor (SNNC) for hyperspectral optimal band selection, claiming the following contributions: (1) the local density of each band is obtained by shared nearest neighbor, which can more accurately reflect the local distribution characteristics; (2) in order to acquire a band subset containing a large amount of information, the information entropy is taken as one of the weight factors; (3) a method for automatically selecting the optimal band subset is designed by the slope change. The experimental results reveal that compared with other methods, the proposed method has competitive computational time and the selected bands achieve higher overall classification accuracy on different data sets, especially when the number of bands is small.

Object Detection in Remote Sensing Images Based on a Scene-Contextual Feature Pyramid Network

Object detection has attracted increasing attention in the field of remote sensing image analysis. Complex backgrounds, vertical views, and variations in target kind and size in remote sensing images make object detection a challenging task. In this work, considering that the types of objects are often closely related to the scene in which they are located, we propose a convolutional neural network (CNN) by combining scene-contextual information for object detection. Specifically, we put forward the scene-contextual feature pyramid network (SCFPN), which aims to strengthen the relationship between the target and the scene and solve problems resulting from variations in target size. Additionally, to improve the capability of feature extraction, the network is constructed by repeating a building aggregated residual block. This block increases the receptive field, which can extract richer information for targets and achieve excellent performance with respect to small object detection. Moreover, to improve the proposed model performance, we use group normalization, which divides the channels into groups and computes the mean and variance for normalization within each group, to solve the limitation of the batch normalization. The proposed method is validated on a public and challenging dataset. The experimental results demonstrate that our proposed method outperforms other state-of-the-art object detection models.

Unsupervised Deep Noise Modeling for Hyperspectral Image Change Detection

Hyperspectral image (HSI) change detection plays an important role in remote sensing applications, and considerable research has been done focused on improving change detection performance. However, the high dimension of hyperspectral data makes it hard to extract discriminative features for hyperspectral processing tasks. Though deep convolutional neural networks (CNN) have superior capability in high-level semantic feature learning, it is difficult to employ CNN for change detection tasks. As a ground truth map is usually used for the evaluation of change detection algorithms, it cannot be directly used for supervised learning. In order to better extract discriminative CNN features, a novel noise modeling-based unsupervised fully convolutional network (FCN) framework is presented for HSI change detection in this paper. Specifically, the proposed method utilizes the change detection maps of existing unsupervised change detection methods to train the deep CNN, and then removes the noise during the end-to-end training process. The main contributions of this paper are threefold: (1) A new end-to-end FCN-based deep network architecture for HSI change detection is presented with powerful learning features; (2) An unsupervised noise modeling method is introduced for the robust training of the proposed deep network; (3) Experimental results on three datasets confirm the effectiveness of the proposed method.

Locally Weighted Discriminant Analysis for Hyperspectral Image Classification

A hyperspectral image (HSI) contains a great number of spectral bands for each pixel, which will limit the conventional image classification methods to distinguish land-cover types of each pixel. Dimensionality reduction is an effective way to improve the performance of classification. Linear discriminant analysis (LDA) is a popular dimensionality reduction method for HSI classification, which assumes all the samples obey the same distribution. However, different samples may have different contributions in the computation of scatter matrices. To address the problem of feature redundancy, a new supervised HSI classification method based on locally weighted discriminant analysis (LWDA) is presented. The proposed LWDA method constructs a weighted discriminant scatter matrix model and an optimal projection matrix model for each training sample, which is on the basis of discriminant information and spatial-spectral information. For each test sample, LWDA searches its nearest training sample with spatial information and then uses the corresponding projection matrix to project the test sample and all the training samples into a low-dimensional feature space. LWDA can effectively preserve the spatial-spectral local structures of the original HSI data and improve the discriminating power of the projected data for the final classification. Experimental results on two real-world HSI datasets show the effectiveness of the proposed LWDA method compared with some state-of-the-art algorithms. Especially when the data partition factor is small, i.e., 0.05, the overall accuracy obtained by LWDA increases by about 20 % for Indian Pines and 17 % for Kennedy Space Center (KSC) in comparison with the results obtained when directly using the original high-dimensional data.

SHNN-CAD⁺: An Improvement on SHNN-CAD for Adaptive Online Trajectory Anomaly Detection

To perform anomaly detection for trajectory data, we study the Sequential Hausdorff Nearest-Neighbor Conformal Anomaly Detector (SHNN-CAD) approach, and propose an enhanced version called SHNN-CAD+. SHNN-CAD was introduced based on the theory of conformal prediction dealing with the problem of online detection. Unlike most related approaches requiring several not intuitive parameters, SHNN-CAD has the advantage of being parameter-light which enables the easy reproduction of experiments. We propose to adaptively determine the anomaly threshold during the online detection procedure instead of predefining it without any prior knowledge, which makes the algorithm more usable in practical applications. We present a modified Hausdorff distance measure that takes into account the direction difference and also reduces the computational complexity. In addition, the anomaly detection is more flexible and accurate via a re-do strategy. Extensive experiments on both real-world and synthetic data show that SHNN-CAD+ outperforms SHNN-CAD with regard to accuracy and running time.

Utilizing Multilevel Features for Cloud Detection on Satellite Imagery

Cloud detection, which is defined as the pixel-wise binary classification, is significant in satellite imagery processing. In current remote sensing literature, cloud detection methods are linked to the relationships of imagery bands or based on simple image feature analysis. These methods, which only focus on low-level features, are not robust enough on the images with difficult land covers, for clouds share similar image features such as color and texture with the land covers. To solve the problem, in this paper, we propose a novel deep learning method for cloud detection on satellite imagery by utilizing multilevel image features with two major processes. The first process is to obtain the cloud probability map from the designed deep convolutional neural network, which concatenates deep neural network features from low-level to high-level. The second part of the method is to get refined cloud masks through a composite image filter technique, where the specific filter captures multilevel features of cloud structures and the surroundings of the input imagery. In the experiments, the proposed method achieves 85.38% intersection over union of cloud in the testing set which contains 100 Gaofen-1 wide field of view images and obtains satisfactory visual cloud masks, especially for those hard images. The experimental results show that utilizing multilevel features by the combination of the network with feature concatenation and the particular filter tackles the cloud detection problem with improved cloud masks.