Constructing the L2-Graph for Robust Subspace Learning and Subspace Clustering

Nonparametric Estimation of Probabilistic Membership for Subspace Clustering

Recent advances of subspace clustering have provided a new way of constructing affinity matrices for clustering. Unlike the kernel-based subspace clustering, which needs tedious tuning among infinitely many kernel candidates, the self-expressive models derived from linear subspace assumptions in modern subspace clustering methods are rigorously combined with sparse or low-rank optimization theory to yield an affinity matrix as a solution of an optimization problem. Despite this nice theoretical aspect, the affinity matrices of modern subspace clustering have quite different meanings from the traditional ones, and even though the affinity matrices are expected to have a rough block-diagonal structure, it is unclear whether these are good enough to apply spectral clustering. In fact, most of the subspace clustering methods perform some sort of affinity value rearrangement to apply spectral clustering, but its adequacy for the spectral clustering is not clear; even though the spectral clustering step can also have a critical impact on the overall performance. To resolve this issue, in this paper, we provide a nonparametric method to estimate the probabilistic cluster membership from these affinity matrices, which we can directly find the clusters from. The likelihood for an affinity matrix is defined nonparametrically based on histograms given the probabilistic membership, which is defined as a combination of probability simplices, and an additional prior probability is defined to regularize the membership as a Bernoulli distribution. Solving this maximum a posteriori problem replaces the spectral clustering procedure, and the final discrete cluster membership can be calculated by selecting the clusters with maximum probabilities. The proposed method provides state-of-the-art performance for well-known subspace clustering methods on popular benchmark databases.

Probabilistic Rank-One Discriminant Analysis via Collective and Individual Variation Modeling

Linear discriminant analysis (LDA) is a classical supervised subspace learning technique that has wide applications. However, it is designed for vector only, which cannot exploit the tensor structures and may lead to suboptimal results when dealing with tensorial data. To address this problem, several multilinear LDA (MLDA) methods have been proposed to learn the subspaces from tensors. By exploiting the tensor structures, they achieve compact subspace representations, reduced parameter sizes, and improved robustness against the small sample size problem. However, existing MLDA methods do not take data uncertainty into account, fail to converge properly, or have to introduce additional tuning parameters for good convergence properties. In this paper, we therefore solve these limitations by proposing a probabilistic MLDA method for matrix inputs. Specifically, we propose a new generative model to incorporate structural information into the probabilistic framework, where each observed matrix is represented as a linear combination of collective and individual rank-one matrices. This provides our method with both the expressiveness of capturing discriminative features and nondiscriminative noise, and the capability of exploiting the 2-D tensor structures. To overcome the convergence problem of existing MLDAs, we develop an EM-type algorithm for parameter estimation, which has closed-form solutions with convergence guarantees. Experimental results on real-world datasets show the superiority of the proposed method to other probabilistic and MLDA variants.

l1 -Norm Heteroscedastic Discriminant Analysis Under Mixture of Gaussian Distributions

Fisher's criterion is one of the most popular discriminant criteria for feature extraction. It is defined as the generalized Rayleigh quotient of the between-class scatter distance to the within-class scatter distance. Consequently, Fisher's criterion does not take advantage of the discriminant information in the class covariance differences, and hence, its discriminant ability largely depends on the class mean differences. If the class mean distances are relatively large compared with the within-class scatter distance, Fisher's criterion-based discriminant analysis methods may achieve a good discriminant performance. Otherwise, it may not deliver good results. Moreover, we observe that the between-class distance of Fisher's criterion is based on the l₂ -norm, which would be disadvantageous to separate the classes with smaller class mean distances. To overcome the drawback of Fisher's criterion, in this paper, we first derive a new discriminant criterion, expressed as a mixture of absolute generalized Rayleigh quotients, based on a Bayes error upper bound estimation, where mixture of Gaussians is adopted to approximate the real distribution of data samples. Then, the criterion is further modified by replacing l₂ -norm with l₁ one to better describe the between-class scatter distance, such that it would be more effective to separate the different classes. Moreover, we propose a novel l₁ -norm heteroscedastic discriminant analysis method based on the new discriminant analysis (L1-HDA/GM) for heteroscedastic feature extraction, in which the optimization problem of L1-HDA/GM can be efficiently solved by using the eigenvalue decomposition approach. Finally, we conduct extensive experiments on four real data sets and demonstrate that the proposed method achieves much competitive results compared with the state-of-the-art methods.

Hyperspectral Image Denoising Using Global Weighted Tensor Norm Minimum and Nonlocal Low-Rank Approximation

A hyperspectral image (HSI) contains abundant spatial and spectral information, but it is always corrupted by various noises, especially Gaussian noise. Global correlation (GC) across spectral domain and nonlocal self-similarity (NSS) across spatial domain are two important characteristics for an HSI. To keep the integrity of the global structure and improve the details of the restored HSI, we propose a global and nonlocal weighted tensor norm minimum denoising method which jointly utilizes GC and NSS. The weighted multilinear rank is utilized to depict the GC information. To preserve structural information with NSS, a patch-group-based low-rank-tensor-approximation (LRTA) model is designed. The LRTA makes use of Tucker decompositions of 4D patches, which are composed of a similar 3D patch group of HSI. The alternating direction method of multipliers (ADMM) is adapted to solve the proposed models. Experimental results show that the proposed algorithm can preserve the structural information and outperforms several state-of-the-art denoising methods.

Simultaneous Subspace Clustering and Cluster Number Estimating Based on Triplet Relationship

In this paper, we propose a unified framework to discover the number of clusters and group the data points into different clusters using subspace clustering simultaneously. Real data distributed in a high-dimensional space can be disentangled into a union of low-dimensional subspaces, which can benefit various applications. To explore such intrinsic structure, state-of-the-art subspace clustering approaches often optimize a self-representation problem among all samples, to construct a pairwise affinity graph for spectral clustering. However, a graph with pairwise similarities lacks robustness for segmentation, especially for samples which lie on the intersection of two subspaces. To address this problem, we design a hyper-correlation-based data structure termed as the triplet relationship, which reveals high relevance and local compactness among three samples. The triplet relationship can be derived from the self-representation matrix, and be utilized to iteratively assign the data points to clusters. Based on the triplet relationship, we propose a unified optimizing scheme to automatically calculate clustering assignments. Specifically, we optimize a model selection reward and a fusion reward by simultaneously maximizing the similarity of triplets from different clusters while minimizing the correlation of triplets from the same cluster. The proposed algorithm also automatically reveals the number of clusters and fuses groups to avoid over-segmentation. Extensive experimental results on both synthetic and real-world datasets validate the effectiveness and robustness of the proposed method.

Robust Subspace Clustering by Cauchy Loss Function

Subspace clustering is a problem of exploring the low-dimensional subspaces of high-dimensional data. State-of-the-art approaches are designed by following the model of spectral clustering-based method. These methods pay much attention to learn the representation matrix to construct a suitable similarity matrix and overlook the influence of the noise term on subspace clustering. However, the real data are always contaminated by the noise and the noise usually has a complicated statistical distribution. To alleviate this problem, in this paper, we propose a subspace clustering method based on Cauchy loss function (CLF). Particularly, it uses CLF to penalize the noise term for suppressing the large noise mixed in the real data. This is due to that the CLF's influence function has an upper bound that can alleviate the influence of a single sample, especially the sample with a large noise, on estimating the residuals. Furthermore, we theoretically prove the grouping effect of our proposed method, which means that highly correlated data can be grouped together. Finally, experimental results on five real data sets reveal that our proposed method outperforms several representative clustering methods.

Feature Extraction for Incomplete Data Via Low-Rank Tensor Decomposition With Feature Regularization

Multidimensional data (i.e., tensors) with missing entries are common in practice. Extracting features from incomplete tensors is an important yet challenging problem in many fields such as machine learning, pattern recognition, and computer vision. Although the missing entries can be recovered by tensor completion techniques, these completion methods focus only on missing data estimation instead of effective feature extraction. To the best of our knowledge, the problem of feature extraction from incomplete tensors has yet to be well explored in the literature. In this paper, we therefore tackle this problem within the unsupervised learning environment. Specifically, we incorporate low-rank tensor decomposition with feature variance maximization (TDVM) in a unified framework. Based on orthogonal Tucker and CP decompositions, we design two TDVM methods, TDVM-Tucker and TDVM-CP, to learn low-dimensional features viewing the core tensors of the Tucker model as features and viewing the weight vectors of the CP model as features. TDVM explores the relationship among data samples via maximizing feature variance and simultaneously estimates the missing entries via low-rank Tucker/CP approximation, leading to informative features extracted directly from observed entries. Furthermore, we generalize the proposed methods by formulating a general model that incorporates feature regularization into low-rank tensor approximation. In addition, we develop a joint optimization scheme to solve the proposed methods by integrating the alternating direction method of multipliers with the block coordinate descent method. Finally, we evaluate our methods on six real-world image and video data sets under a newly designed multiblock missing setting. The extracted features are evaluated in face recognition, object/action classification, and face/gait clustering. Experimental results demonstrate the superior performance of the proposed methods compared with the state-of-the-art approaches.

An Efficient Framework for Remote Sensing Parallel Processing: Integrating the Artificial Bee Colony Algorithm and Multiagent Technology

Remote sensing (RS) image processing can be converted to an optimization problem, which can then be solved by swarm intelligence algorithms, such as the artificial bee colony (ABC) algorithm, to improve the accuracy of the results. However, such optimization algorithms often result in a heavy computational burden. To realize the intrinsic parallel computing ability of ABC to address the computational challenges of RS optimization, an improved multiagent (MA)-based ABC framework with a reduced communication cost among agents is proposed by utilizing MA technology. Two types of agents, massive bee agents and one administration agent, located in multiple computing nodes are designed. Based on the communication and cooperation among agents, RS optimization computing is realized in a distributed and concurrent manner. Using hyperspectral RS clustering and endmember extraction as case studies, experimental results indicate that the proposed MA-based ABC approach can effectively improve the computing efficiency while maintaining optimization accuracy.

Fast Kronecker Product Kernel Methods via Generalized Vec Trick

Kronecker product kernel provides the standard approach in the kernel methods' literature for learning from graph data, where edges are labeled and both start and end vertices have their own feature representations. The methods allow generalization to such new edges, whose start and end vertices do not appear in the training data, a setting known as zero-shot or zero-data learning. Such a setting occurs in numerous applications, including drug-target interaction prediction, collaborative filtering, and information retrieval. Efficient training algorithms based on the so-called vec trick that makes use of the special structure of the Kronecker product are known for the case where the training data are a complete bipartite graph. In this paper, we generalize these results to noncomplete training graphs. This allows us to derive a general framework for training Kronecker product kernel methods, as specific examples we implement Kronecker ridge regression and support vector machine algorithms. Experimental results demonstrate that the proposed approach leads to accurate models, while allowing order of magnitude improvements in training and prediction time.

Discriminative Transformation Learning for Fuzzy Sparse Subspace Clustering

This paper develops a novel iterative framework for subspace clustering (SC) in a learned discriminative feature domain. This framework consists of two modules of fuzzy sparse SC and discriminative transformation learning. In the first module, fuzzy latent labels containing discriminative information and latent representations capturing the subspace structure will be simultaneously evaluated in a feature domain. Then the linear transforming operator with respect to the feature domain will be successively updated in the second module with the advantages of more discrimination, subspace structure preservation, and robustness to outliers. These two modules will be alternatively carried out and both theoretical analysis and empirical evaluations will demonstrate its effectiveness and superiorities. In particular, experimental results on three benchmark databases for SC clearly illustrate that the proposed framework can achieve significant improvements than other state-of-the-art approaches in terms of clustering accuracy.

Neighborhood-Based Stopping Criterion for Contrastive Divergence

Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence (CD) learning algorithm, an approximation to the gradient of the data log-likelihood (logL). A simple reconstruction error is often used as a stopping criterion for CD, although several authors have raised doubts concerning the feasibility of this procedure. In many cases, the evolution curve of the reconstruction error is monotonic, while the logL is not, thus indicating that the former is not a good estimator of the optimal stopping point for learning. However, not many alternatives to the reconstruction error have been discussed in the literature. An estimation of the logL of the training data based on annealed importance sampling is feasible but computationally very expensive. In this manuscript, we present a simple and cheap alternative, based on the inclusion of information contained in neighboring states to the training set, as a stopping criterion for CD learning.

Structured AutoEncoders for Subspace Clustering

Existing subspace clustering methods typically employ shallow models to estimate underlying subspaces of unlabeled data points and cluster them into corresponding groups. However, due to the limited representative capacity of the employed shallow models, those methods may fail in handling realistic data without the linear subspace structure. To address this issue, we propose a novel subspace clustering approach by introducing a new deep model-Structured AutoEncoder (StructAE). The StructAE learns a set of explicit transformations to progressively map input data points into nonlinear latent spaces while preserving the local and global subspace structure. In particular, to preserve local structure, the StructAE learns representations for each data point by minimizing reconstruction error w.r.t. itself. To preserve global structure, the StructAE incorporates a prior structured information by encouraging the learned representation to preserve specified reconstruction patterns over the entire data set. To the best of our knowledge, StructAE is one of first deep subspace clustering approaches. Extensive experiments show that the proposed StructAE significantly outperforms 15 state-of-the-art subspace clustering approaches in terms of five evaluation metrics.

Quaternion Locality-Constrained Coding for Color Face Hallucination

Recently, the locality linear coding (LLC) has attracted more and more attentions in the areas of image processing and computer vision. However, the conventional LLC with real setting is just designed for the grayscale image. For the color image, it usually treats each color channel individually or encodes the monochrome image by concatenating all the color channels, which ignores the correlations among different channels. In this paper, we propose a quaternion-based locality-constrained coding (QLC) model for color face hallucination in the quaternion space. In QLC, the face images are represented as quaternion matrices. By transforming the channel images into an orthogonal feature space and encoding the coefficients in the quaternion domain, the proposed QLC is expected to learn the advantages of both quaternion algebra and locality coding scheme. Hence, the QLC cannot only expose the true topology of image patch manifold but also preserve the inherent correlations among different color channels. Experimental results demonstrated that our proposed QLC method achieved superior performance in color face hallucination compared with other state-of-the-art methods.

Angle 2DPCA: A New Formulation for 2DPCA

2-D principal component analysis (2DPCA), which employs squared -norm as the distance metric, has been widely used in dimensionality reduction for data representation and classification. It, however, is commonly known that squared -norm is very sensitivity to outliers. To handle this problem, we present a novel formulation for 2DPCA, namely Angle-2DPCA. It employs -norm as the distance metric and takes into consideration the relationship between reconstruction error and variance in the objective function. We present a fast iterative algorithm to solve the solution of Angle-2DPCA. Experimental results on the Extended Yale B, AR, and PIE face image databases illustrate the effectiveness of our proposed approach.

Robust Face Hallucination via Locality-Constrained Bi-Layer Representation

Recently, locality-constrained linear coding (LLC) has been drawn great attentions and been widely used in image processing and computer vision tasks. However, the conventional LLC model is always fragile to outliers. In this paper, we present a robust locality-constrained bi-layer representation model to simultaneously hallucinate the face images and suppress noise and outliers with the assistant of a group of training samples. The proposed scheme is not only able to capture the nonlinear manifold structure but also robust to outliers by incorporating a weight vector into the objective function to subtly tune the contribution of each pixel offered in the objective. Furthermore, a high-resolution (HR) layer is employed to compensate the missed information in the low-resolution (LR) space for coding. The use of two layers (the LR layer and the HR layer) is expected to expose the complicated correlation between the LR and HR patch spaces, which helps to obtain the desirable coefficients to reconstruct the final HR face. The experimental results demonstrate that the proposed method outperforms the state-of-the-art image super-resolution methods in terms of both quantitative measurements and visual effects.

LRR for Subspace Segmentation via Tractable Schatten-$p$ Norm Minimization and Factorization

Recently, nuclear norm-based low rank representation (LRR) methods have been popular in several applications, such as subspace segmentation. However, there exist two limitations: one is that nuclear norm as the relaxation of rank function will lead to the suboptimal solution since nuclear norm-based minimization subproblem tends to the over-relaxations of singular value elements and treats each of them equally; the other is that solving LRR problems may cause more time consumption due to involving singular value decomposition of the large scale matrix at each iteration. To overcome both disadvantages, this paper mainly considers two tractable variants of LRR: one is Schatten-p norm minimization-based LRR (i.e., SpNM_LRR) and the other is Schatten-p norm factorization-based LRR (i.e., SpNFLRR) for p=1, 2/3 and 1/2. By introducing two or more auxiliary variables in the constraints, the alternating direction method of multiplier (ADMM) with multiple updating variables can be devised to solve these variants of LRR. Furthermore, both computational complexity and convergence property are given to evaluate nonconvex multiblocks ADMM algorithms. Several experiments finally validate the efficacy and efficiency of our methods on both synthetic data and real world data.

Adaptive Discrete Hypergraph Matching

This paper addresses the problem of hypergraph matching using higher-order affinity information. We propose a solver that iteratively updates the solution in the discrete domain by linear assignment approximation. The proposed method is guaranteed to converge to a stationary discrete solution and avoids the annealing procedure and ad-hoc post binarization step that are required in several previous methods. Specifically, we start with a simple iterative discrete gradient assignment solver. This solver can be trapped in an -circle sequence under moderate conditions, where is the order of the graph matching problem. We then devise an adaptive relaxation mechanism to jump out this degenerating case and show that the resulting new path will converge to a fixed solution in the discrete domain. The proposed method is tested on both synthetic and real-world benchmarks. The experimental results corroborate the efficacy of our method.

Connections Between Nuclear-Norm and Frobenius-Norm-Based Representations

A lot of works have shown that frobenius-norm-based representation (FNR) is competitive to sparse representation and nuclear-norm-based representation (NNR) in numerous tasks such as subspace clustering. Despite the success of FNR in experimental studies, less theoretical analysis is provided to understand its working mechanism. In this brief, we fill this gap by building the theoretical connections between FNR and NNR. More specially, we prove that: 1) when the dictionary can provide enough representative capacity, FNR is exactly NNR even though the data set contains the Gaussian noise, Laplacian noise, or sample-specified corruption and 2) otherwise, FNR and NNR are two solutions on the column space of the dictionary.

Sparse Weighted Constrained Energy Minimization for Accurate Remote Sensing Image Target Detection

Target detection is an important task for remote sensing images, while it is still difficult to obtain satisfied performance when some images possess complex and confusion spectrum information, for example, the high similarity between target and background spectrum under some circumstance. Traditional detectors always detect target without any preprocessing procedure, which can increase the difference between target spectrum and background spectrum. Therefore, these methods could not discriminate the target from complex or similar background effectively. In this paper, sparse representation was introduced to weight each pixel for further increasing the difference between target and background spectrum. According to sparse reconstruction error matrix of pixels on images, adaptive weights will be assigned to each pixel for improving the difference between target and background spectrum. Furthermore, the sparse weighted-based constrained energy minimization method only needs to construct target dictionary, which is easier to acquire. Then, according to more distinct spectrum characteristic, the detectors can distinguish target from background more effectively and efficiency. Comparing with state-of-the-arts of target detection on remote sensing images, the proposed method can obtain more sensitive and accurate detection performance. In addition, the method is more robust to complex background than the other methods.

Automatic Subspace Learning via Principal Coefficients Embedding

In this paper, we address two challenging problems in unsupervised subspace learning: 1) how to automatically identify the feature dimension of the learned subspace (i.e., automatic subspace learning) and 2) how to learn the underlying subspace in the presence of Gaussian noise (i.e., robust subspace learning). We show that these two problems can be simultaneously solved by proposing a new method [(called principal coefficients embedding (PCE)]. For a given data set , PCE recovers a clean data set from and simultaneously learns a global reconstruction relation of . By preserving into an -dimensional space, the proposed method obtains a projection matrix that can capture the latent manifold structure of , where is automatically determined by the rank of with theoretical guarantees. PCE has three advantages: 1) it can automatically determine the feature dimension even though data are sampled from a union of multiple linear subspaces in presence of the Gaussian noise; 2) although the objective function of PCE only considers the Gaussian noise, experimental results show that it is robust to the non-Gaussian noise (e.g., random pixel corruption) and real disguises; and 3) our method has a closed-form solution and can be calculated very fast. Extensive experimental results show the superiority of PCE on a range of databases with respect to the classification accuracy, robustness, and efficiency.

Non-Convex Sparse and Low-Rank Based Robust Subspace Segmentation for Data Mining

Parsimony, including sparsity and low-rank, has shown great importance for data mining in social networks, particularly in tasks such as segmentation and recognition. Traditionally, such modeling approaches rely on an iterative algorithm that minimizes an objective function with convex l₁-norm or nuclear norm constraints. However, the obtained results by convex optimization are usually suboptimal to solutions of original sparse or low-rank problems. In this paper, a novel robust subspace segmentation algorithm has been proposed by integrating l_p-norm and Schatten p-norm constraints. Our so-obtained affinity graph can better capture local geometrical structure and the global information of the data. As a consequence, our algorithm is more generative, discriminative and robust. An efficient linearized alternating direction method is derived to realize our model. Extensive segmentation experiments are conducted on public datasets. The proposed algorithm is revealed to be more effective and robust compared to five existing algorithms.

Structured Sparse Subspace Clustering: A Joint Affinity Learning and Subspace Clustering Framework

Subspace clustering refers to the problem of segmenting data drawn from a union of subspaces. State-of-the-art approaches for solving this problem follow a two-stage approach. In the first step, an affinity matrix is learned from the data using sparse or low-rank minimization techniques. In the second step, the segmentation is found by applying spectral clustering to this affinity. While this approach has led to the state-of-the-art results in many applications, it is suboptimal, because it does not exploit the fact that the affinity and the segmentation depend on each other. In this paper, we propose a joint optimization framework - Structured Sparse Subspace Clustering (S³C) - for learning both the affinity and the segmentation. The proposed S³C framework is based on expressing each data point as a structured sparse linear combination of all other data points, where the structure is induced by a norm that depends on the unknown segmentation. Moreover, we extend the proposed S³C framework into Constrained S³C (CS³C) in which available partial side-information is incorporated into the stage of learning the affinity. We show that both the structured sparse representation and the segmentation can be found via a combination of an alternating direction method of multipliers with spectral clustering. Experiments on a synthetic data set, the Extended Yale B face data set, the Hopkins 155 motion segmentation database, and three cancer data sets demonstrate the effectiveness of our approach.