Graph-Based Semi-Supervised Learning: A Comprehensive Review

Semi-supervised learning (SSL) has tremendous value in practice due to the utilization of both labeled and unlabelled data. An essential class of SSL methods, referred to as graph-based semi-supervised learning (GSSL) methods in the literature, is to first represent each sample as a node in an affinity graph, and then, the label information of unlabeled samples can be inferred based on the structure of the constructed graph. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large-scale data. Focusing on GSSL methods only, this work aims to provide both researchers and practitioners with a solid and systematic understanding of relevant advances as well as the underlying connections among them. The concentration on one class of SSL makes this article distinct from recent surveys that cover a more general and broader picture of SSL methods yet often neglect the fundamental understanding of GSSL methods. In particular, a significant contribution of this article lies in a newly generalized taxonomy for GSSL under the unified framework, with the most up-to-date references and valuable resources such as codes, datasets, and applications. Furthermore, we present several potential research directions as future work with our insights into this rapidly growing field.

Semisupervised Classification With Novel Graph Construction for High-Dimensional Data

Graph-based methods have achieved impressive performance on semisupervised classification (SSC). Traditional graph-based methods have two main drawbacks. First, the graph is predefined before training a classifier, which does not leverage the interactions between the classifier training and similarity matrix learning. Second, when handling high-dimensional data with noisy or redundant features, the graph constructed in the original input space is actually unsuitable and may lead to poor performance. In this article, we propose an SSC method with novel graph construction (SSC-NGC), in which the similarity matrix is optimized in both label space and an additional subspace to get a better and more robust result than in original data space. Furthermore, to obtain a high-quality subspace, we learn the projection matrix of the additional subspace by preserving the local and global structure of the data. Finally, we intergrade the classifier training, the graph construction, and the subspace learning into a unified framework. With this framework, the classifier parameters, similarity matrix, and projection matrix of subspace are adaptively learned in an iterative scheme to obtain an optimal joint result. We conduct extensive comparative experiments against state-of-the-art methods over multiple real-world data sets. Experimental results demonstrate the superiority of the proposed method over other state-of-the-art algorithms.

Robust Triple-Matrix-Recovery-Based Auto-Weighted Label Propagation for Classification

The graph-based semisupervised label propagation (LP) algorithm has delivered impressive classification results. However, the estimated soft labels typically contain mixed signs and noise, which cause inaccurate predictions due to the lack of suitable constraints. Moreover, the available methods typically calculate the weights and estimate the labels in the original input space, which typically contains noise and corruption. Thus, the encoded similarities and manifold smoothness may be inaccurate for label estimation. In this article, we present effective schemes for resolving these issues and propose a novel and robust semisupervised classification algorithm, namely the triple matrix recovery-based robust auto-weighted label propagation framework (ALP-TMR). Our ALP-TMR introduces a TMR mechanism to remove noise or mixed signs from the estimated soft labels and improve the robustness to noise and outliers in the steps of assigning weights and predicting the labels simultaneously. Our method can jointly recover the underlying clean data, clean labels, and clean weighting spaces by decomposing the original data, predicted soft labels, or weights into a clean part plus an error part by fitting noise. In addition, ALP-TMR integrates the auto-weighting process by minimizing the reconstruction errors over the recovered clean data and clean soft labels, which can encode the weights more accurately to improve both data representation and classification. By classifying samples in the recovered clean label and weight spaces, one can potentially improve the label prediction results. Extensive simulations verified the effectivenss of our ALP-TMR.

Time-Series Laplacian Semi-Supervised Learning for Indoor Localization †

Machine learning-based indoor localization used to suffer from the collection, construction, and maintenance of labeled training databases for practical implementation. Semi-supervised learning methods have been developed as efficient indoor localization methods to reduce use of labeled training data. To boost the efficiency and the accuracy of indoor localization, this paper proposes a new time-series semi-supervised learning algorithm. The key aspect of the developed method, which distinguishes it from conventional semi-supervised algorithms, is the use of unlabeled data. The learning algorithm finds spatio-temporal relationships in the unlabeled data, and pseudolabels are generated to compensate for the lack of labeled training data. In the next step, another balancing-optimization learning algorithm learns a positioning model. The proposed method is evaluated for estimating the location of a smartphone user by using a Wi-Fi received signal strength indicator (RSSI) measurement. The experimental results show that the developed learning algorithm outperforms some existing semi-supervised algorithms according to the variation of the number of training data and access points. Also, the proposed method is discussed in terms of why it gives better performance, by the analysis of the impact of the learning parameters. Moreover, the extended localization scheme in conjunction with a particle filter is executed to include additional information, such as a floor plan.

Self-Tuned Discrimination-Aware Method for Unsupervised Feature Selection

Unsupervised feature selection is fundamentally important for processing unlabeled high-dimensional data, and several methods have been proposed on this topic. Most existing embedded unsupervised methods just emphasize the data structure in the input space, which may contain large noise. Therefore, they are limited to perceive the discriminative information implied within the low-dimensional manifold. In addition, these methods always involve several parameters to be tuned, which is time-consuming. In this paper, we present a self-tuned discrimination-aware (STDA) approach for unsupervised feature selection. The main contributions of this paper are threefold: 1) it adopts the advantage of discriminant analysis technique to select the valuable features; 2) it learns the local data structure adaptively in the discriminative subspace to alleviate the effect of data noise; and 3) it performs feature selection and clustering simultaneously with an efficient optimization strategy, and saves the additional efforts to tune parameters. Experimental results on a toy data set and various real-world benchmarks justify the effectiveness of STDA on both feature selection and data clustering, and demonstrate its promising performance against the state of the arts.

Robust Adaptive Embedded Label Propagation With Weight Learning for Inductive Classification

We propose a robust inductive semi-supervised label prediction model over the embedded representation, termed adaptive embedded label propagation with weight learning (AELP-WL), for classification. AELP-WL offers several properties. First, our method seamlessly integrates the robust adaptive embedded label propagation with adaptive weight learning into a unified framework. By minimizing the reconstruction errors over embedded features and embedded soft labels jointly, our AELP-WL can explicitly ensure the learned weights to be joint optimal for representation and classification, which differs from most existing LP models that perform weight learning separately by an independent step before label prediction. Second, existing models usually precalculate the weights over the original samples that may contain unfavorable features and noise decreasing performance. To this end, our model adds a constraint that decomposes original data into a sparse component encoding embedded noise-removed sparse representations of samples and a sparse error part fitting noise, and then performs the adaptive weight learning over the embedded sparse representations. Third, our AELP-WL computes the projected soft labels by trading-off the manifold smoothness and label fitness errors over the adaptive weights and the embedded representations for enhancing the label estimation power. By including a regressive label approximation error for simultaneous minimization to correlate sample features with the embedded soft labels, the out-of-sample issue is naturally solved. By minimizing the reconstruction errors over features and embedded soft labels, classification error and label approximation error jointly, state-of-the-art results are delivered.

Joint Sparse Representation and Embedding Propagation Learning: A Framework for Graph-Based Semisupervised Learning

In this paper, we propose a novel graph-based semisupervised learning framework, called joint sparse representation and embedding propagation learning (JSREPL). The idea of JSREPL is to join EPL with sparse representation to perform label propagation. Like most of graph-based semisupervised propagation learning algorithms, JSREPL also constructs weights graph matrix from given data. Different from classical approaches which build weights graph matrix and estimate the labels of unlabeled data in sequence, JSREPL simultaneously builds weights graph matrix and estimates the labels of unlabeled data. We also propose an efficient algorithm to solve the proposed problem. The proposed method is applied to the problem of semisupervised image clustering using the ORL, Yale, PIE, and YaleB data sets. Our experiments demonstrate the effectiveness of our proposed algorithm.

Semisupervised Feature Analysis by Mining Correlations Among Multiple Tasks

In this paper, we propose a novel semisupervised feature selection framework by mining correlations among multiple tasks and apply it to different multimedia applications. Instead of independently computing the importance of features for each task, our algorithm leverages shared knowledge from multiple related tasks, thus improving the performance of feature selection. Note that the proposed algorithm is built upon an assumption that different tasks share some common structures. The proposed algorithm selects features in a batch mode, by which the correlations between various features are taken into consideration. Besides, considering the fact that labeling a large amount of training data in real world is both time-consuming and tedious, we adopt manifold learning, which exploits both labeled and unlabeled training data for a feature space analysis. Since the objective function is nonsmooth and difficult to solve, we propose an iteractive algorithm with fast convergence. Extensive experiments on different applications demonstrate that our algorithm outperforms the other state-of-the-art feature selection algorithms.

Discriminative clustering on manifold for adaptive transductive classification

In this paper, we mainly propose a novel adaptive transductive label propagation approach by joint discriminative clustering on manifolds for representing and classifying high-dimensional data. Our framework seamlessly combines the unsupervised manifold learning, discriminative clustering and adaptive classification into a unified model. Also, our method incorporates the adaptive graph weight construction with label propagation. Specifically, our method is capable of propagating label information using adaptive weights over low-dimensional manifold features, which is different from most existing studies that usually predict the labels and construct the weights in the original Euclidean space. For transductive classification by our formulation, we first perform the joint discriminative K-means clustering and manifold learning to capture the low-dimensional nonlinear manifolds. Then, we construct the adaptive weights over the learnt manifold features, where the adaptive weights are calculated through performing the joint minimization of the reconstruction errors over features and soft labels so that the graph weights can be joint-optimal for data representation and classification. Using the adaptive weights, we can easily estimate the unknown labels of samples. After that, our method returns the updated weights for further updating the manifold features. Extensive simulations on image classification and segmentation show that our proposed algorithm can deliver the state-of-the-art performance on several public datasets.

Bi-Level Semantic Representation Analysis for Multimedia Event Detection

Multimedia event detection has been one of the major endeavors in video event analysis. A variety of approaches have been proposed recently to tackle this problem. Among others, using semantic representation has been accredited for its promising performance and desirable ability for human-understandable reasoning. To generate semantic representation, we usually utilize several external image/video archives and apply the concept detectors trained on them to the event videos. Due to the intrinsic difference of these archives, the resulted representation is presumable to have different predicting capabilities for a certain event. Notwithstanding, not much work is available for assessing the efficacy of semantic representation from the source-level. On the other hand, it is plausible to perceive that some concepts are noisy for detecting a specific event. Motivated by these two shortcomings, we propose a bi-level semantic representation analyzing method. Regarding source-level, our method learns weights of semantic representation attained from different multimedia archives. Meanwhile, it restrains the negative influence of noisy or irrelevant concepts in the overall concept-level. In addition, we particularly focus on efficient multimedia event detection with few positive examples, which is highly appreciated in the real-world scenario. We perform extensive experiments on the challenging TRECVID MED 2013 and 2014 datasets with encouraging results that validate the efficacy of our proposed approach.

Concise biomarker for spatial-temporal change in three-dimensional ultrasound measurement of carotid vessel wall and plaque thickness based on a graph-based random walk framework: Towards sensitive evaluation of response to therapy

Rapid progression in total plaque area and volume measured from ultrasound images has been shown to be associated with an elevated risk of cardiovascular events. Since atherosclerosis is focal and predominantly occurring at the bifurcation, biomarkers that are able to quantify the spatial distribution of vessel-wall-plus-plaque thickness (VWT) change may allow for more sensitive detection of treatment effect. The goal of this paper is to develop simple and sensitive biomarkers to quantify the responsiveness to therapies based on the spatial distribution of VWT-Change on the entire 2D carotid standardized map previously described. Point-wise VWT-Changes computed for each patient were reordered lexicographically to a high-dimensional data node in a graph. A graph-based random walk framework was applied with the novel Weighted Cosine (WCos) similarity function introduced, which was tailored for quantification of responsiveness to therapy. The converging probability of each data node to the VWT regression template in the random walk process served as a scalar descriptor for VWT responsiveness to treatment. The WCos-based biomarker was 14 times more sensitive than the mean VWT-Change in discriminating responsive and unresponsive subjects based on the p-values obtained in T-tests. The proposed framework was extended to quantify where VWT-Change occurred by including multiple VWT-Change distribution templates representing focal changes at different regions. Experimental results show that the framework was effective in classifying carotid arteries with focal VWT-Change at different locations and may facilitate future investigations to correlate risk of cardiovascular events with the location where focal VWT-Change occurs.

Can the Virtual Labels Obtained by Traditional LP Approaches Be Well Encoded in WLR?

Semisupervised dimension reduction via virtual label regression first derives the virtual labels of unlabeled data by employing a newly designed label propagation (LP) approach (called Special random walk (SRW)) and then encodes them in a weighted linear regression model. Nie et al. (2011) highlighted two important characteristics of SRW nonexistent in the previous LP approaches: outlier detection and probability value output, which guarantee the elegant encoding of the resultant virtual labels in the weighted label regression. However, in this brief, we show that the relationship between the SRW and the previous work on LP is very close. Naturally, a problem deserving investigation is whether traditional LP approaches are indeed unable to share the above two characteristics of SRW. We aim to address this problem.

Binarization With Boosting and Oversampling for Multiclass Classification

Using a set of binary classifiers to solve multiclass classification problems has been a popular approach over the years. The decision boundaries learnt by binary classifiers (also called base classifiers) are much simpler than those learnt by multiclass classifiers. This paper proposes a new classification framework, termed binarization with boosting and oversampling (BBO), for efficiently solving multiclass classification problems. The new framework is devised based on the one-versus-all (OVA) binarization technique. Unlike most previous work, BBO employs boosting for solving the hard-to-learn instances and oversampling for handling the class-imbalance problem arising due to OVA binarization. These two features make BBO different from other existing works. Our new framework has been tested extensively on several multiclass supervised and semi-supervised classification problems using five different base classifiers, including neural networks, C4.5, k -nearest neighbor, repeated incremental pruning to produce error reduction, support vector machine, random forest, and learning with local and global consistency. Experimental results show that BBO can exhibit better performance compared to its counterparts on supervised and semi-supervised classification problems.

Semi-Supervised Text Classification With Universum Learning

Universum, a collection of nonexamples that do not belong to any class of interest, has become a new research topic in machine learning. This paper devises a semi-supervised learning with Universum algorithm based on boosting technique, and focuses on situations where only a few labeled examples are available. We also show that the training error of AdaBoost with Universum is bounded by the product of normalization factor, and the training error drops exponentially fast when each weak classifier is slightly better than random guessing. Finally, the experiments use four data sets with several combinations. Experimental results indicate that the proposed algorithm can benefit from Universum examples and outperform several alternative methods, particularly when insufficient labeled examples are available. When the number of labeled examples is insufficient to estimate the parameters of classification functions, the Universum can be used to approximate the prior distribution of the classification functions. The experimental results can be explained using the concept of Universum introduced by Vapnik, that is, Universum examples implicitly specify a prior distribution on the set of classification functions.

Learning Flexible Graph-Based Semi-Supervised Embedding

This paper introduces a graph-based semi-supervised embedding method as well as its kernelized version for generic classification and recognition tasks. The aim is to combine the merits of flexible manifold embedding and nonlinear graph-based embedding for semi-supervised learning. The proposed linear method will be flexible since it estimates a nonlinear manifold that is the closest one to a linear embedding. The proposed kernelized method will also be flexible since it estimates a kernel-based embedding that is the closest to a nonlinear manifold. In both proposed methods, the nonlinear manifold and the mapping (linear transform for the linear method and the kernel multipliers for the kernelized method) are simultaneously estimated, which overcomes the shortcomings of a cascaded estimation. The dimension of the final embedding obtained by the two proposed methods is not limited to the number of classes. They can be used by any kind of classifiers once the data are embedded into the new subspaces. Unlike nonlinear dimensionality reduction approaches, which suffer from out-of-sample problem, our proposed methods have an obvious advantage that the learnt subspace has a direct out-of-sample extension to novel samples, and are thus easily generalized to the entire high-dimensional input space. We provide extensive experiments on seven public databases in order to study the performance of the proposed methods. These experiments demonstrate much improvement over the state-of-the-art algorithms that are based on label propagation or graph-based semi-supervised embedding.

Learning a Nonnegative Sparse Graph for Linear Regression

Previous graph-based semisupervised learning (G-SSL) methods have the following drawbacks: 1) they usually predefine the graph structure and then use it to perform label prediction, which cannot guarantee an overall optimum and 2) they only focus on the label prediction or the graph structure construction but are not competent in handling new samples. To this end, a novel nonnegative sparse graph (NNSG) learning method was first proposed. Then, both the label prediction and projection learning were integrated into linear regression. Finally, the linear regression and graph structure learning were unified within the same framework to overcome these two drawbacks. Therefore, a novel method, named learning a NNSG for linear regression was presented, in which the linear regression and graph learning were simultaneously performed to guarantee an overall optimum. In the learning process, the label information can be accurately propagated via the graph structure so that the linear regression can learn a discriminative projection to better fit sample labels and accurately classify new samples. An effective algorithm was designed to solve the corresponding optimization problem with fast convergence. Furthermore, NNSG provides a unified perceptiveness for a number of graph-based learning methods and linear regression methods. The experimental results showed that NNSG can obtain very high classification accuracy and greatly outperforms conventional G-SSL methods, especially some conventional graph construction methods.

Manifold Adaptive Label Propagation for Face Clustering

In this paper, a novel label propagation (LP) method is presented, called the manifold adaptive label propagation (MALP) method, which is to extend original LP by integrating sparse representation constraint into regularization framework of LP method. Similar to most LP, first of all, MALP also finds graph edges from given data and gives weights to the graph edges. Our goal is to find graph weights matrix adaptively. The key advantage of our approach is that MALP simultaneously finds graph weights matrix and predicts the label of unlabeled data. This paper also derives efficient algorithm to solve the proposed problem. Extensions of our MALP in kernel space and robust version are presented. The proposed method has been applied to the problem of semi-supervised face clustering using the well-known ORL, Yale, extended YaleB, and PIE datasets. Our experimental evaluations show the effectiveness of our method.

Successive overrelaxation for laplacian support vector machine

Semisupervised learning (SSL) problem, which makes use of both a large amount of cheap unlabeled data and a few unlabeled data for training, in the last few years, has attracted amounts of attention in machine learning and data mining. Exploiting the manifold regularization (MR), Belkin et al. proposed a new semisupervised classification algorithm: Laplacian support vector machines (LapSVMs), and have shown the state-of-the-art performance in SSL field. To further improve the LapSVMs, we proposed a fast Laplacian SVM (FLapSVM) solver for classification. Compared with the standard LapSVM, our method has several improved advantages as follows: 1) FLapSVM does not need to deal with the extra matrix and burden the computations related to the variable switching, which make it more suitable for large scale problems; 2) FLapSVM’s dual problem has the same elegant formulation as that of standard SVMs. This means that the kernel trick can be applied directly into the optimization model; and 3) FLapSVM can be effectively solved by successive overrelaxation technology, which converges linearly to a solution and can process very large data sets that need not reside in memory. In practice, combining the strategies of random scheduling of subproblem and two stopping conditions, the computing speed of FLapSVM is rigidly quicker to that of LapSVM and it is a valid alternative to PLapSVM.

Compact Graph based Semi-Supervised Learning for Medical Diagnosis in Alzheimer's Disease

Dementia is one of the most common neurological disorders among the elderly. Identifying those who are of high risk suffering dementia is important for early diagnosis in order to slow down the disease progression and help preserve some cognitive functions of the brain. To achieve accurate classification, significant amount of subject feature information are involved. Hence identification of demented subjects can be transformed into a pattern classification problem. In this letter, we introduce a graph based semi-supervised learning algorithm for Medical Diagnosis by using partly labeled samples and large amount of unlabeled samples. The new method is derived by a compact graph that can well grasp the manifold structure of medical data. Simulation results show that the proposed method can achieve better sensitivities and specificities compared with other state-of-art graph based semi-supervised learning methods.

Joint embedding learning and sparse regression: a framework for unsupervised feature selection

Feature selection has aroused considerable research interests during the last few decades. Traditional learning-based feature selection methods separate embedding learning and feature ranking. In this paper, we propose a novel unsupervised feature selection framework, termed as the joint embedding learning and sparse regression (JELSR), in which the embedding learning and sparse regression are jointly performed. Specifically, the proposed JELSR joins embedding learning with sparse regression to perform feature selection. To show the effectiveness of the proposed framework, we also provide a method using the weight via local linear approximation and adding the l2,1 -norm regularization, and design an effective algorithm to solve the corresponding optimization problem. Furthermore, we also conduct some insightful discussion on the proposed feature selection approach, including the convergence analysis, computational complexity, and parameter determination. In all, the proposed framework not only provides a new perspective to view traditional methods but also evokes some other deep researches for feature selection. Compared with traditional unsupervised feature selection methods, our approach could integrate the merits of embedding learning and sparse regression. Promising experimental results on different kinds of data sets, including image, voice data and biological data, have validated the effectiveness of our proposed algorithm.

Error analysis of stochastic gradient descent ranking

Ranking is always an important task in machine learning and information retrieval, e.g., collaborative filtering, recommender systems, drug discovery, etc. A kernel-based stochastic gradient descent algorithm with the least squares loss is proposed for ranking in this paper. The implementation of this algorithm is simple, and an expression of the solution is derived via a sampling operator and an integral operator. An explicit convergence rate for leaning a ranking function is given in terms of the suitable choices of the step size and the regularization parameter. The analysis technique used here is capacity independent and is novel in error analysis of ranking learning. Experimental results on real-world data have shown the effectiveness of the proposed algorithm in ranking tasks, which verifies the theoretical analysis in ranking error.