Pareto-path multitask multiple kernel learning

Cognitive Assessment Prediction in Alzheimer's Disease by Multi-Layer Multi-Target Regression

Accurate and automatic prediction of cognitive assessment from multiple neuroimaging biomarkers is crucial for early detection of Alzheimer's disease. The major challenges arise from the nonlinear relationship between biomarkers and assessment scores and the inter-correlation among them, which have not yet been well addressed. In this paper, we propose multi-layer multi-target regression (MMR) which enables simultaneously modeling intrinsic inter-target correlations and nonlinear input-output relationships in a general compositional framework. Specifically, by kernelized dictionary learning, the MMR can effectively handle highly nonlinear relationship between biomarkers and assessment scores; by robust low-rank linear learning via matrix elastic nets, the MMR can explicitly encode inter-correlations among multiple assessment scores; moreover, the MMR is flexibly and allows to work with non-smooth ℓ2,1-norm loss function, which enables calibration of multiple targets with disparate noise levels for more robust parameter estimation. The MMR can be efficiently solved by an alternating optimization algorithm via gradient descent with guaranteed convergence. The MMR has been evaluated by extensive experiments on the ADNI database with MRI data, and produced high accuracy surpassing previous regression models, which demonstrates its great effectiveness as a new multi-target regression model for clinical multivariate prediction.

On the Impact of Regularization Variation on Localized Multiple Kernel Learning

This brief analyzes the effects of regularization variations in the localized kernel weights on the hypothesis generated by localized multiple kernel learning (LMKL) algorithms. Recent research on LMKL includes imposing different regularizations on the localized kernel weights and has led to varying formulations and solution strategies. Following the stability analysis theory as presented by Bousquet and Elisseeff, we give stability bounds based on the norm of the variation of localized kernel weights for three LMKL methods cast in the support vector machine classification framework, including vector -norm LMKL, matrix-regularized -norm LMKL, and samplewise -norm LMKL. Further comparison of these bounds helps to qualitatively reveal the performance differences produced by these regularization methods, that is, matrix-regularized LMKL achieves superior performance, followed by vector -norm LMKL and samplewise -norm LMKL. Finally, a set of experimental results on ten benchmark machine learning UCI data sets is reported and shown to empirically support our theoretical analysis.

On Better Exploring and Exploiting Task Relationships in Multitask Learning: Joint Model and Feature Learning

Multitask learning (MTL) aims to learn multiple tasks simultaneously through the interdependence between different tasks. The way to measure the relatedness between tasks is always a popular issue. There are mainly two ways to measure relatedness between tasks: common parameters sharing and common features sharing across different tasks. However, these two types of relatedness are mainly learned independently, leading to a loss of information. In this paper, we propose a new strategy to measure the relatedness that jointly learns shared parameters and shared feature representations. The objective of our proposed method is to transform the features of different tasks into a common feature space in which the tasks are closely related and the shared parameters can be better optimized. We give a detailed introduction to our proposed MTL method. Additionally, an alternating algorithm is introduced to optimize the nonconvex objection. A theoretical bound is given to demonstrate that the relatedness between tasks can be better measured by our proposed MTL algorithm. We conduct various experiments to verify the superiority of the proposed joint model and feature MTL method.

Feature Combination via Clustering

In image classification, feature combination is often used to combine the merits of multiple complementary features and improve the classification accuracy compared with one single feature. Existing feature combination algorithms, e.g., multiple kernel learning, usually determine the weights of features based on the optimization with respect to some classifier-dependent objective function. These algorithms are often computationally expensive, and in some cases are found to perform no better than simple baselines. In this paper, we solve the feature combination problem from a totally different perspective. Our algorithm is based on the simple idea of combining only base kernels suitable to be combined. Since the very aim of feature combination is to obtain the highest possible classification accuracy, we measure the combination suitableness of two base kernels by the maximum possible cross-validation accuracy of their combined kernel. By regarding the pairwise suitableness as the kernel adjacency, we obtain a weighted graph of all base kernels and find that the base kernels suitable to be combined correspond to a cluster in the graph. We then use the dominant sets algorithm to find the cluster and determine the weights of base kernels automatically. In this way, we transform the kernel combination problem into a clustering one. Our algorithm can be implemented in parallel easily and the running time can be adjusted based on available memory to a large extent. In experiments on several data sets, our algorithm generates comparable classification accuracy with the state of the art.

Probabilistic Low-Rank Multitask Learning

In this paper, we consider the problem of learning multiple related tasks simultaneously with the goal of improving the generalization performance of individual tasks. The key challenge is to effectively exploit the shared information across multiple tasks as well as preserve the discriminative information for each individual task. To address this, we propose a novel probabilistic model for multitask learning (MTL) that can automatically balance between low-rank and sparsity constraints. The former assumes a low-rank structure of the underlying predictive hypothesis space to explicitly capture the relationship of different tasks and the latter learns the incoherent sparse patterns private to each task. We derive and perform inference via variational Bayesian methods. Experimental results on both regression and classification tasks on real-world applications demonstrate the effectiveness of the proposed method in dealing with the MTL problems.

Localized Multiple Kernel Learning With Dynamical Clustering and Matrix Regularization

Localized multiple kernel learning (LMKL) is an attractive strategy for combining multiple heterogeneous features with regard to their discriminative power for each individual sample. However, the learning of numerous local solutions may not scale well even for a moderately sized training set, and the independently learned local models may suffer from overfitting. Hence, in existing local methods, the distributed samples are typically assumed to share the same weights, and various unsupervised clustering methods are applied as preprocessing. In this paper, to enable the learner to discover and benefit from the underlying local coherence and diversity of the samples, we incorporate the clustering procedure into the canonical support vector machine-based LMKL framework. Then, to explore the relatedness among different samples, which has been ignored in a vector -norm analysis, we organize the cluster-specific kernel weights into a matrix and introduce a matrix-based extension of the -norm for constraint enforcement. By casting the joint optimization problem as a problem of alternating optimization, we show how the cluster structure is gradually revealed and how the matrix-regularized kernel weights are obtained. A theoretical analysis of such a regularizer is performed using a Rademacher complexity bound, and complementary empirical experiments on real-world data sets demonstrate the effectiveness of our technique.

Matrix-Regularized Multiple Kernel Learning via (r,p) Norms

This paper examines a matrix-regularized multiple kernel learning (MKL) technique based on a notion of (r,p) norms. For the problem of learning a linear combination in the support vector machine-based framework, model complexity is typically controlled using various regularization strategies on the combined kernel weights. Recent research has developed a generalized ℓp-norm MKL framework with tunable variable p(p≥1) to support controlled intrinsic sparsity. Unfortunately, this ``1-D'' vector ℓp-norm hardly exploits potentially useful information on how the base kernels ``interact.'' To allow for higher order kernel-pair relationships, we extend the ``1-D'' vector ℓp-MKL to the ``2-D'' matrix (r,p) norms (1 ≤ r,p < ∞). We develop a new formulation and an efficient optimization strategy for (r,p)-MKL with guaranteed convergence. A theoretical analysis and experiments on seven UCI data sets shed light on the superiority of (r,p)-MKL over ℓp-MKL in various scenarios.