Generalization Bounds of Multitask Learning From Perspective of Vector-Valued Function Learning

A Survey on Scenario Theory, Complexity, and Compression-Based Learning and Generalization

This work investigates formal generalization error bounds that apply to support vector machines (SVMs) in realizable and agnostic learning problems. We focus on recently observed parallels between probably approximately correct (PAC)-learning bounds, such as compression and complexity-based bounds, and novel error guarantees derived within scenario theory. Scenario theory provides nonasymptotic and distributional-free error bounds for models trained by solving data-driven decision-making problems. Relevant theorems and assumptions are reviewed and discussed. We propose a numerical comparison of the tightness and effectiveness of theoretical error bounds for support vector classifiers trained on several randomized experiments from 13 real-life problems. This analysis allows for a fair comparison of different approaches from both conceptual and experimental standpoints. Based on the numerical results, we argue that the error guarantees derived from scenario theory are often tighter for realizable problems and always yield informative results, i.e., probability bounds tighter than a vacuous [0, 1] interval. This work promotes scenario theory as an alternative tool for model selection, structural-risk minimization, and generalization error analysis of SVMs. In this way, we hope to bring the communities of scenario and statistical learning theory closer, so that they can benefit from each other's insights.

Regularizing Scale-Adaptive Central Moment Sharpness for Neural Networks

In deep learning, finding flat minima of loss function is a hot research topic in improving generalization. The existing methods usually find flat minima by sharpness minimization algorithms. However, these methods suffer from insufficient flexibility for optimization and generalization due to their ignorance of loss value. This article theoretically and experimentally explores the sharpness minimization algorithms for neural networks. First, a novel scale-invariant sharpness which is called scale-adaptive central moment sharpness (SA-CMS) is proposed. This sharpness is not only scale-invariant but can characterize the nature of loss surface clearly. Based on the proposed sharpness, this article further derives a new regularization term by integrating the different orders of the sharpness. Particularly, a host of sharpness minimization functions such as local entropy can be covered by this regularization term. Then the central moment sharpness generating function is introduced as a new objective function. Moreover, theoretical analyses indicate that the new objective function has a smoother landscape and prefer converging to flat local minima. Furthermore, a computationally efficient two-stage algorithm is developed to minimize the objective function. Compared with other algorithms, the two-stage loss-sharpness minimization (TSLSM) algorithm offers a more flexible optimization target for different training stages. On a variety of learning tasks with both small and large batch sizes, this algorithm is more universal and effective, and meanwhile achieves or surpasses the generalization performance of the state-of-the-art sharpness minimization algorithms.

Efficacy of Regularized Multitask Learning Based on SVM Models

This article investigates the efficacy of a regularized multitask learning (MTL) framework based on SVM (M-SVM) to answer whether MTL always provides reliable results and how MTL outperforms independent learning. We first find that the M-SVM is Bayes risk consistent in the limit of a large sample size. This implies that despite the task dissimilarities, the M-SVM always produces a reliable decision rule for each task in terms of the misclassification error when the data size is large enough. Furthermore, we find that the task-interaction vanishes as the data size goes to infinity, and the convergence rates of the M-SVM and its single-task counterpart have the same upper bound. The former suggests that the M-SVM cannot improve the limit classifier's performance; based on the latter, we conjecture that the optimal convergence rate is not improved when the task number is fixed. As a novel insight into MTL, our theoretical and experimental results achieved an excellent agreement that the benefit of the MTL methods lies in the improvement of the preconvergence-rate (PCR) factor (to be denoted in Section III) rather than the convergence rate. Moreover, this improvement of PCR factors is more significant when the data size is small. In addition, our experimental results of five other MTL methods demonstrate the generality of this new insight.