Softly Associative Transfer Learning for Cross-Domain Classification

Fast Transfer Learning Method Using Random Layer Freezing and Feature Refinement Strategy

Recently, Moore-Penrose inverse (MPI)-based parameter fine-tuning of fully connected (FC) layers in pretrained deep convolutional neural networks (DCNNs) has emerged within the inductive transfer learning (ITL) paradigm. However, this approach has not gained significant traction in practical applications due to its stringent computational requirements. This work addresses this issue through a novel fast retraining strategy that enhances applicability of the MPI-based ITL. Specifically, during each retraining epoch, a random layer freezing protocol is utilized to manage the number of layers undergoing feature refinement. Additionally, this work incorporates an MPI-based approach for refining the trainable parameters of FC layers under batch processing, contributing to expedited convergence. Extensive experiments on several ImageNet pretrained benchmark DCNNs demonstrate that the proposed ITL achieves competitive performance with excellent convergence speed compared to conventional ITL methods. For instance, the proposed strategy converges nearly 1.5 times faster than retraining the ImageNet pretrained ResNet-50 using stochastic gradient descent with momentum (SGDM).

Genetic Programming for Document Classification: A Transductive Transfer Learning System

Document classification is a challenging task to the data being high-dimensional and sparse. Many transfer learning methods have been investigated for improving the classification performance by effectively transferring knowledge from a source domain to a target domain, which is similar to but different from the source domain. However, most of the existing methods cannot handle the case that the training data of the target domain does not have labels. In this study, we propose a transductive transfer learning system, utilizing solutions evolved by genetic programming (GP) on a source domain to automatically pseudolabel the training data in the target domain in order to train classifiers. Different from many other transfer learning techniques, the proposed system pseudolabels target-domain training data to retrains classifiers using all target-domain features. The proposed method is examined on nine transfer learning tasks, and the results show that the proposed transductive GP system has better prediction accuracy on the test data in the target domain than existing transfer learning approaches including subspace alignment-domain adaptation methods, feature-level-domain adaptation methods, and one latest pseudolabeling strategy-based method.

Guided Discrimination and Correlation Subspace Learning for Domain Adaptation

As a branch of transfer learning, domain adaptation leverages useful knowledge from a source domain to a target domain for solving target tasks. Most of the existing domain adaptation methods focus on how to diminish the conditional distribution shift and learn invariant features between different domains. However, two important factors are overlooked by most existing methods: 1) the transferred features should be not only domain invariant but also discriminative and correlated, and 2) negative transfer should be avoided as much as possible for the target tasks. To fully consider these factors in domain adaptation, we propose a guided discrimination and correlation subspace learning (GDCSL) method for cross-domain image classification. GDCSL considers the domain-invariant, category-discriminative, and correlation learning of data. Specifically, GDCSL introduces the discriminative information associated with the source and target data by minimizing the intraclass scatter and maximizing the interclass distance. By designing a new correlation term, GDCSL extracts the most correlated features from the source and target domains for image classification. The global structure of the data can be preserved in GDCSL because the target samples are represented by the source samples. To avoid negative transfer issues, we use a sample reweighting method to detect target samples with different confidence levels. A semi-supervised extension of GDCSL (Semi-GDCSL) is also proposed, and a novel label selection scheme is introduced to ensure the correction of the target pseudo-labels. Comprehensive and extensive experiments are conducted on several cross-domain data benchmarks. The experimental results verify the effectiveness of the proposed methods over state-of-the-art domain adaptation methods.

Subdomain Adaptation With Manifolds Discrepancy Alignment

Reducing domain divergence is a key step in transfer learning. Existing works focus on the minimization of global domain divergence. However, two domains may consist of several shared subdomains, and differ from each other in each subdomain. In this article, we take the local divergence of subdomains into account in transfer. Specifically, we propose to use the low-dimensional manifold to represent the subdomain, and align the local data distribution discrepancy in each manifold across domains. A manifold maximum mean discrepancy (M3D) is developed to measure the local distribution discrepancy in each manifold. We then propose a general framework, called transfer with manifolds discrepancy alignment (TMDA), to couple the discovery of data manifolds with the minimization of M3D. We instantiate TMDA in the subspace learning case considering both the linear and nonlinear mappings. We also instantiate TMDA in the deep learning framework. Experimental studies show that TMDA is a promising method for various transfer learning tasks.

Distribution matching and structure preservation for domain adaptation

Cross-domain classification refers to completing the corresponding classification task in a target domain which lacks label information, by exploring useful knowledge in a related source domain but with different data distribution. Domain adaptation can deal with such cross-domain classification, by reducing divergence of domains and transferring the relevant knowledge from the source to the target. To mine the discriminant information of the source domain samples and the geometric structure information of domains, and thus improve domain adaptation performance, this paper proposes a novel method involving distribution matching and structure preservation for domain adaptation (DMSP). First, it aligns the subspaces of the source domain and target domain on the Grassmann manifold; and learns the non-distorted embedded feature representations of the two domains. Second, in this embedded feature space, the empirical structure risk minimization method with distribution adaptation regularization and intra-domain graph regularization is used to learn an adaptive classifier, further adapting the source and target domains. Finally, we perform extensive experiments on widely used cross-domain classification datasets to validate the superiority of DMSP. The average classification accuracy of DMSP on these datasets is the highest compared with several state-of-the-art domain adaptation methods.

Dual-Space Transfer Learning Based on an Indirect Mutual Promotion Strategy

Transfer learning is designed to leverage knowledge in the source domain with labels to help build classification models in the target domain where labels are scarce or even unavailable. Previous studies have shown that high-level concepts extracted from original features are more suitable for cross-domain classification tasks, so many transfer learning methods transfer knowledge by modeling high-level concepts on the original feature space. However, there are two limitations to this method: First, learning high-level concepts directly on the original feature space will reduce the proportion of shared information contained in common features in the process of knowledge transfer bridge construction. Second, only learning multiple high-level concepts on the original feature space, the latent shared information contained in the domain-specific features cannot be targeted learned, so the latent shared information in the domain-specific features cannot be effectively used. To overcome these limitations, this paper proposes a novel method named Dual-Space Transfer Learning based on an Indirect Mutual Promotion Strategy (DSTL). The DSTL method is formalized as an optimization problem based on non-negative matrix tri-factorization. DSTL first extracts the common features between domains and constructs the common feature space. Then, the learning of the high-level concepts of the common feature space and the original feature space is integrated through an indirect promotion strategy, which can enhance the learning effect of common features and domain-specific features through the mutual help of the two feature spaces. The system test on benchmark data sets shows the superiority of the DSTL method.

Efficacy of Regularized Multitask Learning Based on SVM Models

This article investigates the efficacy of a regularized multitask learning (MTL) framework based on SVM (M-SVM) to answer whether MTL always provides reliable results and how MTL outperforms independent learning. We first find that the M-SVM is Bayes risk consistent in the limit of a large sample size. This implies that despite the task dissimilarities, the M-SVM always produces a reliable decision rule for each task in terms of the misclassification error when the data size is large enough. Furthermore, we find that the task-interaction vanishes as the data size goes to infinity, and the convergence rates of the M-SVM and its single-task counterpart have the same upper bound. The former suggests that the M-SVM cannot improve the limit classifier's performance; based on the latter, we conjecture that the optimal convergence rate is not improved when the task number is fixed. As a novel insight into MTL, our theoretical and experimental results achieved an excellent agreement that the benefit of the MTL methods lies in the improvement of the preconvergence-rate (PCR) factor (to be denoted in Section III) rather than the convergence rate. Moreover, this improvement of PCR factors is more significant when the data size is small. In addition, our experimental results of five other MTL methods demonstrate the generality of this new insight.

I-Keyboard: Fully Imaginary Keyboard on Touch Devices Empowered by Deep Neural Decoder

Text entry aims to provide an effective and efficient pathway for humans to deliver their messages to computers. With the advent of mobile computing, the recent focus of text-entry research has moved from physical keyboards to soft keyboards. Current soft keyboards, however, increase the typo rate due to a lack of tactile feedback and degrade the usability of mobile devices due to their large portion on screens. To tackle these limitations, we propose a fully imaginary keyboard (I-Keyboard) with a deep neural decoder (DND). The invisibility of I-Keyboard maximizes the usability of mobile devices and DND empowered by a deep neural architecture allows users to start typing from any position on the touch screens at any angle. To the best of our knowledge, the eyes-free ten-finger typing scenario of I-Keyboard which does not necessitate both a calibration step and a predefined region for typing is first explored in this article. For the purpose of training DND, we collected the largest user data in the process of developing I-Keyboard. We verified the performance of the proposed I-Keyboard and DND by conducting a series of comprehensive simulations and experiments under various conditions. I-Keyboard showed 18.95% and 4.06% increases in typing speed (45.57 words per minute) and accuracy (95.84%), respectively, over the baseline.