Robust C-Loss Kernel Classifiers

Sparse Bayesian correntropy learning for robust muscle activity reconstruction from noisy brain recordings

Sparse Bayesian learning has promoted many effective frameworks of brain activity decoding for the brain-computer interface, including the direct reconstruction of muscle activity using brain recordings. However, existing sparse Bayesian learning algorithms mainly use Gaussian distribution as error assumption in the reconstruction task, which is not necessarily the truth in the real-world application. On the other hand, brain recording is known to be highly noisy and contains many non-Gaussian noises, which could lead to large performance degradation for sparse Bayesian learning algorithms. The goal of this paper is to propose a novel robust implementation of sparse Bayesian learning so that robustness and sparseness can be realized simultaneously. Motivated by the exceptional robustness of maximum correntropy criterion (MCC), we proposed integrating MCC to the sparse Bayesian learning regime. To be specific, we derived the explicit error assumption inherent in the MCC, and then leveraged it for the likelihood function. Meanwhile, we utilized the automatic relevance determination technique as the sparse prior distribution. To fully evaluate the proposed method, a synthetic example and a real-world muscle activity reconstruction task with two different brain modalities were leveraged. Experimental results showed, our proposed sparse Bayesian correntropy learning framework significantly improves the robustness for the noisy regression tasks. Our proposed algorithm could realize higher correlation coefficients and lower root mean squared errors for the real-world muscle activity reconstruction scenario. Sparse Bayesian correntropy learning provides a powerful approach for brain activity decoding which will promote the development of brain-computer interface technology.

A robust semi-supervised regressor with correntropy-induced manifold regularization and adaptive graph

For semi-supervised regression tasks, most existing methods ignore the impact of noise. However, the data inevitably contain noise. Therefore, this study proposes a novel correntropy-induced semi-supervised regression (CSSR) method that mitigates the adverse effects of noise. To implement the robustness of CSSR, a novel correntropy-induced manifold regularization (CMR) and a correntropy-induced adaptive graph (CAG) are designed. Specifically, CMR is inspired by the principles of correntropy and aims to learn a graph representation, whereas CAG inherits the robust characteristics of the correntropy metric and adaptively constructs an adjacency matrix. Finally, by incorporating CMR, CAG, and the correntropy-induced loss seamlessly, CSSR demonstrates the ability to deliver promising joint performance. The final solution of CSSR is achieved through an iterative process. Moreover, we validated the convergence of CSSR through a combination of theoretical analyses and empirical experiments. The experimental evaluation encompassed three synthetic, 15 benchmark, and two image datasets. The findings demonstrate that CSSR surpasses similar methods in the realm of semi-supervised regression tasks, demonstrating its effectiveness and robustness.

Sparse Additive Machine With the Correntropy-Induced Loss

Sparse additive machines (SAMs) have shown competitive performance on variable selection and classification in high-dimensional data due to their representation flexibility and interpretability. However, the existing methods often employ the unbounded or nonsmooth functions as the surrogates of 0-1 classification loss, which may encounter the degraded performance for data with outliers. To alleviate this problem, we propose a robust classification method, named SAM with the correntropy-induced loss (CSAM), by integrating the correntropy-induced loss (C-loss), the data-dependent hypothesis space, and the weighted -norm regularizer ( ) into additive machines. In theory, the generalization error bound is estimated via a novel error decomposition and the concentration estimation techniques, which shows that the convergence rate can be achieved under proper parameter conditions. In addition, the theoretical guarantee on variable selection consistency is analyzed. Experimental evaluations on both synthetic and real-world datasets consistently validate the effectiveness and robustness of the proposed approach.

A novel bounded loss framework for support vector machines

This paper introduces a novel bounded loss framework for SVM and SVR. Specifically, using the Pinball loss as an illustration, we devise a novel bounded exponential quantile loss (Leq-loss) for both support vector machine classification and regression tasks. For Leq-loss, it not only enhances the robustness of SVM and SVR against outliers but also improves the robustness of SVM to resampling from a different perspective. Furthermore, EQSVM and EQSVR were constructed based on Leq-loss, and the influence functions and breakdown point lower bounds of their estimators are derived. It is proved that the influence functions are bounded, and the breakdown point lower bounds can reach the highest asymptotic breakdown point of 1/2. Additionally, we demonstrated the robustness of EQSVM to resampling and derived its generalization error bound based on Rademacher complexity. Due to the Leq-loss being non-convex, we can use the concave-convex procedure (CCCP) technique to transform the problem into a series of convex optimization problems and use the ClipDCD algorithm to solve these convex optimization problems. Numerous experiments have been conducted to confirm the effectiveness of the proposed EQSVM and EQSVR.

Identification of drug-side effect association via correntropy-loss based matrix factorization with neural tangent kernel

Adverse drug reactions include side effects, allergic reactions, and secondary infections. Severe adverse reactions can cause cancer, deformity, or mutation. The monitoring of drug side effects is an important support for post marketing safety supervision of drugs, and an important basis for revising drug instructions. Its purpose is to timely detect and control drug safety risks. Traditional methods are time-consuming. To accelerate the discovery of side effects, we propose a machine learning based method, called correntropy-loss based matrix factorization with neural tangent kernel (CLMF-NTK), to solve the prediction of drug side effects. Our method and other computational methods are tested on three benchmark datasets, and the results show that our method achieves the best predictive performance.

Capped Asymmetric Elastic Net Support Vector Machine for Robust Binary Classification

Recently, there are lots of literature on improving the robustness of SVM by constructing nonconvex functions, but they seldom theoretically study the robust property of the constructed functions. In this paper, based on our recent work, we present a novel capped asymmetric elastic net (CaEN) loss and equip it with the SVM as CaENSVM. We derive the influence function of the estimators of the CaENSVM to theoretically explain the robustness of the proposed method. Our results can be easily extended to other similar nonconvex loss functions. We further show that the influence function of the CaENSVM is bounded, so that the robustness of the CaENSVM can be theoretically explained. Other theoretical analysis demonstrates that the CaENSVM satisfies the Bayes rule and the corresponding generalization error bound based on Rademacher complexity guarantees its good generalization capability. Since CaEN loss is concave, we implement an efficient DC procedure based on the stochastic gradient descent algorithm (Pegasos) to solve the optimization problem. A host of experiments are conducted to verify the effectiveness of our proposed CaENSVM model.

Multikernel Correntropy for Robust Learning

As a novel similarity measure that is defined as the expectation of a kernel function between two random variables, correntropy has been successfully applied in robust machine learning and signal processing to combat large outliers. The kernel function in correntropy is usually a zero-mean Gaussian kernel. In a recent work, the concept of mixture correntropy (MC) was proposed to improve the learning performance, where the kernel function is a mixture Gaussian kernel, namely, a linear combination of several zero-mean Gaussian kernels with different widths. In both correntropy and MC, the center of the kernel function is, however, always located at zero. In the present work, to further improve the learning performance, we propose the concept of multikernel correntropy (MKC), in which each component of the mixture Gaussian kernel can be centered at a different location. The properties of the MKC are investigated and an efficient approach is proposed to determine the free parameters in MKC. Experimental results show that the learning algorithms under the maximum MKC criterion (MMKCC) can outperform those under the original maximum correntropy criterion (MCC) and the maximum MC criterion (MMCC).

Restricted Minimum Error Entropy Criterion for Robust Classification

The minimum error entropy (MEE) criterion is a powerful approach for non-Gaussian signal processing and robust machine learning. However, the instantiation of MEE on robust classification is a rather vacancy in the literature. The original MEE purely focuses on minimizing Renyi's quadratic entropy of the prediction errors, which could exhibit inferior capability in noisy classification tasks. To this end, we analyze the optimal error distribution with adverse outliers and introduce a specific codebook for restriction, which optimizes the error distribution toward the optimal case. Half-quadratic-based optimization and convergence analysis of the proposed learning criterion, called restricted MEE (RMEE), are provided. The experimental results considering logistic regression and extreme learning machine on synthetic data and UCI datasets, respectively, are presented to demonstrate the superior robustness of RMEE. Furthermore, we evaluate RMEE on a noisy electroencephalogram dataset, so as to strengthen its practical impact.

Kernel Correntropy Conjugate Gradient Algorithms Based on Half-Quadratic Optimization

As a nonlinear similarity measure defined in the kernel space, the correntropic loss (C-Loss) can address the stability issues of second-order similarity measures thanks to its ability to extract high-order statistics of data. However, the kernel adaptive filter (KAF) based on the C-Loss uses the stochastic gradient descent (SGD) method to update its weights and, thus, suffers from poor performance and a slow convergence rate. To address these issues, the conjugate gradient (CG)-based correntropy algorithm is developed by solving the combination of half-quadratic (HQ) optimization and weighted least-squares (LS) problems, generating a novel robust kernel correntropy CG (KCCG) algorithm. The proposed KCCG with less computational complexity achieves comparable performance to the kernel recursive maximum correntropy (KRMC) algorithm. To further curb the growth of the network in KCCG, the random Fourier features KCCG (RFFKCCG) algorithm is proposed by transforming the original input data into a fixed-dimensional random Fourier features space (RFFS). Since only one current error information is used in the loss function of RFFKCCG, it can provide a more efficient filter structure than the other KAFs with sparsification. The Monte Carlo simulations conducted in the prediction of synthetic and real-world chaotic time series and the regression for large-scale datasets validate the superiorities of the proposed algorithms in terms of robustness, filtering accuracy, and complexity.

δ-Norm-Based Robust Regression With Applications to Image Analysis

Up to now, various matrix norms (e.g., l₁ -norm, l₂ -norm, l_2,1 -norm, etc.) have been widely leveraged to form the loss function of different regression models, and have played an important role in image analysis. However, the previous regression models adopting the existing norms are sensitive to outliers and, thus, often bring about unsatisfactory results on the heavily corrupted images. This is because their adopted norms for measuring the data residual can hardly suppress the negative influence of noisy data, which will probably mislead the regression process. To address this issue, this paper proposes a novel δ (delta)-norm to count the nonzero blocks around an element in a vector or matrix, which weakens the impacts of outliers and also takes the structure property of examples into account. After that, we present the δ -norm-based robust regression (DRR) in which the data examples are mapped to the kernel space and measured by the proposed δ -norm. By exploring an explicit kernel function, we show that DRR has a closed-form solution, which suggests that DRR can be efficiently solved. To further handle the influences from mixed noise, DRR is extended to a multiscale version. The experimental results on image classification and background modeling datasets validate the superiority of the proposed approach to the existing state-of-the-art robust regression models.

Correntropy induced loss based sparse robust graph regularized extreme learning machine for cancer classification

Background

As a machine learning method with high performance and excellent generalization ability, extreme learning machine (ELM) is gaining popularity in various studies. Various ELM-based methods for different fields have been proposed. However, the robustness to noise and outliers is always the main problem affecting the performance of ELM.

Results

In this paper, an integrated method named correntropy induced loss based sparse robust graph regularized extreme learning machine (CSRGELM) is proposed. The introduction of correntropy induced loss improves the robustness of ELM and weakens the negative effects of noise and outliers. By using the L_2,1-norm to constrain the output weight matrix, we tend to obtain a sparse output weight matrix to construct a simpler single hidden layer feedforward neural network model. By introducing the graph regularization to preserve the local structural information of the data, the classification performance of the new method is further improved. Besides, we design an iterative optimization method based on the idea of half quadratic optimization to solve the non-convex problem of CSRGELM.

Conclusions

The classification results on the benchmark dataset show that CSRGELM can obtain better classification results compared with other methods. More importantly, we also apply the new method to the classification problems of cancer samples and get a good classification effect.