Robust echo state networks based on correntropy induced loss function

Broad learning system based on maximum multi-kernel correntropy criterion

The broad learning system (BLS) is an effective machine learning model that exhibits excellent feature extraction ability and fast training speed. However, the traditional BLS is derived from the minimum mean square error (MMSE) criterion, which is highly sensitive to non-Gaussian noise. In order to enhance the robustness of BLS, this paper reconstructs the objective function of BLS based on the maximum multi-kernel correntropy criterion (MMKCC), and obtains a new robust variant of BLS (MKC-BLS). For the multitude of parameters involved in MMKCC, an effective parameter optimization method is presented. The fixed-point iteration method is employed to further optimize the model, and a reliable convergence proof is provided. In comparison to the existing robust variants of BLS, MKC-BLS exhibits superior performance in the non-Gaussian noise environment, particularly in the multi-modal noise environment. Experiments on multiple public datasets and real application validate the efficacy of the proposed method.

Broad Learning System Based on Maximum Correntropy Criterion

As an effective and efficient discriminative learning method, broad learning system (BLS) has received increasing attention due to its outstanding performance in various regression and classification problems. However, the standard BLS is derived under the minimum mean square error (MMSE) criterion, which is, of course, not always a good choice due to its sensitivity to outliers. To enhance the robustness of BLS, we propose in this work to adopt the maximum correntropy criterion (MCC) to train the output weights, obtaining a correntropy-based BLS (C-BLS). Due to the inherent superiorities of MCC, the proposed C-BLS is expected to achieve excellent robustness to outliers while maintaining the original performance of the standard BLS in the Gaussian or noise-free environment. In addition, three alternative incremental learning algorithms, derived from a weighted regularized least-squares solution rather than pseudoinverse formula, for C-BLS are developed. With the incremental learning algorithms, the system can be updated quickly without the entire retraining process from the beginning when some new samples arrive or the network deems to be expanded. Experiments on various regression and classification data sets are reported to demonstrate the desirable performance of the new methods.

Quantized Minimum Error Entropy Criterion

Comparing with traditional learning criteria, such as mean square error, the minimum error entropy (MEE) criterion is superior in nonlinear and non-Gaussian signal processing and machine learning. The argument of the logarithm in Renyi's entropy estimator, called information potential (IP), is a popular MEE cost in information theoretic learning. The computational complexity of IP is, however, quadratic in terms of sample number due to double summation. This creates the computational bottlenecks, especially for large-scale data sets. To address this problem, in this paper, we propose an efficient quantization approach to reduce the computational burden of IP, which decreases the complexity from O(N²) to O(MN) with M << N . The new learning criterion is called the quantized MEE (QMEE). Some basic properties of QMEE are presented. Illustrative examples with linear-in-parameter models are provided to verify the excellent performance of QMEE.