Insights Into the Robustness of Minimum Error Entropy Estimation

Complex quantized minimum error entropy with fiducial points: theory and application in model regression

Minimum error entropy with fiducial points (MEEF) has gained significant attention due to its excellent performance in mitigating the adverse effects of non-Gaussian noise in the fields of machine learning and signal processing. However, the original MEEF algorithm suffers from high computational complexity due to the double summation of error samples. The quantized MEEF (QMEEF), proposed by Zheng et al. alleviates this computational burden through strategic quantization techniques, providing a more efficient solution. In this paper, we extend the application of these techniques to the complex domain, introducing complex QMEEF (CQMEEF). We theoretically introduce and prove the fundamental properties and convergence of CQMEEF. Furthermore, we apply this novel method to the training of a range of Linear-in-parameters (LIP) models, demonstrating its broad applicability. Experimental results show that CQMEEF achieves high precision in regression tasks involving various noise-corrupted datasets, exhibiting effectiveness under unfavorable conditions, and surpassing existing methods across critical performance metrics. Consequently, CQMEEF not only offers an efficient computational alternative but also opens up new avenues for dealing with complex data in regression tasks.

Computationally efficient robust adaptive filtering algorithm based on improved minimum error entropy criterion with fiducial points

Recently, there has been a strong interest in the minimum error entropy (MEE) criterion derived from information theoretic learning, which is effective in dealing with the multimodal non-Gaussian noise case. However, the kernel function is shift invariant resulting in the MEE criterion being insensitive to the error location. An existing solution is to combine the maximum correntropy (MC) with MEE criteria, leading to the MEE criterion with fiducial points (MEEF). Nevertheless, the algorithms based on the MEEF criterion usually require higher computational complexity. To remedy this problem, an improved MEEF (IMEEF) criterion is devised, aiming to avoid repetitive calculations of the aposteriori error, and an adaptive filtering algorithm based on gradient descent (GD) method is proposed, namely, GD-based IMEEF (IMEEF-GD) algorithm. In addition, we provide the convergence condition in terms of mean sense, along with an analysis of the steady-state and transient behaviors of IMEEF-GD in the mean-square sense. Its computational complexity is also analyzed. Simulation results demonstrate that the computational requirement of our algorithm does not vary significantly with the error sample number and the derived theoretical model is highly consistent with the learning curve. Ultimately, we employ the IMEEF-GD algorithm in tasks such as system identification, wind signal magnitude prediction, temperature prediction, and acoustic echo cancellation (AEC) to validate the effectiveness of the IMEEF-GD algorithm.

Error Loss Networks

A novel model called error loss network (ELN) is proposed to build an error loss function for supervised learning. The ELN is similar in structure to a radial basis function (RBF) neural network, but its input is an error sample and output is a loss corresponding to that error sample. That means the nonlinear input-output mapper of the ELN creates an error loss function. The proposed ELN provides a unified model for a large class of error loss functions, which includes some information-theoretic learning (ITL) loss functions as special cases. The activation function, weight parameters, and network size of the ELN can be predetermined or learned from the error samples. On this basis, we propose a new machine learning paradigm where the learning process is divided into two stages: first, learning a loss function using an ELN; second, using the learned loss function to continue to perform the learning. Experimental results are presented to demonstrate the desirable performance of the new method.

Quantized minimum error entropy with fiducial points for robust regression

Minimum error entropy with fiducial points (MEEF) has received a lot of attention, due to its outstanding performance to curb the negative influence caused by non-Gaussian noises in the fields of machine learning and signal processing. However, the estimate of the information potential of MEEF involves a double summation operator based on all available error samples, which can result in large computational burden in many practical scenarios. In this paper, an efficient quantization method is therefore adopted to represent the primary set of error samples with a smaller subset, generating a quantized MEEF (QMEEF). Some basic properties of QMEEF are presented and proved from theoretical perspectives. In addition, we have applied this new criterion to train a class of linear-in-parameters models, including the commonly used linear regression model, random vector functional link network, and broad learning system as special cases. Experimental results on various datasets are reported to demonstrate the desirable performance of the proposed methods to perform regression tasks with contaminated data.

Restricted Minimum Error Entropy Criterion for Robust Classification

The minimum error entropy (MEE) criterion is a powerful approach for non-Gaussian signal processing and robust machine learning. However, the instantiation of MEE on robust classification is a rather vacancy in the literature. The original MEE purely focuses on minimizing Renyi's quadratic entropy of the prediction errors, which could exhibit inferior capability in noisy classification tasks. To this end, we analyze the optimal error distribution with adverse outliers and introduce a specific codebook for restriction, which optimizes the error distribution toward the optimal case. Half-quadratic-based optimization and convergence analysis of the proposed learning criterion, called restricted MEE (RMEE), are provided. The experimental results considering logistic regression and extreme learning machine on synthetic data and UCI datasets, respectively, are presented to demonstrate the superior robustness of RMEE. Furthermore, we evaluate RMEE on a noisy electroencephalogram dataset, so as to strengthen its practical impact.

Robust Spike-Based Continual Meta-Learning Improved by Restricted Minimum Error Entropy Criterion

The spiking neural network (SNN) is regarded as a promising candidate to deal with the great challenges presented by current machine learning techniques, including the high energy consumption induced by deep neural networks. However, there is still a great gap between SNNs and the online meta-learning performance of artificial neural networks. Importantly, existing spike-based online meta-learning models do not target the robust learning based on spatio-temporal dynamics and superior machine learning theory. In this invited article, we propose a novel spike-based framework with minimum error entropy, called MeMEE, using the entropy theory to establish the gradient-based online meta-learning scheme in a recurrent SNN architecture. We examine the performance based on various types of tasks, including autonomous navigation and the working memory test. The experimental results show that the proposed MeMEE model can effectively improve the accuracy and the robustness of the spike-based meta-learning performance. More importantly, the proposed MeMEE model emphasizes the application of the modern information theoretic learning approach on the state-of-the-art spike-based learning algorithms. Therefore, in this invited paper, we provide new perspectives for further integration of advanced information theory in machine learning to improve the learning performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic systems.

A Fuzzy Logic Model for Hourly Electrical Power Demand Modeling

In this article, a fuzzy logic model is proposed for more precise hourly electrical power demand modeling in New England. The issue that exists when considering hourly electrical power demand modeling is that these types of plants have a large amount of data. In order to obtain a more precise model of plants with a large amount of data, the main characteristics of the proposed fuzzy logic model are as follows: (1) it is in accordance with the conditions under which a fuzzy logic model and a radial basis mapping model are equivalent to obtain a new scheme, (2) it uses a combination of the descending gradient and the mini-lots approach to avoid applying the descending gradient to all data.

State of Charge Estimation of Lithium Battery Based on Improved Correntropy Extended Kalman Filter

State of charge (SOC) estimation plays a crucial role in battery management systems. Among all the existing SOC estimation approaches, the model-driven extended Kalman filter (EKF) has been widely utilized to estimate SOC due to its simple implementation and nonlinear property. However, the traditional EKF derived from the mean square error (MSE) loss is sensitive to non-Gaussian noise which especially exists in practice, thus the SOC estimation based on the traditional EKF may result in undesirable performance. Hence, a novel robust EKF method with correntropy loss is employed to perform SOC estimation to improve the accuracy under non-Gaussian environments firstly. Secondly, a novel robust EKF, called C-WLS-EKF, is developed by combining the advantages of correntropy and weighted least squares (WLS) to improve the digital stability of the correntropy EKF (C-EKF). In addition, the convergence of the proposed algorithm is verified by the Cramér–Rao low bound. Finally, a C-WLS-EKF method based on an equivalent circuit model is designed to perform SOC estimation. The experiment results clarify that the SOC estimation error in terms of the MSE via the proposed C-WLS-EKF method can efficiently be reduced from 1.361% to 0.512% under non-Gaussian noise conditions.

Diffusion Logarithm-Correntropy Algorithm for Parameter Estimation in Non-Stationary Environments over Sensor Networks

This paper considers the parameter estimation problem under non-stationary environments in sensor networks. The unknown parameter vector is considered to be a time-varying sequence. To further promote estimation performance, this paper suggests a novel diffusion logarithm-correntropy algorithm for each node in the network. Such an algorithm can adopt both the logarithm operation and correntropy criterion to the estimation error. Moreover, if the error gets larger due to the non-stationary environments, the algorithm can respond immediately by taking relatively steeper steps. Thus, the proposed algorithm achieves smaller error in time. The tracking performance of the proposed logarithm-correntropy algorithm is analyzed. Finally, experiments verify the validity of the proposed algorithmic schemes, which are compared to other recent algorithms that have been proposed for parameter estimation.

Recognition of Traveling Surges in HVDC with Wavelet Entropy

Traveling surges are commonly adopted in protection devices of high-voltage direct current (HVDC) transmission systems. Lightning strikes also can produce large-amplitude traveling surges which lead to the malfunction of relays. To ensure the reliable operation of protection devices, recognition of traveling surges must be considered. Wavelet entropy, which can reveal time-frequency distribution features, is a potential tool for traveling surge recognition. In this paper, the effectiveness of wavelet entropy in characterizing traveling surges is demonstrated by comparing its representations of different kinds of surges and discussing its stability with the effects of propagation distance and fault resistance. A wavelet entropy-based recognition method is proposed and tested by simulated traveling surges. The results show wavelet entropy can discriminate fault traveling surges with a good recognition rate.

A Robust Diffusion Estimation Algorithm with Self-Adjusting Step-Size in WSNs

In wireless sensor networks (WSNs), each sensor node can estimate the global parameter from the local data in a distributed manner. This paper proposed a robust diffusion estimation algorithm based on a minimum error entropy criterion with a self-adjusting step-size, which are referred to as the diffusion MEE-SAS (DMEE-SAS) algorithm. The DMEE-SAS algorithm has a fast speed of convergence and is robust against non-Gaussian noise in the measurements. The detailed performance analysis of the DMEE-SAS algorithm is performed. By combining the DMEE-SAS algorithm with the diffusion minimum error entropy (DMEE) algorithm, an Improving DMEE-SAS algorithm is proposed for a non-stationary environment where tracking is very important. The Improving DMEE-SAS algorithm can avoid insensitivity of the DMEE-SAS algorithm due to the small effective step-size near the optimal estimator and obtain a fast convergence speed. Numerical simulations are given to verify the effectiveness and advantages of these proposed algorithms.