Laplace mixture of linear experts

Component selection for exponential power mixture models

Exponential Power (EP) family is a much flexible distribution family including Gaussian family as a sub-family. In this article, we study component selection and estimation for EP mixture models and regressions. The assumption on zero component mean in [X. Cao, Q. Zhao, D. Meng, Y. Chen, and Z. Xu, Robust low-rank matrix factorization under general mixture noise distributions, IEEE. Trans. Image. Process. 25 (2016), pp. 4677-4690.] is relaxed. To select components and estimate parameters simultaneously, we propose a penalized likelihood method, which can shrink mixing proportions to zero to achieve components selection. Modified EM algorithms are proposed, and the consistency of estimated component number is obtained. Simulation studies show the advantages of the proposed methods on accuracies of component number selection, parameter estimation, and density estimation. Analysis of value at risk of SHIBOR and a climate change data are given as illustration.

Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models

Mixture of experts (MoE) models are widely applied for conditional probability density estimation problems. We demonstrate the richness of the class of MoE models by proving denseness results in Lebesgue spaces, when inputs and outputs variables are both compactly supported. We further prove an almost uniform convergence result when the input is univariate. Auxiliary lemmas are proved regarding the richness of the soft-max gating function class, and their relationships to the class of Gaussian gating functions.

A Universal Approximation Theorem for Mixture-of-Experts Models

The mixture-of-experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from a Sobolev space that is sufficiently differentiable and that the domain of estimation is a compact unit hypercube. We provide an alternative result, which shows that the class of MoE mean functions is dense in the class of all continuous functions over arbitrary compact domains of estimation. Our result can be viewed as a universal approximation theorem for MoE models. The theorem we present allows MoE users to be confident in applying such models for estimation when data arise from nonlinear and nondifferentiable generative processes.

Robust mixture of experts modeling using the t distribution

Mixture of Experts (MoE) is a popular framework for modeling heterogeneity in data for regression, classification, and clustering. For regression and cluster analyses of continuous data, MoE usually uses normal experts following the Gaussian distribution. However, for a set of data containing a group or groups of observations with heavy tails or atypical observations, the use of normal experts is unsuitable and can unduly affect the fit of the MoE model. We introduce a robust MoE modeling using the t distribution. The proposed t MoE (TMoE) deals with these issues regarding heavy-tailed and noisy data. We develop a dedicated expectation-maximization (EM) algorithm to estimate the parameters of the proposed model by monotonically maximizing the observed data log-likelihood. We describe how the presented model can be used in prediction and in model-based clustering of regression data. The proposed model is validated on numerical experiments carried out on simulated data, which show the effectiveness and the robustness of the proposed model in terms of modeling non-linear regression functions as well as in model-based clustering. Then, it is applied to the real-world data of tone perception for musical data analysis, and the one of temperature anomalies for the analysis of climate change data. The obtained results show the usefulness of the TMoE model for practical applications.