Nonparametric Density Estimation Based on Self-Organizing Incremental Neural Network for Large Noisy Data

Learning Generative Models Using Denoising Density Estimators

Learning probabilistic models that can estimate the density of a given set of samples, and generate samples from that density, is one of the fundamental challenges in unsupervised machine learning. We introduce a new generative model based on denoising density estimators (DDEs), which are scalar functions parametrized by neural networks, that are efficiently trained to represent kernel density estimators of the data. Leveraging DDEs, our main contribution is a novel technique to obtain generative models by minimizing the Kullback-Leibler (KL)-divergence directly. We prove that our algorithm for obtaining generative models is guaranteed to converge consistently to the correct solution. Our approach does not require specific network architecture as in normalizing flows (NFs), nor use ordinary differential equation (ODE) solvers as in continuous NFs. Experimental results demonstrate substantial improvement in density estimation and competitive performance in generative model training.

Multivariate Density Estimation by Neural Networks

We propose nonparametric methods to obtain the Probability Density Function (PDF) to assess the properties of the underlying data generating process (DGP) without imposing any assumptions on the DGP, using neural networks (NNs). The proposed NN has advantages compared to well-known parametric and nonparametric density estimators. Our approach builds on literature on cumulative distribution function (CDF) estimation using NN. We extend this literature by providing analytical derivatives of this obtained CDF. Our approach hence removes the numerical approximation error in differentiating the CDF output, leading to more accurate PDF estimates. The proposed solution applies to any NN model, i.e., for any number of hidden layers or hidden neurons in the multilayer perceptron (MLP) structure. The proposed solution applies the PDF estimation by NN to continuous distributions as well as discrete distributions. We also show that the proposed solution to obtain the PDF leads to good approximations when applied to correlated variables in a multivariate setting. We test the performance of our method in a large Monte Carlo simulation using various complex distributions. Subsequently, we apply our method to estimate the density of the number of vehicle counts per minute measured with road sensors for a time window of 24 h.

Optimal modeling of anti-breast cancer candidate drugs screening based on multi-model ensemble learning with imbalanced data

The imbalanced data makes the machine learning model seriously biased, which leads to false positive in screening of therapeutic drugs for breast cancer. In order to deal with this problem, a multi-model ensemble framework based on tree-model, linear model and deep-learning model is proposed. Based on the methodology constructed in this study, we screened the 20 most critical molecular descriptors from 729 molecular descriptors of 1974 anti-breast cancer drug candidates and, in order to measure the pharmacokinetic properties and safety of the drug candidates, the screened molecular descriptors were used in this study for subsequent bioactivity, absorption, distribution metabolism, excretion, toxicity, and other prediction tasks. The results show that the method constructed in this study is superior and more stable than the individual models used in the ensemble approach.

Memory-Efficient Class-Incremental Learning for Image Classification

With the memory-resource-limited constraints, class-incremental learning (CIL) usually suffers from the "catastrophic forgetting" problem when updating the joint classification model on the arrival of newly added classes. To cope with the forgetting problem, many CIL methods transfer the knowledge of old classes by preserving some exemplar samples into the size-constrained memory buffer. To utilize the memory buffer more efficiently, we propose to keep more auxiliary low-fidelity exemplar samples, rather than the original real-high-fidelity exemplar samples. Such a memory-efficient exemplar preserving scheme makes the old-class knowledge transfer more effective. However, the low-fidelity exemplar samples are often distributed in a different domain away from that of the original exemplar samples, that is, a domain shift. To alleviate this problem, we propose a duplet learning scheme that seeks to construct domain-compatible feature extractors and classifiers, which greatly narrows down the above domain gap. As a result, these low-fidelity auxiliary exemplar samples have the ability to moderately replace the original exemplar samples with a lower memory cost. In addition, we present a robust classifier adaptation scheme, which further refines the biased classifier (learned with the samples containing distillation label knowledge about old classes) with the help of the samples of pure true class labels. Experimental results demonstrate the effectiveness of this work against the state-of-the-art approaches. We will release the code, baselines, and training statistics for all models to facilitate future research.

Artificial Evolution Network: A Computational Perspective on the Expansibility of the Nervous System

Neurobiologists recently found the brain can use sudden emerged channels to process information. Based on this finding, we put forward a question whether we can build a computation model that is able to integrate a sudden emerged new type of perceptual channel into itself in an online way. If such a computation model can be established, it will introduce a channel-free property to the computation model and meanwhile deepen our understanding about the extendibility of the brain. In this article, a biologically inspired neural network named artificial evolution (AE) network is proposed to handle the problem. When a new perceptual channel emerges, the neurons in the network can grow new connections to connect the emerged channel according to the Hebb rule. In this article, we design a sensory channel expansion experiment to test the AE network. The experimental results demonstrate that the AE network can handle the sudden emerged perceptual channels effectively.

One-dimensional convolutional neural network (1D-CNN) image reconstruction for electrical impedance tomography

In recent years, due to the strong autonomous learning ability of neural network algorithms, they have been applied for electrical impedance tomography (EIT). Although their imaging accuracy is greatly improved compared with traditional algorithms, generalization for both simulation and experimental data is required to be improved. According to the characteristics of voltage data collected in EIT, a one-dimensional convolutional neural network (1D-CNN) is proposed to solve the inverse problem of image reconstruction. Abundant samples are generated with numerical simulation to improve the edge-preservation of reconstructed images. The TensorFlow-graphics processing unit environment and Adam optimizer are used to train and optimize the network, respectively. The reconstruction results of the new network are compared with the Deep Neural Network (DNN) and 2D-CNN to prove the effectiveness and edge-preservation. The anti-noise and generalization capabilities of the new network are also validated. Furthermore, experiments with the EIT system are carried out to verify the practicability of the new network. The average image correlation coefficient of the new network increases 0.0320 and 0.0616 compared with the DNN and 2D-CNN, respectively, which demonstrates that the proposed method could give better reconstruction results, especially for the distribution of complex geometries.

Online Topology Learning by a Gaussian Membership-Based Self-Organizing Incremental Neural Network

In order to extract useful information from data streams, incremental learning has been introduced in more and more data mining algorithms. For instance, a self-organizing incremental neural network (SOINN) has been proposed to extract a topological structure that consists of one or more neural networks to closely reflect the data distribution of data streams. However, SOINN has the tradeoff between deleting previously learned nodes and inserting new nodes, i.e., the stability-plasticity dilemma. Therefore, it is not guaranteed that the topological structure obtained by the SOINN will closely represent data distribution. For solving the stability-plasticity dilemma, we propose a Gaussian membership-based SOINN (Gm-SOINN). Unlike other SOINN-based methods that allow only one node to be identified as a "winner" (the nearest node), the Gm-SOINN uses a Gaussian membership to indicate to which degree the node is a winner. Hence, the Gm-SOINN avoids the topological structure that cannot represent the data distribution because previously learned nodes overly deleted or noisy nodes inserted. In addition, an evolving Gaussian mixture model is integrated into the Gm-SOINN to estimate the density distribution of nodes, thereby avoiding the wrong connection between two nodes. Experiments involving both artificial and real-world data sets indicate that our proposed Gm-SOINN achieves better performance than other topology learning methods.

On the Parzen Kernel-Based Probability Density Function Learning Procedures Over Time-Varying Streaming Data With Applications to Pattern Classification

In this paper, we propose a recursive variant of the Parzen kernel density estimator (KDE) to track changes of dynamic density over data streams in a nonstationary environment. In stationary environments, well-established traditional KDE techniques have nice asymptotic properties. Their existing extensions to deal with stream data are mostly based on various heuristic concepts (losing convergence properties). In this paper, we study recursive KDEs, called recursive concept drift tracking KDEs, and prove their weak (in probability) and strong (with probability one) convergence, resulting in perfect tracking properties as the sample size approaches infinity. In three theorems and subsequent examples, we show how to choose the bandwidth and learning rate of a recursive KDE in order to ensure weak and strong convergence. The simulation results illustrate the effectiveness of our algorithm both for density estimation and classification over time-varying stream data.

A Kernel Bayesian Adaptive Resonance Theory with A Topological Structure

This paper attempts to solve the typical problems of self-organizing growing network models, i.e. (a) an influence of the order of input data on the self-organizing ability, (b) an instability to high-dimensional data and an excessive sensitivity to noise, and (c) an expensive computational cost by integrating Kernel Bayes Rule (KBR) and Correntropy-Induced Metric (CIM) into Adaptive Resonance Theory (ART) framework. KBR performs a covariance-free Bayesian computation which is able to maintain a fast and stable computation. CIM is a generalized similarity measurement which can maintain a high-noise reduction ability even in a high-dimensional space. In addition, a Growing Neural Gas (GNG)-based topology construction process is integrated into the ART framework to enhance its self-organizing ability. The simulation experiments with synthetic and real-world datasets show that the proposed model has an outstanding stable self-organizing ability for various test environments.

Simultaneous Forecasting of Meteorological Data Based on a Self-Organizing Incremental Neural Network

In this study, we propose a simultaneous forecasting model for meteorological time-series data based on a self-organizing incremental neural network (SOINN). Meteorological parameters (i.e., temperature, wet bulb temperature, humidity, wind speed, atmospheric pressure, and total solar radiation on a horizontal surface) are considered as input data for the prediction of meteorological time-series information. Based on a SOINN within normalized-refined-meteorological data, proposed model succeeded forecasting temperature, humidity, wind speed and atmospheric pressure simultaneously. In addition, proposed model does not take more than 2 s in training half-year period and 15 s in testing half-year period. This paper also elucidates the SOINN and the algorithm of the learning process. The effectiveness of our model is established by comparison of our results with experimental results and with results obtained by another model. Three advantages of our model are also described. The obtained information can be effective in applications based on neural networks, and the proposed model for handling meteorological phenomena may be helpful for other studies worldwide including energy management system.

Online Density Estimation of Nonstationary Sources Using Exponential Family of Distributions

We investigate online probability density estimation (or learning) of nonstationary (and memoryless) sources using exponential family of distributions. To this end, we introduce a truly sequential algorithm that achieves Hannan-consistent log-loss regret performance against true probability distribution without requiring any information about the observation sequence (e.g., the time horizon $T$ and the drift of the underlying distribution $C$ ) to optimize its parameters. Our results are guaranteed to hold in an individual sequence manner. Our log-loss performance with respect to the true probability density has regret bounds of $O(({CT})^{1/2})$ , where $C$ is the total change (drift) in the natural parameters of the underlying distribution. To achieve this, we design a variety of probability density estimators with exponentially quantized learning rates and merge them with a mixture-of-experts notion. Hence, we achieve this square-root regret with computational complexity only logarithmic in the time horizon. Thus, our algorithm can be efficiently used in big data applications. Apart from the regret bounds, through synthetic and real-life experiments, we demonstrate substantial performance gains with respect to the state-of-the-art probability density estimation algorithms in the literature.