Deep Learning in High Voltage Engineering: A Literature Review

Condition monitoring of high voltage apparatus is of much importance for the maintenance of electric power systems. Whether it is detecting faults or partial discharges that take place in high voltage equipment, or detecting contamination and degradation of outdoor insulators, deep learning which is a branch of machine learning has been extensively investigated. Instead of using hand-crafted manual features as an input for the traditional machine learning algorithms, deep learning algorithms use raw data as the input where the feature extraction stage is integrated in the learning stage, resulting in a more automated process. This is the main advantage of using deep learning instead of traditional machine learning techniques. This paper presents a review of the recent literature on the application of deep learning techniques in monitoring high voltage apparatus such as GIS, transformers, cables, rotating machines, and outdoor insulators.

Asymptotic Convergence of Soft-Constrained Neural Networks for Density Estimation

A soft-constrained neural network for density estimation (SC-NN-4pdf) has recently been introduced to tackle the issues arising from the application of neural networks to density estimation problems (in particular, the satisfaction of the second Kolmogorov axiom). Although the SC-NN-4pdf has been shown to outperform parametric and non-parametric approaches (from both the machine learning and the statistics areas) over a variety of univariate and multivariate density estimation tasks, no clear rationale behind its performance has been put forward so far. Neither has there been any analysis of the fundamental theoretical properties of the SC-NN-4pdf. This paper narrows the gaps, delivering a formal statement of the class of density functions that can be modeled to any degree of precision by SC-NN-4pdfs, as well as a proof of asymptotic convergence in probability of the SC-NN-4pdf training algorithm under mild conditions for a popular class of neural architectures. These properties of the SC-NN-4pdf lay the groundwork for understanding the strong estimation capabilities that SC-NN-4pdfs have only exhibited empirically so far.

The Forbidden Region Self-Organizing Map Neural Network

Self-organizing maps (SOMs) are aimed to learn a representation of the input distribution which faithfully describes the topological relations among the clusters of the distribution. For some data sets and applications, it is known beforehand that some regions of the input space cannot contain any samples. Those are known as forbidden regions. In these cases, any prototype which lies in a forbidden region is meaningless. However, previous self-organizing models do not address this problem. In this paper, we propose a new SOM model which is guaranteed to keep all prototypes out of a set of prespecified forbidden regions. Experimental results are reported, which show that our proposal outperforms the SOM both in terms of vector quantization error and quality of the learned topological maps.

Multi-Step Time Series Forecasting with an Ensemble of Varied Length Mixture Models

Many real-world problems require modeling and forecasting of time series, such as weather temperature, electricity demand, stock prices and foreign exchange (FX) rates. Often, the tasks involve predicting over a long-term period, e.g. several weeks or months. Most existing time series models are inheritably for one-step prediction, that is, predicting one time point ahead. Multi-step or long-term prediction is difficult and challenging due to the lack of information and uncertainty or error accumulation. The main existing approaches, iterative and independent, either use one-step model recursively or treat the multi-step task as an independent model. They generally perform poorly in practical applications. In this paper, as an extension of the self-organizing mixture autoregressive (AR) model, the varied length mixture (VLM) models are proposed to model and forecast time series over multi-steps. The key idea is to preserve the dependencies between the time points within the prediction horizon. Training data are segmented to various lengths corresponding to various forecasting horizons, and the VLM models are trained in a self-organizing fashion on these segments to capture these dependencies in its component AR models of various predicting horizons. The VLM models form a probabilistic mixture of these varied length models. A combination of short and long VLM models and an ensemble of them are proposed to further enhance the prediction performance. The effectiveness of the proposed methods and their marked improvements over the existing methods are demonstrated through a number of experiments on synthetic data, real-world FX rates and weather temperatures.

Parzen neural networks: Fundamentals, properties, and an application to forensic anthropology

A novel, unsupervised nonparametric model of multivariate probability density functions (pdf) is introduced, namely the Parzen neural network (PNN). The PNN is intended to overcome the major limitations of traditional (either statistical or neural) pdf estimation techniques. Besides being profitably simple, the PNN turns out to have nice properties in terms of unbiased modeling capability, asymptotic convergence, and efficiency at test time. Several matters pertaining the practical application of the PNN are faced in the paper, too. Experiments are reported, involving (i) synthetic datasets, and (ii) a challenging sex determination task from 1400 scout-view CT-scan images of human crania. Incidentally, the empirical evidence entails also some conclusions of high anthropological relevance.

Hazard Detection for Motorcycles via Accelerometers: A Self-Organizing Map Approach

This paper deals with collision and hazard detection for motorcycles via inertial measurements. For this kind of vehicles, the most difficult challenge is to distinguish road's anomalies from real hazards. This is usually done by setting absolute thresholds on the accelerometer measurements. These thresholds are heuristically tuned from expensive crash tests. This empirical method is expensive and not intuitive when the number of signals to deal with grows. We propose a method based on self-organized neural networks that can deal with a large number of inputs from different types of sensors. The method uses accelerometer and gyro measurements. The proposed approach is capable of recognizing dangerous conditions although no crash test is needed for training. The method is tested in a simulation environment; the comparison with a benchmark method shows the advantages of the proposed approach.

Sparse Density Estimation on the Multinomial Manifold

A new sparse kernel density estimator is introduced based on the minimum integrated square error criterion for the finite mixture model. Since the constraint on the mixing coefficients of the finite mixture model is on the multinomial manifold, we use the well-known Riemannian trust-region (RTR) algorithm for solving this problem. The first- and second-order Riemannian geometry of the multinomial manifold are derived and utilized in the RTR algorithm. Numerical examples are employed to demonstrate that the proposed approach is effective in constructing sparse kernel density estimators with an accuracy competitive with those of existing kernel density estimators.

Hierarchical max-flow segmentation framework for multi-atlas segmentation with Kohonen self-organizing map based Gaussian mixture modeling

The incorporation of intensity, spatial, and topological information into large-scale multi-region segmentation has been a topic of ongoing research in medical image analysis. Multi-region segmentation problems, such as segmentation of brain structures, pose unique challenges in image segmentation in which regions may not have a defined intensity, spatial, or topological distinction, but rely on a combination of the three. We propose a novel framework within the Advanced segmentation tools (ASETS)(2), which combines large-scale Gaussian mixture models trained via Kohonen self-organizing maps, with deformable registration, and a convex max-flow optimization algorithm incorporating region topology as a hierarchy or tree. Our framework is validated on two publicly available neuroimaging datasets, the OASIS and MRBrainS13 databases, against the more conventional Potts model, achieving more accurate segmentations. Each component is accelerated using general-purpose programming on graphics processing Units to ensure computational feasibility.

Confabulation-inspired association rule mining for rare and frequent itemsets

A new confabulation-inspired association rule mining (CARM) algorithm is proposed using an interestingness measure inspired by cogency. Cogency is only computed based on pairwise item conditional probability, so the proposed algorithm mines association rules by only one pass through the file. The proposed algorithm is also more efficient for dealing with infrequent items due to its cogency-inspired approach. The problem of associative classification is used here for evaluating the proposed algorithm. We evaluate CARM over both synthetic and real benchmark data sets obtained from the UC Irvine machine learning repository. Experiments show that the proposed algorithm is consistently faster due to its one time file access and consumes less memory space than the Conditional Frequent Patterns growth algorithm. In addition, statistical analysis reveals the superiority of the approach for classifying minority classes in unbalanced data sets.

Bregman divergences for growing hierarchical self-organizing networks

Growing hierarchical self-organizing models are characterized by the flexibility of their structure, which can easily accommodate for complex input datasets. However, most proposals use the Euclidean distance as the only error measure. Here we propose a way to introduce Bregman divergences in these models, which is based on stochastic approximation principles, so that more general distortion measures can be employed. A procedure is derived to compare the performance of networks using different divergences. Moreover, a probabilistic interpretation of the model is provided, which enables its use as a Bayesian classifier. Experimental results are presented for classification and data visualization applications, which show the advantages of these divergences with respect to the classical Euclidean distance.

SOMKE: kernel density estimation over data streams by sequences of self-organizing maps

In this paper, we propose a novel method SOMKE, for kernel density estimation (KDE) over data streams based on sequences of self-organizing map (SOM). In many stream data mining applications, the traditional KDE methods are infeasible because of the high computational cost, processing time, and memory requirement. To reduce the time and space complexity, we propose a SOM structure in this paper to obtain well-defined data clusters to estimate the underlying probability distributions of incoming data streams. The main idea of this paper is to build a series of SOMs over the data streams via two operations, that is, creating and merging the SOM sequences. The creation phase produces the SOM sequence entries for windows of the data, which obtains clustering information of the incoming data streams. The size of the SOM sequences can be further reduced by combining the consecutive entries in the sequence based on the measure of Kullback-Leibler divergence. Finally, the probability density functions over arbitrary time periods along the data streams can be estimated using such SOM sequences. We compare SOMKE with two other KDE methods for data streams, the M-kernel approach and the cluster kernel approach, in terms of accuracy and processing time for various stationary data streams. Furthermore, we also investigate the use of SOMKE over nonstationary (evolving) data streams, including a synthetic nonstationary data stream, a real-world financial data stream and a group of network traffic data streams. The simulation results illustrate the effectiveness and efficiency of the proposed approach.

FOREGROUND DETECTION IN VIDEO SEQUENCES WITH PROBABILISTIC SELF-ORGANIZING MAPS

Background modeling and foreground detection are key parts of any computer vision system. These problems have been addressed in literature with several probabilistic approaches based on mixture models. Here we propose a new kind of probabilistic background models which is based on probabilistic self-organising maps. This way, the background pixels are modeled with more flexibility. On the other hand, a statistical correlation measure is used to test the similarity among nearby pixels, so as to enhance the detection performance by providing a feedback to the process. Several well known benchmark videos have been used to assess the relative performance of our proposal with respect to traditional neural and non neural based methods, with favourable results, both qualitatively and quantitatively. A statistical analysis of the differences among methods demonstrates that our method is significantly better than its competitors. This way, a strong alternative to classical methods is presented.

Topographic map formation of factorized Edgeworth-expanded kernels

We introduce a new learning algorithm for topographic map formation of Edgeworth-expanded Gaussian activation kernels. In order to avoid the rapid increase in kernel parameters, as the problem dimensionality increases, we factorize the kernels using a linear ICA algorithm. We apply the algorithm to a number of real-world cases, and show the advantage of the Edgeworth-expanded kernels in clustering.

On the equivalence between kernel self-organising maps and self-organising mixture density networks

The kernel method has become a useful trick and has been widely applied to various learning models to extend their nonlinear approximation and classification capabilities. Such extensions have also recently occurred to the Self-Organising Map (SOM). In this paper, two recently proposed kernel SOMs are reviewed, together with their link to an energy function. The Self-Organising Mixture Network is an extension of the SOM for mixture density modelling. This paper shows that with an isotropic, density-type kernel function, the kernel SOM is equivalent to a homoscedastic Self-Organising Mixture Network, an entropy-based density estimator. This revelation on the one hand explains that kernelising SOM can improve classification performance by acquiring better probability models of the data; but on the other hand it also explains that the SOM already naturally approximates the kernel method.

Connectionist-based Dempster-Shafer evidential reasoning for data fusion

Dempster-Shafer evidence theory (DSET) is a popular paradigm for dealing with uncertainty and imprecision. Its corresponding evidential reasoning framework is theoretically attractive. However, there are outstanding issues that hinder its use in real-life applications. Two prominent issues in this regard are 1) the issue of basic probability assignments (masses) and 2) the issue of dependence among information sources. This paper attempts to deal with these issues by utilizing neural networks in the context of pattern classification application. First, a multilayer perceptron neural network with the mean squared error as a cost function is implemented to calculate, for each information source, posteriori probabilities for all classes. Second, an evidence structure construction scheme is developed for transferring the estimated posteriori probabilities to a set of masses along with the corresponding focal elements, from a Bayesian decision point of view. Third, a network realization of the Dempster-Shafer evidential reasoning is designed and analyzed, and it is further extended to a DSET-based neural network, referred to as DSETNN, to manipulate the evidence structures. In order to tackle the issue of dependence between sources, DSETNN is tuned for optimal performance through a supervised learning process. To demonstrate the effectiveness of the proposed approach, we apply it to three benchmark pattern classification problems. Experiments reveal that the DSETNN out-performs DSET and provide encouraging results in terms of classification accuracy and the speed of learning convergence.

Self-Organizing Topological Tree for Online Vector Quantization and Data Clustering

The self-organizing Maps (SOM) introduced by Kohonen implement two important operations: vector quantization (VQ) and a topology-preserving mapping. In this paper, an online self-organizing topological tree (SOTT) with faster learning is proposed. A new learning rule delivers the efficiency and topology preservation, which is superior of other structures of SOMs. The computational complexity of the proposed SOTT is O(log N) rather than O(N) as for the basic SOM. The experimental results demonstrate that the reconstruction performance of SOTT is comparable to the full-search SOM and its computation time is much shorter than the full-search SOM and other vector quantizers. In addition, SOTT delivers the hierarchical mapping of codevectors and the progressive transmission and decoding property, which are rarely supported by other vector quantizers at the same time. To circumvent the shortcomings of clustering performance of classical partition clustering algorithms, a hybrid clustering algorithm that fully exploit the online learning and multiresolution characteristics of SOTT is devised. A new linkage metric is proposed which can be updated online to accelerate the time consuming agglomerative hierarchical clustering stage. Besides the enhanced clustering performance, due to the online learning capability, the memory requirement of the proposed SOTT hybrid clustering algorithm is independent of the size of the data set, making it attractive for large database.

Maximum Likelihood Topographic Map Formation

We introduce a new unsupervised learning algorithm for kernel-based topographic map formation of heteroscedastic gaussian mixtures that allows for a unified account of distortion error (vector quantization), log-likelihood, and Kullback-Leibler divergence.

Entropy-Based Kernel Mixture Modeling for Topographic Map Formation

A new information-theoretic learning algorithm for kernel-based topographic map formation is introduced. In the one-dimensional case, the algorithm is aimed at uniformizing the cumulative distribution of the kernel mixture densities by maximizing its differential entropy. A nonparametric differential entropy estimator is used on which normalized gradient ascent is performed. Both differentiable and nondifferentiable kernels are in principle supported, such as Gaussian and rectangular (on/off) kernels. The relation is shown with joint entropy maximization of the kernel outputs. The learning algorithm's performance is assessed and compared with the theoretically optimal performance. A fixed-point rule is derived for the case of heterogeneous kernel mixtures. Finally, an extension of the algorithm to the multidimensional case is suggested.

Kernel-based topographic map formation achieved with an information-theoretic approach

A new information-theoretic learning algorithm is introduced for kernel-based topographic map formation. The kernels are allowed to overlap and move freely in the input space, and to have differing kernel ranges. We start with Linsker's infomax principle and observe that it cannot be readily extended to our case, exactly due to the presence of kernels. We then consider Bell and Sejnowski's generalization of Linsker's infomax principle, which suggests differential entropy maximization, and add a second component to be optimized, namely, mutual information minimization between the kernel outputs, in order to take into account the kernel overlap, and thus the topographic map's output redundancy. The result is joint entropy maximization of the kernel outputs, which we adopt as our learning criterion. We derive a learning algorithm and verify its performance both for a synthetic example, for which the optimal result can be derived analytically, and for a classic real-world example.

Joint entropy maximization in kernel-based topographic maps

A new learning algorithm for kernel-based topographic map formation is introduced. The kernel parameters are adjusted individually so as to maximize the joint entropy of the kernel outputs. This is done by maximizing the differential entropies of the individual kernel outputs, given that the map's output redundancy, due to the kernel overlap, needs to be minimized. The latter is achieved by minimizing the mutual information between the kernel outputs. As a kernel, the (radial) incomplete gamma distribution is taken since, for a gaussian input density, the differential entropy of the kernel output will be maximal. Since the theoretically optimal joint entropy performance can be derived for the case of nonoverlapping gaussian mixture densities, a new clustering algorithm is suggested that uses this optimum as its "null" distribution. Finally, it is shown that the learning algorithm is similar to one that performs stochastic gradient descent on the Kullback-Leibler divergence for a heteroskedastic gaussian mixture density model.

Kernel-based topographic map formation by local density modeling

We introduce a new learning algorithm for kernel-based topographic map formation. The algorithm generates a gaussian mixture density model by individually adapting the gaussian kernels' centers and radii to the assumed gaussian local input densities.