Comments on ;Bayes statistical behavior and valid generalization of pattern classifying neural networks' [with reply]

Higher-order probabilistic perceptrons as Bayesian inference engines

An explicit structural connection is established between the Bayes optimal classifier operating on K binary input variables and a corresponding two-layer perceptron having normalized output activities and couplings from input to output units of all orders up to K. With suitable modification of connection weights and biases, such a higher-order probabilistic perceptron should in principle be able to learn the statistics of the classification problem and match the a posteriori probabilities given by Bayes optimal inference. Specific training algorithms are developed that allow this goal to be approximated in a controlled variational sense. An application to the task of discriminating between stable and unstable nuclides in nuclear physics yields network models with predictive performance comparable to the best that has been achieved with conventional multilayer perceptrons containing only pairwise connections.

A neural-network approach to nonparametric and robust classification procedures

In this paper algorithms of neural-network type are introduced for solving estimation and classification problems when assumptions about independence, Gaussianity, and stationarity of the observation samples are no longer valid. Specifically, the asymptotic normality of several nonparametric classification tests is demonstrated and their implementation using a neural-network approach is presented. Initially, the neural nets train themselves via learning samples for nominal noise and alternative hypotheses distributions resulting in near optimum performance in a particular stochastic environment. In other than the nominal environments, however, high efficiency is maintained by adapting the optimum nonlinearities to changing conditions during operation via parallel networks, without disturbing the classification process. Furthermore, the superiority in performance of the proposed networks over more traditional neural nets is demonstrated in an application involving pattern recognition.

New nonleast-squares neural network learning algorithms for hypothesis testing

Hypothesis testing is a collective name for problems such as classification, detection, and pattern recognition. In this paper we propose two new classes of supervised learning algorithms for feedforward, binary-output neural network structures whose objective is hypothesis testing. All the algorithms are applications of stochastic approximation and are guaranteed to provide optimization with probability one. The first class of algorithms follows the Neyman-Pearson approach and maximizes the probability of detection, subject to a given false alarm constraint. These algorithms produce layer-by-layer optimal Neyman-Pearson designs. The second class of algorithms minimizes the probability of error and leads to layer-by-layer Bayes optimal designs. Deviating from the layer-by-layer optimization assumption, we propose more powerful learning techniques which unify, in some sense, the already existing algorithms. The proposed algorithms were implemented and tested on a simulated hypothesis testing problem. Backpropagation and perceptron learning were also included in the comparisons.