Neural networks for blind separation with unknown number of sources

Unsupervised learning in images and audio to produce neural receptive fields: a primer and accessible notebook

Sensory processing relies on efficient computation driven by a combination of low-level unsupervised, statistical structural learning, and high-level task-dependent learning. In the earliest stages of sensory processing, sparse and independent coding strategies are capable of modeling neural processing using the same coding strategy with only a change in the input (e.g., grayscale images, color images, and audio). We present a consolidated review of Independent Component Analysis (ICA) as an efficient neural coding scheme with the ability to model early visual and auditory neural processing. We created a self-contained, accessible Jupyter notebook using Python to demonstrate the efficient coding principle for different modalities following a consistent five-step strategy. For each modality, derived receptive field models from natural and non-natural inputs are contrasted, demonstrating how neural codes are not produced when the inputs sufficiently deviate from those animals were evolved to process. Additionally, the demonstration shows that ICA produces more neurally-appropriate receptive field models than those based on common compression strategies, such as Principal Component Analysis. The five-step strategy not only produces neural-like models but also promotes reuse of code to emphasize the input-agnostic nature where each modality can be modeled with only a change in inputs. This notebook can be used to readily observe the links between unsupervised machine learning strategies and early sensory neuroscience, improving our understanding of flexible data-driven neural development in nature and future applications.

New Families of Skewed Higher-Order Kernel Estimators to Solve the BSS/ICA Problem for Multimodal Sources Mixtures

This letter suggests two new types of asymmetrical higher-order kernels (HOK) that are generated using the orthogonal polynomials Laguerre (positive or right skew) and Bessel (negative or left skew). These skewed HOK are implemented in the blind source separation/independent component analysis (BSS/ICA) algorithm. The tests for these proposed HOK are accomplished using three scenarios to simulate a real environment using actual sound sources, an environment of mixtures of multimodal fast-changing probability density function (pdf) sources that represent a challenge to the symmetrical HOK, and an environment of an adverse case (near gaussian). The separation is performed by minimizing the mutual information (MI) among the mixed sources. The performance of the skewed kernels is compared to the performance of the standard kernels such as Epanechnikov, bisquare, trisquare, and gaussian and the performance of the symmetrical HOK generated using the polynomials Chebyshev1, Chebyshev2, Gegenbauer, Jacobi, and Legendre to the tenth order. The gaussian HOK are generated using the Hermite polynomial and the Wand and Schucany procedure. The comparison among the 96 kernels is based on the average intersymbol interference ratio (AISIR) and the time needed to complete the separation. In terms of AISIR, the skewed kernels' performance is better than that of the standard kernels and rivals most of the symmetrical kernels' performance. The importance of these new skewed HOK is manifested in the environment of the multimodal pdf mixtures. In such an environment, the skewed HOK come in first place compared with the symmetrical HOK. These new families can substitute for symmetrical HOKs in such applications.

Error-Gated Hebbian Rule: A Local Learning Rule for Principal and Independent Component Analysis

We developed a biologically plausible unsupervised learning algorithm, error-gated Hebbian rule (EGHR)-β, that performs principal component analysis (PCA) and independent component analysis (ICA) in a single-layer feedforward neural network. If parameter β = 1, it can extract the subspace that major principal components span similarly to Oja’s subspace rule for PCA. If β = 0, it can separate independent sources similarly to Bell-Sejnowski’s ICA rule but without requiring the same number of input and output neurons. Unlike these engineering rules, the EGHR-β can be easily implemented in a biological or neuromorphic circuit because it only uses local information available at each synapse. We analytically and numerically demonstrate the reliability of the EGHR-β in extracting and separating major sources given high-dimensional input. By adjusting β, the EGHR-β can extract sources that are missed by the conventional engineering approach that first applies PCA and then ICA. Namely, the proposed rule can successfully extract hidden natural images even in the presence of dominant or non-Gaussian noise components. The results highlight the reliability and utility of the EGHR-β for large-scale parallel computation of PCA and ICA and its future implementation in a neuromorphic hardware.

A Local Learning Rule for Independent Component Analysis

Humans can separately recognize independent sources when they sense their superposition. This decomposition is mathematically formulated as independent component analysis (ICA). While a few biologically plausible learning rules, so-called local learning rules, have been proposed to achieve ICA, their performance varies depending on the parameters characterizing the mixed signals. Here, we propose a new learning rule that is both easy to implement and reliable. Both mathematical and numerical analyses confirm that the proposed rule outperforms other local learning rules over a wide range of parameters. Notably, unlike other rules, the proposed rule can separate independent sources without any preprocessing, even if the number of sources is unknown. The successful performance of the proposed rule is then demonstrated using natural images and movies. We discuss the implications of this finding for our understanding of neuronal information processing and its promising applications to neuromorphic engineering.

A novel algorithm for independent component analysis with reference and methods for its applications

This paper presents a stable and fast algorithm for independent component analysis with reference (ICA-R). This is a technique for incorporating available reference signals into the ICA contrast function so as to form an augmented Lagrangian function under the framework of constrained ICA (cICA). The previous ICA-R algorithm was constructed by solving the optimization problem via a Newton-like learning style. Unfortunately, the slow convergence and potential misconvergence limit the capability of ICA-R. This paper first investigates and probes the flaws of the previous algorithm and then introduces a new stable algorithm with a faster convergence speed. There are two other highlights in this paper: first, new approaches, including the reference deflation technique and a direct way of obtaining references, are introduced to facilitate the application of ICA-R; second, a new method is proposed that the new ICA-R is used to recover the complete underlying sources with new advantages compared with other classical ICA methods. Finally, the experiments on both synthetic and real-world data verify the better performance of the new algorithm over both previous ICA-R and other well-known methods.

A robust approach to independent component analysis of signals with high-level noise measurements

We propose a robust approach for independent component analysis (ICA) of signals where observations are contaminated with high-level additive noise and/or outliers. The source signals may contain mixtures of both sub-Gaussian and super-Gaussian components, and the number of sources is unknown. Our robust approach includes two procedures. In the first procedure, a robust prewhitening technique is used to reduce the power of additive noise, the dimensionality and the correlation among sources. A cross-validation technique is introduced to estimate the number of sources in this first procedure. In the second procedure, a nonlinear function is derived using the parameterized t-distribution density model. This nonlinear function is robust against the undue influence of outliers fundamentally. Moreover, the stability of the proposed algorithm and the robust property of misestimating the parameters (kurtosis) have been studied. By combining the t-distribution model with a family of light-tailed distributions (sub-Gaussian) model, we can separate the mixture of sub-Gaussian and super-Gaussian source components. Through the analysis of artificially synthesized data and real-world magnetoencephalographic (MEG) data, we illustrate the efficacy of this robust approach.

Mixing matrix estimation from sparse mixtures with unknown number of sources

In blind source separation, many methods have been proposed to estimate the mixing matrix by exploiting sparsity. However, they often need to know the source number a priori, which is very inconvenient in practice. In this paper, a new method, namely nonlinear projection and column masking (NPCM), is proposed to estimate the mixing matrix. A major advantage of NPCM is that it does not need any knowledge of the source number. In NPCM, the objective function is based on a nonlinear projection and its maxima just correspond to the columns of the mixing matrix. Thus a column can be estimated first by locating a maximum and then deflated by a masking operation. This procedure is repeated until the evaluation of the objective function decreases to zero dramatically. Thus the mixing matrix and the number of sources are estimated simultaneously. Because the masking procedure may result in some small and useless local maxima, particle swarm optimization (PSO) is introduced to optimize the objective function. Feasibility and efficiency of PSO are also discussed. Comparative experimental results show the efficiency of NPCM, especially in the cases where the number of sources is unknown and the sources are relatively less sparse.

Automated detection of regional wall motion abnormalities based on a statistical model applied to multislice short-axis cardiac MR images

In this paper, a statistical shape analysis method for myocardial contraction is presented that was built to detect and locate regional wall motion abnormalities (RWMA). For each slice level (base, middle, and apex), 44 short-axis magnetic resonance images were selected from healthy volunteers to train a statistical model of normal myocardial contraction using independent component analysis (ICA). A classification algorithm was constructed from the ICA components to automatically detect and localize abnormally contracting regions of the myocardium. The algorithm was validated on 45 patients suffering from ischemic heart disease. Two validations were performed; one with visual wall motion scores (VWMS) and the other with wall thickening (WT) used as references. Accuracy of the ICA-based method on each slice level was 69.93% (base), 89.63% (middle), and 72.78% (apex) when WT was used as reference, and 63.70% (base), 67.41% (middle), and 66.67% (apex) when VWMS was used as reference. From this we conclude that the proposed method is a promising diagnostic support tool to assist clinicians in reducing the subjectivity in VWMS.

Automatic prediction of myocardial contractility improvement in stress MRI using shape morphometrics with independent component analysis

An important assessment in patients with ischemic heart disease is whether myocardial contractility may improve after treatment. The prediction of myocardial contractility improvement is generally performed under physical or pharmalogical stress conditions. In this paper, we present a technique to build a statistical model of healthy myocardial contraction using independent component analysis. The model is used to detect regions with abnormal contraction in patients both during rest and stress.

A Generalized Contrast Function and Stability Analysis for Overdetermined Blind Separation of Instantaneous Mixtures

In this letter, the problem of blind separation of n independent sources from their m linear instantaneous mixtures is considered. First, a generalized contrast function is defined as a valuable extension of the existing classical and nonsymmetrical contrast functions. It is applicable to the overdetermined blind separation (m > n) with an unknown number of sources, because not only independent components but also redundant ones are allowed in the outputs of a separation system. Second, a natural gradient learning algorithm developed primarily for the complete case (m = n) is shown to work as well with an n × m or m × m separating matrix, for each optimizes a certain mutual information contrast function. Finally, we present stability analysis for a newly proposed generalized orthogonal natural gradient algorithm (which can perform the overdetermined blind separation when n is unknown), obtaining an expectable result that its local stability conditions are slightly stricter than those of the conventional natural gradient algorithm using an invertible mixing matrix (m = n).

Approach and Applications of Constrained ICA

This paper presents the technique of constrained independent component analysis (cICA) and demonstrates two applications, less-complete ICA, and ICA with reference (ICA-R). The cICA is proposed as a general framework to incorporate additional requirements and prior information in the form of constraints into the ICA contrast function. The adaptive solutions using the Newton-like learning are proposed to solve the constrained optimization problem. The applications illustrate the versatility of the cICA by separating subspaces of independent components according to density types and extracting a set of desired sources when rough templates are available. The experiments using face images and functional MR images demonstrate the usage and efficacy of the cICA.

Adaptive Blind Separation with an Unknown Number of Sources

The blind source separation (BSS) problem with an unknown number of sources is an important practical issue that is usually skipped by assuming that the source number n is known and equal to the number m of sensors. This letter studies the general BSS problem satisfying m ≥ n. First, it is shown that the mutual information of outputs of the separation network is a cost function for BSS, provided that the mixing matrix is of full column rank and the m×m separating matrix is nonsingular. The mutual information reaches its local minima at the separation points, where the m outputs consist of n desired source signals and m−n redundant signals. Second, it is proved that the natural gradient algorithm proposed primarily for complete BSS (m n) can be generalized to deal with the overdetermined BSS problem (m>n), but it would diverge inevitably due to lack of a stationary point. To overcome this shortcoming, we present a modified algorithm, which can perform BSS steadily and provide the desired source signals at specified channels if some matrix is designed properly. Finally, the validity of the proposed algorithm is confirmed by computer simulations on artificially synthesized data.

Overview of independent component analysis technique with an application to synthetic aperture radar (SAR) imagery processing

We present an overview of independent component analysis, an emerging signal processing technique based on neural networks, with the aim to provide an up-to-date survey of the theoretical streams in this discipline and of the current applications in the engineering area. We also focus on a particular application, dealing with a remote sensing technique based on synthetic aperture radar imagery processing: we briefly review the features and main applications of synthetic aperture radar and show how blind signal processing by neural networks may be advantageously employed to enhance the quality of remote sensing data.

Unsupervised neural learning on lie group

The present paper aims at introducing the concepts and mathematical details of unsupervised neural learning with orthonormality constrains. The neural structures considered are single non-linear layers and the learnable parameters are organized in matrices, as usual, which gives the parameters spaces the geometrical structure of the Euclidean manifold. The constraint of orthonormality for the connection-matrices further restricts the parameters spaces to differential manifolds such as the orthogonal group, the compact Stiefel manifold and its extensions. For these reasons, the instruments for characterizing and studying the behavior of learning equations for these particular networks are provided by the differential geometry of Lie groups. In particular, two sub-classes of the general Lie-group learning theories are studied in detail, dealing with first-order (gradient-based) and second-order (non-gradient-based) learning. Although the considered class of learning theories is very general, in the present paper special attention is paid to unsupervised learning paradigms.

Effect of signal length on the performance of independent component analysis when extracting the lambda wave

The aim of the study was to investigate the effect of signal length on the performance of a signal source separation method, independent component analysis (ICA), when extracting the visual evoked potential (EP) lambda wave from saccade-related electro-encephalogram (EEG) waveforms. A method was devised that enabled the effective length of the recorded EEG traces to be increased prior to processing by ICA. This involved abutting EEG traces from an appropriate number of successive trials (a trial was a set of waveforms recorded from 64 electrode locations in a study investigating saccade performance). ICA was applied to the saccade-related EEG and electro-oculogram (EOG) waveforms recorded from the electrode locations. One spatial and five temporal features of the lambda wave were monitored to assess the performance of ICA applied to both abutted and non-abutted waveforms. ICA applied to abutted trials managed to extract all six features across all seven subjects included in the study. This was not the case when ICA was applied to the non-abutted trials. It was quantitatively demonstrated that the process of abutting EEG waveforms was useful for ICA preprocessing when extracting lambda waves.

A Theory for Learning by Weight Flow on Stiefel-Grassman Manifold

Recently we introduced the concept of neural network learning on Stiefel-Grassman manifold for multilayer perceptron—like networks. Contributions of other authors have also appeared in the scientific literature about this topic. This article presents a general theory for it and illustrates how existing theories may be explained within the general framework proposed here.

Noise reduction in brain evoked potentials based on third-order correlations

In this paper, we use third-order correlations (TOC) in developing a filtering technique for the recovery of brain evoked potentials (EPs). The main idea behind the presented technique is to pass the noisy signal through a finite impulse response filter whose impulse response is matched with the shape of the noise-free signal. It is shown that it is possible to estimate the filter impulse response on basis of a selected third-order correlation slice (TOCS) of the input noisy signal. This is justified by two facts. The first one is that the noise-free EPs can be modeled as a sum of damped sinusoidal signals and the selected TOCS preserve the signal structure. The second fact is that the TOCS is insensitive to both Gaussian noise and other symmetrically distributed non-Gaussian noise, (white or colored). Furthermore, the approach can be applied to either nonaveraged or averaged EP observation data. In the nonaveraged data case, the approach therefore preserves information about amplitude and latency changes. Both fixed and adaptive versions of the proposed filtering technique are described. Extensive simulation results are provided to show the validity and effectiveness of the proposed cumulant-based filtering technique in comparison with the conventional correlation-based counterpart.

Blind signal processing by the adaptive activation function neurons

The aim of this paper is to study an Information Theory based learning theory for neural units endowed with adaptive activation functions. The learning theory has the target to force the neuron to approximate the input-output transference that makes it flat (uniform) the probability density function of its output or, equivalently, that maximizes the entropy of the neuron response. Then, a network of adaptive activation function neurons is studied, and the effectiveness of the new structure is tested on Independent Component Analysis (ICA) problems. The new ICA neural algorithm is compared with the closely related 'Mixture of Densities' (MOD) technique by Xu et al.. Both simulation results and structural comparison show the new method is effective and more efficient in computational complexity.