Convergence and rate analysis of neural networks for sparse approximation

Positive Competitive Networks for Sparse Reconstruction

We propose and analyze a continuous-time firing-rate neural network, the positive firing-rate competitive network (PFCN), to tackle sparse reconstruction problems with non-negativity constraints. These problems, which involve approximating a given input stimulus from a dictionary using a set of sparse (active) neurons, play a key role in a wide range of domains, including, for example, neuroscience, signal processing, and machine learning. First, by leveraging the theory of proximal operators, we relate the equilibria of a family of continuous-time firing-rate neural networks to the optimal solutions of sparse reconstruction problems. Then we prove that the PFCN is a positive system and give rigorous conditions for the convergence to the equilibrium. Specifically, we show that the convergence depends only on a property of the dictionary and is linear-exponential in the sense that initially, the convergence rate is at worst linear and then, after a transient, becomes exponential. We also prove a number of technical results to assess the contractivity properties of the neural dynamics of interest. Our analysis leverages contraction theory to characterize the behavior of a family of firing-rate competitive networks for sparse reconstruction with and without non-negativity constraints. Finally, we validate the effectiveness of our approach via a numerical example.

Diversified Regularization Enhanced Training for Effective Manipulator Calibration

Recently, robot arms have become an irreplaceable production tool, which play an important role in the industrial production. It is necessary to ensure the absolute positioning accuracy of the robot to realize automatic production. Due to the influence of machining tolerance, assembly tolerance, the robot positioning accuracy is poor. Therefore, in order to enable the precise operation of the robot, it is necessary to calibrate the robotic kinematic parameters. The least square method and Levenberg-Marquardt (LM) algorithm are commonly used to identify the positioning error of robot. However, it generally has the overfitting caused by improper regularization schemes. To solve this problem, this article discusses six regularization schemes based on its error models, i.e., L1 , L2 , dropout, elastic, log, and swish. Moreover, this article proposes a scheme with six regularization to obtain a reliable ensemble, which can effectively avoid overfitting. The positioning accuracy of the robot is improved significantly after calibration by enough experiments, which verifies the feasibility of the proposed method.

A Fixed-Time Projection Neural Network for Solving L₁-Minimization Problem

In this article, a new projection neural network (PNN) for solving L₁ -minimization problem is proposed, which is based on classic PNN and sliding mode control technique. Furthermore, the proposed network can be used to make sparse signal reconstruction and image reconstruction. First, a sign function is introduced into the PNN model to design fixed-time PNN (FPNN). Then, under the condition that the projection matrix satisfies the restricted isometry property (RIP), the stability and fixed-time convergence of the proposed FPNN are proved by the Lyapunov method. Finally, based on the experimental results of signal simulation and image reconstruction, the proposed FPNN shows the effectiveness and superiority compared with that of the existing PNNs.

Constrained brain volume in an efficient coding model explains the fraction of excitatory and inhibitory neurons in sensory cortices

The number of neurons in mammalian cortex varies by multiple orders of magnitude across different species. In contrast, the ratio of excitatory to inhibitory neurons (E:I ratio) varies in a much smaller range, from 3:1 to 9:1 and remains roughly constant for different sensory areas within a species. Despite this structure being important for understanding the function of neural circuits, the reason for this consistency is not yet understood. While recent models of vision based on the efficient coding hypothesis show that increasing the number of both excitatory and inhibitory cells improves stimulus representation, the two cannot increase simultaneously due to constraints on brain volume. In this work, we implement an efficient coding model of vision under a constraint on the volume (using number of neurons as a surrogate) while varying the E:I ratio. We show that the performance of the model is optimal at biologically observed E:I ratios under several metrics. We argue that this happens due to trade-offs between the computational accuracy and the representation capacity for natural stimuli. Further, we make experimentally testable predictions that 1) the optimal E:I ratio should be higher for species with a higher sparsity in the neural activity and 2) the character of inhibitory synaptic distributions and firing rates should change depending on E:I ratio. Our findings, which are supported by our new preliminary analyses of publicly available data, provide the first quantitative and testable hypothesis based on optimal coding models for the distribution of excitatory and inhibitory neural types in the mammalian sensory cortices.

Neurons in the brain come in two main types: excitatory and inhibitory. The interplay between them shapes neural computation. Despite brain sizes varying by several orders of magnitude across species, the ratio of excitatory and inhibitory sub-populations (E:I ratio) remains relatively constant, and we don’t know why. Simulations of theoretical models of the brain can help answer such questions, especially when experiments are prohibitive or impossible. Here we placed one such theoretical model of sensory coding (’sparse coding’ that minimizes the simultaneously active neurons) under a biophysical ‘volume’ constraint that fixes the total number of neurons available. We vary the E:I ratio in the model (which cannot be done in experiments), and reveal an optimal E:I ratio where the representation of sensory stimulus and energy consumption within the circuit are concurrently optimal. We also show that varying the population sparsity changes the optimal E:I ratio, spanning the relatively narrow ranges observed in biology. Crucially, this minimally parameterized theoretical model makes predictions about structure (recurrent connectivity) and activity (population sparsity) in neural circuits with different E:I ratios (i.e., different species), of which we verify the latter in a first-of-its-kind inter-species comparison using newly publicly available data.

Convergence Analysis of Single Latent Factor-Dependent, Nonnegative, and Multiplicative Update-Based Nonnegative Latent Factor Models

A single latent factor (LF)-dependent, nonnegative, and multiplicative update (SLF-NMU) learning algorithm is highly efficient in building a nonnegative LF (NLF) model defined on a high-dimensional and sparse (HiDS) matrix. However, convergence characteristics of such NLF models are never justified in theory. To address this issue, this study conducts rigorous convergence analysis for an SLF-NMU-based NLF model. The main idea is twofold: 1) proving that its learning objective keeps nonincreasing with its SLF-NMU-based learning rules via constructing specific auxiliary functions; and 2) proving that it converges to a stable equilibrium point with its SLF-NMU-based learning rules via analyzing the Karush-Kuhn-Tucker (KKT) conditions of its learning objective. Experimental results on ten HiDS matrices from real applications provide numerical evidence that indicates the correctness of the achieved proof.

Synthesis of recurrent neural dynamics for monotone inclusion with application to Bayesian inference

We propose a top-down approach to construct recurrent neural circuit dynamics for the mathematical problem of monotone inclusion (MoI). MoI in a general optimization framework that encompasses a wide range of contemporary problems, including Bayesian inference and Markov decision making. We show that in a recurrent neural circuit/network with Poisson neurons, each neuron's firing curve can be understood as a proximal operator of a local objective function, while the overall circuit dynamics constitutes an operator-splitting system of ordinary differential equations whose equilibrium point corresponds to the solution of the MoI problem. Our analysis thus establishes that neural circuits are a substrate for solving a broad class of computational tasks. In this regard, we provide an explicit synthesis procedure for building neural circuits for specific MoI problems and demonstrate it for the specific case of Bayesian inference and sparse neural coding.

Sparse Coding Using the Locally Competitive Algorithm on the TrueNorth Neurosynaptic System

The Locally Competitive Algorithm (LCA) is a biologically plausible computational architecture for sparse coding, where a signal is represented as a linear combination of elements from an over-complete dictionary. In this paper we map the LCA algorithm on the brain-inspired, IBM TrueNorth Neurosynaptic System. We discuss data structures and representation as well as the architecture of functional processing units that perform non-linear threshold, vector-matrix multiplication. We also present the design of the micro-architectural units that facilitate the implementation of dynamical based iterative algorithms. Experimental results with the LCA algorithm using the limited precision, fixed-point arithmetic on TrueNorth compare favorably with results using floating-point computations on a general purpose computer. The scaling of the LCA algorithm within the constraints of the TrueNorth is also discussed.

A Two-Timescale Duplex Neurodynamic Approach to Biconvex Optimization

This paper presents a two-timescale duplex neurodynamic system for constrained biconvex optimization. The two-timescale duplex neurodynamic system consists of two recurrent neural networks (RNNs) operating collaboratively at two timescales. By operating on two timescales, RNNs are able to avoid instability. In addition, based on the convergent states of the two RNNs, particle swarm optimization is used to optimize initial states of the RNNs to avoid local minima. It is proven that the proposed system is globally convergent to the global optimum with probability one. The performance of the two-timescale duplex neurodynamic system is substantiated based on the benchmark problems. Furthermore, the proposed system is applied for L₁ -constrained nonnegative matrix factorization.

Robustness Analysis on Dual Neural Network-based $k$ WTA With Input Noise

This paper studies the effects of uniform input noise and Gaussian input noise on the dual neural network-based WTA (DNN- WTA) model. We show that the state of the network (under either uniform input noise or Gaussian input noise) converges to one of the equilibrium points. We then derive a formula to check if the network produce correct outputs or not. Furthermore, for the uniformly distributed inputs, two lower bounds (one for each type of input noise) on the probability that the network produces the correct outputs are presented. Besides, when the minimum separation amongst inputs is given, we derive the condition for the network producing the correct outputs. Finally, experimental results are presented to verify our theoretical results. Since random drift in the comparators can be considered as input noise, our results can be applied to the random drift situation.

A Neurodynamic Model to Solve Nonlinear Pseudo-Monotone Projection Equation and Its Applications

In this paper, a neurodynamic model is given to solve nonlinear pseudo-monotone projection equation. Under pseudo-monotonicity condition and Lipschitz continuous condition, the projection neurodynamic model is proved to be stable in the sense of Lyapunov, globally convergent, globally asymptotically stable, and globally exponentially stable. Also, we show that, our new neurodynamic model is effective to solve the nonconvex optimization problems. Moreover, since monotonicity is a special case of pseudo-monotonicity and also since a co-coercive mapping is Lipschitz continuous and monotone, and a strongly pseudo-monotone mapping is pseudo-monotone, the neurodynamic model can be applied to solve a broader classes of constrained optimization problems related to variational inequalities, pseudo-convex optimization problem, linear and nonlinear complementarity problems, and linear and convex quadratic programming problems. Finally, several illustrative examples are stated to demonstrate the effectiveness and efficiency of our new neurodynamic model.

Lagrange Programming Neural Network for Nondifferentiable Optimization Problems in Sparse Approximation

The major limitation of the Lagrange programming neural network (LPNN) approach is that the objective function and the constraints should be twice differentiable. Since sparse approximation involves nondifferentiable functions, the original LPNN approach is not suitable for recovering sparse signals. This paper proposes a new formulation of the LPNN approach based on the concept of the locally competitive algorithm (LCA). Unlike the classical LCA approach which is able to solve unconstrained optimization problems only, the proposed LPNN approach is able to solve the constrained optimization problems. Two problems in sparse approximation are considered. They are basis pursuit (BP) and constrained BP denoise (CBPDN). We propose two LPNN models, namely, BP-LPNN and CBPDN-LPNN, to solve these two problems. For these two models, we show that the equilibrium points of the models are the optimal solutions of the two problems, and that the optimal solutions of the two problems are the equilibrium points of the two models. Besides, the equilibrium points are stable. Simulations are carried out to verify the effectiveness of these two LPNN models.

Recurrent networks with soft-thresholding nonlinearities for lightweight coding

A long-standing and influential hypothesis in neural information processing is that early sensory networks adapt themselves to produce efficient codes of afferent inputs. Here, we show how a nonlinear recurrent network provides an optimal solution for the efficient coding of an afferent input and its history. We specifically consider the problem of producing lightweight codes, ones that minimize both ℓ₁ and ℓ₂ constraints on sparsity and energy, respectively. When embedded in a linear coding paradigm, this problem results in a non-smooth convex optimization problem. We employ a proximal gradient descent technique to develop the solution, showing that the optimal code is realized through a recurrent network endowed with a nonlinear soft thresholding operator. The training of the network connection weights is readily achieved through gradient-based local learning. If such learning is assumed to occur on a slower time-scale than the (faster) recurrent dynamics, then the network as a whole converges to an optimal set of codes and weights via what is, in effect, an alternative minimization procedure. Our results show how the addition of thresholding nonlinearities to a recurrent network may enable the production of lightweight, history-sensitive encoding schemes.

Rate of Convergence of the FOCUSS Algorithm

Focal underdetermined system solver (FOCUSS) is a powerful method for basis selection and sparse representation, where it employs the [Formula: see text]-norm with p ∈ (0,2) to measure the sparsity of solutions. In this paper, we give a systematical analysis on the rate of convergence of the FOCUSS algorithm with respect to p ∈ (0,2) . We prove that the FOCUSS algorithm converges superlinearly for and linearly for usually, but may superlinearly in some very special scenarios. In addition, we verify its rates of convergence with respect to p by numerical experiments.

Echo State Networks With Orthogonal Pigeon-Inspired Optimization for Image Restoration

In this paper, a neurodynamic approach for image restoration is proposed. Image restoration is a process of estimating original images from blurred and/or noisy images. It can be considered as a mapping problem that can be solved by neural networks. Echo state network (ESN) is a recurrent neural network with a simplified training process, which is adopted to estimate the original images in this paper. The parameter selection is important to the performance of the ESN. Thus, the pigeon-inspired optimization (PIO) approach is employed in the training process of the ESN to obtain desired parameters. Moreover, the orthogonal design strategy is utilized in the initialization of PIO to improve the diversity of individuals. The proposed method is tested on several deteriorated images with different sorts and levels of blur and/or noise. Results obtained by the improved ESN are compared with those obtained by several state-of-the-art methods. It is verified experimentally that better image restorations can be obtained for different blurred and/or noisy instances with the proposed neurodynamic method. In addition, the performance of the orthogonal PIO algorithm is compared with that of several existing bioinspired optimization algorithms to confirm its superiority.

L₁-Minimization Algorithms for Sparse Signal Reconstruction Based on a Projection Neural Network

This paper presents several L1-minimization algorithms for sparse signal reconstruction based on a continuous-time projection neural network (PNN). First, a one-layer projection neural network is designed based on a projection operator and a projection matrix. The stability and global convergence of the proposed neural network are proved. Then, based on a discrete-time version of the PNN, several L1-minimization algorithms for sparse signal reconstruction are developed and analyzed. Experimental results based on random Gaussian sparse signals show the effectiveness and performance of the proposed algorithms. Moreover, experimental results based on two face image databases are presented that reveal the influence of sparsity to the recognition rate. The algorithms are shown to be robust to the amplitude and sparsity level of signals as well as efficient with high convergence rate compared with several existing L1-minimization algorithms.

A Generalized Hopfield Network for Nonsmooth Constrained Convex Optimization: Lie Derivative Approach

This paper proposes a generalized Hopfield network for solving general constrained convex optimization problems. First, the existence and the uniqueness of solutions to the generalized Hopfield network in the Filippov sense are proved. Then, the Lie derivative is introduced to analyze the stability of the network using a differential inclusion. The optimality of the solution to the nonsmooth constrained optimization problems is shown to be guaranteed by the enhanced Fritz John conditions. The convergence rate of the generalized Hopfield network can be estimated by the second-order derivative of the energy function. The effectiveness of the proposed network is evaluated on several typical nonsmooth optimization problems and used to solve the hierarchical and distributed model predictive control four-tank benchmark.

Modeling Inhibitory Interneurons in Efficient Sensory Coding Models

There is still much unknown regarding the computational role of inhibitory cells in the sensory cortex. While modeling studies could potentially shed light on the critical role played by inhibition in cortical computation, there is a gap between the simplicity of many models of sensory coding and the biological complexity of the inhibitory subpopulation. In particular, many models do not respect that inhibition must be implemented in a separate subpopulation, with those inhibitory interneurons having a diversity of tuning properties and characteristic E/I cell ratios. In this study we demonstrate a computational framework for implementing inhibition in dynamical systems models that better respects these biophysical observations about inhibitory interneurons. The main approach leverages recent work related to decomposing matrices into low-rank and sparse components via convex optimization, and explicitly exploits the fact that models and input statistics often have low-dimensional structure that can be exploited for efficient implementations. While this approach is applicable to a wide range of sensory coding models (including a family of models based on Bayesian inference in a linear generative model), for concreteness we demonstrate the approach on a network implementing sparse coding. We show that the resulting implementation stays faithful to the original coding goals while using inhibitory interneurons that are much more biophysically plausible.

Cortical function is a result of coordinated interactions between excitatory and inhibitory neural populations. In previous theoretical models of sensory systems, inhibitory neurons are often ignored or modeled too simplistically to contribute to understanding their role in cortical computation. In biophysical reality, inhibition is implemented with interneurons that have different characteristics from the population of excitatory cells. In this study, we propose a computational approach for including inhibition in theoretical models of neural coding in a way that respects several of these important characteristics, such as the relative number of inhibitory cells and the diversity of their response properties. The main idea is that the significant structure of the sensory world is reflected in very structured models of sensory coding, which can then be exploited in the implementation of the model using modern computational techniques. We demonstrate this approach on one specific model of sensory coding (called “sparse coding”) that has been successful at modeling other aspects of sensory cortex.

Convergence analysis of the FOCUSS algorithm

Focal Underdetermined System Solver (FOCUSS) is a powerful and easy to implement tool for basis selection and inverse problems. One of the fundamental problems regarding this method is its convergence, which remains unsolved until now. We investigate the convergence of the FOCUSS algorithm in this paper. We first give a rigorous derivation for the FOCUSS algorithm by exploiting the auxiliary function. Following this, we further prove its convergence by stability analysis.

A compressed sensing perspective of hippocampal function

Hippocampus is one of the most important information processing units in the brain. Input from the cortex passes through convergent axon pathways to the downstream hippocampal subregions and, after being appropriately processed, is fanned out back to the cortex. Here, we review evidence of the hypothesis that information flow and processing in the hippocampus complies with the principles of Compressed Sensing (CS). The CS theory comprises a mathematical framework that describes how and under which conditions, restricted sampling of information (data set) can lead to condensed, yet concise, forms of the initial, subsampled information entity (i.e., of the original data set). In this work, hippocampus related regions and their respective circuitry are presented as a CS-based system whose different components collaborate to realize efficient memory encoding and decoding processes. This proposition introduces a unifying mathematical framework for hippocampal function and opens new avenues for exploring coding and decoding strategies in the brain.

OPTIMAL SPARSE APPROXIMATION WITH INTEGRATE AND FIRE NEURONS

Sparse approximation is a hypothesized coding strategy where a population of sensory neurons (e.g. V1) encodes a stimulus using as few active neurons as possible. We present the Spiking LCA (locally competitive algorithm), a rate encoded Spiking Neural Network (SNN) of integrate and fire neurons that calculate sparse approximations. The Spiking LCA is designed to be equivalent to the nonspiking LCA, an analog dynamical system that converges on a ℓ1-norm sparse approximations exponentially. We show that the firing rate of the Spiking LCA converges on the same solution as the analog LCA, with an error inversely proportional to the sampling time. We simulate in NEURON a network of 128 neuron pairs that encode 8 × 8 pixel image patches, demonstrating that the network converges to nearly optimal encodings within 20 ms of biological time. We also show that when using more biophysically realistic parameters in the neurons, the gain function encourages additional ℓ0-norm sparsity in the encoding, relative both to ideal neurons and digital solvers.

Short-term memory capacity in networks via the restricted isometry property

Cortical networks are hypothesized to rely on transient network activity to support short-term memory (STM). In this letter, we study the capacity of randomly connected recurrent linear networks for performing STM when the input signals are approximately sparse in some basis. We leverage results from compressed sensing to provide rigorous nonasymptotic recovery guarantees, quantifying the impact of the input sparsity level, the input sparsity basis, and the network characteristics on the system capacity. Our analysis demonstrates that network memory capacities can scale superlinearly with the number of nodes and in some situations can achieve STM capacities that are much larger than the network size. We provide perfect recovery guarantees for finite sequences and recovery bounds for infinite sequences. The latter analysis predicts that network STM systems may have an optimal recovery length that balances errors due to omission and recall mistakes. Furthermore, we show that the conditions yielding optimal STM capacity can be embodied in several network topologies, including networks with sparse or dense connectivities.

Visual nonclassical receptive field effects emerge from sparse coding in a dynamical system

Extensive electrophysiology studies have shown that many V1 simple cells have nonlinear response properties to stimuli within their classical receptive field (CRF) and receive contextual influence from stimuli outside the CRF modulating the cell's response. Models seeking to explain these non-classical receptive field (nCRF) effects in terms of circuit mechanisms, input-output descriptions, or individual visual tasks provide limited insight into the functional significance of these response properties, because they do not connect the full range of nCRF effects to optimal sensory coding strategies. The (population) sparse coding hypothesis conjectures an optimal sensory coding approach where a neural population uses as few active units as possible to represent a stimulus. We demonstrate that a wide variety of nCRF effects are emergent properties of a single sparse coding model implemented in a neurally plausible network structure (requiring no parameter tuning to produce different effects). Specifically, we replicate a wide variety of nCRF electrophysiology experiments (e.g., end-stopping, surround suppression, contrast invariance of orientation tuning, cross-orientation suppression, etc.) on a dynamical system implementing sparse coding, showing that this model produces individual units that reproduce the canonical nCRF effects. Furthermore, when the population diversity of an nCRF effect has also been reported in the literature, we show that this model produces many of the same population characteristics. These results show that the sparse coding hypothesis, when coupled with a biophysically plausible implementation, can provide a unified high-level functional interpretation to many response properties that have generally been viewed through distinct mechanistic or phenomenological models.

Simple cells in the primary visual cortex (V1) demonstrate many response properties that are either nonlinear or involve response modulations (i.e., stimuli that do not cause a response in isolation alter the cell's response to other stimuli). These non-classical receptive field (nCRF) effects are generally modeled individually and their collective role in biological vision is not well understood. Previous work has shown that classical receptive field (CRF) properties of V1 cells (i.e., the spatial structure of the visual field responsive to stimuli) could be explained by the sparse coding hypothesis, which is an optimal coding model that conjectures a neural population should use the fewest number of cells simultaneously to represent each stimulus. In this paper, we have performed extensive simulated physiology experiments to show that many nCRF response properties are simply emergent effects of a dynamical system implementing this same sparse coding model. These results suggest that rather than representing disparate information processing operations themselves, these nCRF effects could be consequences of an optimal sensory coding strategy that attempts to represent each stimulus most efficiently. This interpretation provides a potentially unifying high-level functional interpretation to many response properties that have generally been viewed through distinct models.

Fast-convergent double-sigmoid Hopfield neural network as applied to optimization problems

The Hopfield neural network (HNN) has been widely used in numerous different optimization problems since the early 1980s. The convergence speed of the HNN (already in high gain) eventually plays a critical role in various real-time applications. In this brief, we propose and analyze a generalized HNN which drastically improves the convergence speed of the network, and thus allows benefiting from the HNN capabilities in solving the optimization problems in real time. By examining the channel allocation optimization problem in cellular radio systems, which is NP-complete and in which fast solution is necessary due to time-varying link gains, as well as the associative memory problem, computer simulations confirm the dramatic improvement in convergence speed at the expense of using a second nonlinear function in the proposed network.

Common nature of learning between back-propagation and Hopfield-type neural networks for generalized matrix inversion with simplified models

In this paper, two simple-structure neural networks based on the error back-propagation (BP) algorithm (i.e., BP-type neural networks, BPNNs) are proposed, developed, and investigated for online generalized matrix inversion. Specifically, the BPNN-L and BPNN-R models are proposed and investigated for the left and right generalized matrix inversion, respectively. In addition, for the same problem-solving task, two discrete-time Hopfield-type neural networks (HNNs) are developed and investigated in this paper. Similar to the classification of the presented BPNN-L and BPNN-R models, the presented HNN-L and HNN-R models correspond to the left and right generalized matrix inversion, respectively. Comparing the BPNN weight-updating formula with the HNN state-transition equation for the specific (i.e., left or right) generalized matrix inversion, we show that such two derived learning-expressions turn out to be the same (in mathematics), although the BP and Hopfield-type neural networks are evidently different from each other a great deal, in terms of network architecture, physical meaning, and training patterns. Numerical results with different illustrative examples further demonstrate the efficacy of the presented BPNNs and HNNs for online generalized matrix inversion and, more importantly, their common natures of learning.

Configurable hardware integrate and fire neurons for sparse approximation

Sparse approximation is an important optimization problem in signal and image processing applications. A Hopfield-Network-like system of integrate and fire (IF) neurons is proposed as a solution, using the Locally Competitive Algorithm (LCA) to solve an overcomplete L1 sparse approximation problem. A scalable system architecture is described, including IF neurons with a nonlinear firing function, and current-based synapses to provide linear computation. A network of 18 neurons with 12 inputs is implemented on the RASP 2.9v chip, a Field Programmable Analog Array (FPAA) with directly programmable floating gate elements. Said system uses over 1400 floating gates, the largest system programmed on a FPAA to date. The circuit successfully reproduced the outputs of a digital optimization program, converging to within 4.8% RMS, and an objective cost only 1.7% higher on average. The active circuit consumed 559 μA of current at 2.4 V and converges on solutions in 25 μs, with measurement of the converged spike rate taking an additional 1 ms. Extrapolating the scaling trends to a N=1000 node system, the spiking LCA compares favorably with state-of-the-art digital solutions, and analog solutions using a non-spiking approach.

A common network architecture efficiently implements a variety of sparsity-based inference problems

The sparse coding hypothesis has generated significant interest in the computational and theoretical neuroscience communities, but there remain open questions about the exact quantitative form of the sparsity penalty and the implementation of such a coding rule in neurally plausible architectures. The main contribution of this work is to show that a wide variety of sparsity-based probabilistic inference problems proposed in the signal processing and statistics literatures can be implemented exactly in the common network architecture known as the locally competitive algorithm (LCA). Among the cost functions we examine are approximate l(p) norms (0 ≤ p ≤ 2), modified l(p)-norms, block-l1 norms, and reweighted algorithms. Of particular interest is that we show significantly increased performance in reweighted l1 algorithms by inferring all parameters jointly in a dynamical system rather than using an iterative approach native to digital computational architectures.