Analog "neuronal" networks in early vision

Survey of Datafusion Techniques for Laser and Vision Based Sensor Integration for Autonomous Navigation

This paper focuses on data fusion, which is fundamental to one of the most important modules in any autonomous system: perception. Over the past decade, there has been a surge in the usage of smart/autonomous mobility systems. Such systems can be used in various areas of life like safe mobility for the disabled, senior citizens, and so on and are dependent on accurate sensor information in order to function optimally. This information may be from a single sensor or a suite of sensors with the same or different modalities. We review various types of sensors, their data, and the need for fusion of the data with each other to output the best data for the task at hand, which in this case is autonomous navigation. In order to obtain such accurate data, we need to have optimal technology to read the sensor data, process the data, eliminate or at least reduce the noise and then use the data for the required tasks. We present a survey of the current data processing techniques that implement data fusion using different sensors like LiDAR that use light scan technology, stereo/depth cameras, Red Green Blue monocular (RGB) and Time-of-flight (TOF) cameras that use optical technology and review the efficiency of using fused data from multiple sensors rather than a single sensor in autonomous navigation tasks like mapping, obstacle detection, and avoidance or localization. This survey will provide sensor information to researchers who intend to accomplish the task of motion control of a robot and detail the use of LiDAR and cameras to accomplish robot navigation.

Lighting-from-above prior in biological motion perception

The visual system is able to recognize body motion from impoverished stimuli. This requires combining stimulus information with visual priors. We present a new visual illusion showing that one of these priors is the assumption that bodies are typically illuminated from above. A change of illumination direction from above to below flips the perceived locomotion direction of a biological motion stimulus. Control experiments show that the underlying mechanism is different from shape-from-shading and directly combines information about body motion with a lighting-from-above prior. We further show that the illusion is critically dependent on the intrinsic luminance gradients of the most mobile parts of the moving body. We present a neural model with physiologically plausible mechanisms that accounts for the illusion and shows how the illumination prior might be encoded within the visual pathway. Our experiments demonstrate, for the first time, a direct influence of illumination priors in high-level motion vision.

Relating functional connectivity in V1 neural circuits and 3D natural scenes using Boltzmann machines

Bayesian theory has provided a compelling conceptualization for perceptual inference in the brain. Central to Bayesian inference is the notion of statistical priors. To understand the neural mechanisms of Bayesian inference, we need to understand the neural representation of statistical regularities in the natural environment. In this paper, we investigated empirically how statistical regularities in natural 3D scenes are represented in the functional connectivity of disparity-tuned neurons in the primary visual cortex of primates. We applied a Boltzmann machine model to learn from 3D natural scenes, and found that the units in the model exhibited cooperative and competitive interactions, forming a "disparity association field", analogous to the contour association field. The cooperative and competitive interactions in the disparity association field are consistent with constraints of computational models for stereo matching. In addition, we simulated neurophysiological experiments on the model, and found the results to be consistent with neurophysiological data in terms of the functional connectivity measurements between disparity-tuned neurons in the macaque primary visual cortex. These findings demonstrate that there is a relationship between the functional connectivity observed in the visual cortex and the statistics of natural scenes. They also suggest that the Boltzmann machine can be a viable model for conceptualizing computations in the visual cortex and, as such, can be used to predict neural circuits in the visual cortex from natural scene statistics.

The visual system's internal model of the world

The Bayesian paradigm has provided a useful conceptual theory for understanding perceptual computation in the brain. While the detailed neural mechanisms of Bayesian inference are not fully understood, recent computational and neurophysiological works have illuminated the underlying computational principles and representational architecture. The fundamental insights are that the visual system is organized as a modular hierarchy to encode an internal model of the world, and that perception is realized by statistical inference based on such internal model. In this paper, I will discuss and analyze the varieties of representational schemes of these internal models and how they might be used to perform learning and inference. I will argue for a unified theoretical framework for relating the internal models to the observed neural phenomena and mechanisms in the visual cortex.

Representing and computing regular languages on massively parallel networks

A general method is proposed for incorporating rule-based constraints corresponding to regular languages into stochastic inference problems, thereby allowing for a unified representation of stochastic and syntactic pattern constraints. The authors' approach establishes the formal connection of rules to Chomsky grammars and generalizes the original work of Shannon on the encoding of rule-based channel sequences to Markov chains of maximum entropy. This maximum entropy probabilistic view leads to Gibbs representations with potentials which have their number of minima growing at precisely the exponential rate that the language of deterministically constrained sequences grow. These representations are coupled to stochastic diffusion algorithms, which sample the language-constrained sequences by visiting the energy minima according to the underlying Gibbs probability law. This coupling yields the result that fully parallel stochastic cellular automata can be derived to generate samples from the rule-based constraint sets. The production rules and neighborhood state structure of the language of sequences directly determine the necessary connection structures of the required parallel computing surface. Representations of this type have been mapped to the DAP-510 massively parallel processor consisting of 1024 mesh-connected bit-serial processing elements for performing automated segmentation of electron-micrograph images.

Tracking improves performance of biological collision avoidance models

Collision avoidance models derived from the study of insect brains do not perform universally well in practical collision scenarios, although the insects themselves may perform well in similar situations. In this article, we present a detailed simulation analysis of two well-known collision avoidance models and illustrate their limitations. In doing so, we present a novel continuous-time implementation of a neuronally based collision avoidance model. We then show that visual tracking can improve performance of these models by allowing an relative computation of the distance between the obstacle and the observer. We compare the results of simulations of the two models with and without tracking to show how tracking improves the ability of the model to detect an imminent collision. We present an implementation of one of these models processing imagery from a camera to show how it performs in real-world scenarios. These results suggest that insects may track looming objects with their gaze.

Recursive segmentation and recognition templates for image parsing

In this paper, we propose a Hierarchical Image Model (HIM) which parses images to perform segmentation and object recognition. The HIM represents the image recursively by segmentation and recognition templates at multiple levels of the hierarchy. This has advantages for representation, inference, and learning. First, the HIM has a coarse-to-fine representation which is capable of capturing long-range dependency and exploiting different levels of contextual information (similar to how natural language models represent sentence structure in terms of hierarchical representations such as verb and noun phrases). Second, the structure of the HIM allows us to design a rapid inference algorithm, based on dynamic programming, which yields the first polynomial time algorithm for image labeling. Third, we learn the HIM efficiently using machine learning methods from a labeled data set. We demonstrate that the HIM is comparable with the state-of-the-art methods by evaluation on the challenging public MSRC and PASCAL VOC 2007 image data sets.

Real-time accumulative computation motion detectors

The neurally inspired accumulative computation (AC) method and its application to motion detection have been introduced in the past years. This paper revisits the fact that many researchers have explored the relationship between neural networks and finite state machines. Indeed, finite state machines constitute the best characterized computational model, whereas artificial neural networks have become a very successful tool for modeling and problem solving. The article shows how to reach real-time performance after using a model described as a finite state machine. This paper introduces two steps towards that direction: (a) A simplification of the general AC method is performed by formally transforming it into a finite state machine. (b) A hardware implementation in FPGA of such a designed AC module, as well as an 8-AC motion detector, providing promising performance results. We also offer two case studies of the use of AC motion detectors in surveillance applications, namely infrared-based people segmentation and color-based people tracking, respectively.

An integrative model of binocular vision: a stereo model utilizing interocularly unpaired points produces both depth and binocular rivalry

Half-occluded points (visible only in one eye) are perceived at a certain depth behind the occluding surface without binocular rivalry, even though no disparity is defined at such points. Here we propose a stereo model that reconstructs 3D structures not only from disparity information of interocularly paired points but also from unpaired points. Starting with an array of depth detection cells, we introduce cells that detect unpaired points visible only in the left eye or the right eye (left and right unpaired point detection cells). They interact cooperatively with each other based on optogeometrical constraints (such as uniqueness, cohesiveness, occlusion) to recover the depth and the border of 3D objects. Since it is contradictory for monocularly visible regions to be visible in both eyes, we introduce mutual inhibition between left and right unpaired point detection cells. When input images satisfy occlusion geometry, the model outputs the depth of unpaired points properly. An interesting finding is that when we input two unmatched images, the model shows an unstable output that alternates between interpretations of monocularly visible regions for the left and the right eyes, thereby reproducing binocular rivalry. The results suggest that binocular rivalry arises from the erroneous output of a stereo mechanism that estimates the depth of half-occluded unpaired points. In this sense, our model integrates stereopsis and binocular rivalry, which are usually treated separately, into a single framework of binocular vision. There are two general theories for what the "rivals" are during binocular rivalry: the two eyes, or representations of two stimulus patterns. We propose a new hypothesis that bridges these two conflicting hypotheses: interocular inhibition between representations of monocularly visible regions causes binocular rivalry. Unlike the traditional eye theory, the level of the interocular inhibition introduced here is after binocular convergence at the stage solving the correspondence problem, and thus open to pattern-specific mechanisms.

Computations in the early visual cortex

This paper reviews some of the recent neurophysiological studies that explore the variety of visual computations in the early visual cortex in relation to geometric inference, i.e. the inference of contours, surfaces and shapes. It attempts to draw connections between ideas from computational vision and findings from awake primate electrophysiology. In the classical feed-forward, modular view of visual processing, the early visual areas (LGN, V1 and V2) are modules that serve to extract local features, while higher extrastriate areas are responsible for shape inference and invariant object recognition. However, recent findings in primate early visual systems reveal that the computations in the early visual cortex are rather complex and dynamic, as well as interactive and plastic, subject to influence from global context, higher order perceptual inference, task requirement and behavioral experience. The evidence argues that the early visual cortex does not merely participate in the first stage of visual processing, but is involved in many levels of visual computation.

Stochastic artificial retinas: algorithm, optoelectronic circuits, and implementation

An analogy can be established between image processing and statistical mechanics. Many early- and intermediate-vision tasks such as restoration, image segmentation, and motion detection can be formulated as optimization problems that consist in finding the ground states of an energy function. This approach yields excellent results, but, once it is implemented in conventional sequential workstations, the computational loads are too extensive for practical purposes, and even fast suboptimal optimization approaches are not sufficient. We elaborate on dedicated massively-parallel integrated circuits, called stochastic artificial retinas, that minimize the energy function at a video rate. We consider several components of these artificial retinas: stochastic algorithms for restoration tasks in the presence of discontinuities, dedicated optoelectronic hardware to implement thermal motion by photodetection of speckles, and hybrid architectures that combine optoelectronic, asynchronous-analog, and clocked-digital circuits.

Attractor dynamics in feedforward neural networks

We study the probabilistic generative models parameterized by feedforward neural networks. An attractor dynamics for probabilistic inference in these models is derived from a mean field approximation for large, layered sigmoidal networks. Fixed points of the dynamics correspond to solutions of the mean field equations, which relate the statistics of each unit to those of its Markov blanket. We establish global convergence of the dynamics by providing a Lyapunov function and show that the dynamics generate the signals required for unsupervised learning. Our results for feedforward networks provide a counterpart to those of Cohen-Grossberg and Hopfield for symmetric networks.

Stereo algorithm that extracts a depth cue from interocularly unpaired points

In binocular vision, interocularly unpaired points are often generated as a result of occlusions. These points can only produce false matches, and have no information about binocular disparities. However, recent psychophysical experiments have suggested that interocularly unpaired points play an important role in human stereo perception. This article introduces a stereo algorithm that can extract depth cues from not only interocularly paired but also unpaired points. Computer simulation demonstrates that our algorithm can correctly process the da Vinci stereogram as well as natural stereo pairs.

The role of the primary visual cortex in higher level vision

In the classical feed-forward, modular view of visual processing, the primary visual cortex (area V1) is a module that serves to extract local features such as edges and bars. Representation and recognition of objects are thought to be functions of higher extrastriate cortical areas. This paper presents neurophysiological data that show the later part of V1 neurons' responses reflecting higher order perceptual computations related to Ullman's (Cognition 1984; 18:97-159) visual routines and Marr's (Vision NJ: Freeman 1982) full primal sketch, 2 1/2D sketch and 3D model. Based on theoretical reasoning and the experimental evidence, we propose a possible reinterpretation of the functional role of V1. In this framework, because of V1 neurons' precise encoding of orientation and spatial information, higher level perceptual computations and representations that involve high resolution details, fine geometry and spatial precision would necessarily involve V1 and be reflected in the later part of its neurons' activities.

Do neurons have a voltage or a current threshold for action potential initiation?

The majority of neural network models consider the output of single neurons to be a continuous, positive, and saturating firing rate f (t), while a minority treat neuronal output as a series of delta pulses sigma delta (t-ti). We here argue that the issue of the proper output representation relates to the biophysics of the cells in question and, in particular, to whether initiation of somatic action potentials occurs when a certain threshold voltage or a threshold current is exceeded. We approach this issue using numerical simulations of the electrical behavior of a layer 5 pyramidal cell from cat visual cortex. The dendritic tree is passive while the cell body includes eight voltage- and calcium-dependent membrane conductances. We compute both the steady-state (Istatic(infinity)(Vm)) and the instantaneous (I0(Vm)) I-V relationships and argue that the amplitude of the local maximum in Istatic(infinity)(Vm) corresponds to the current threshold Ith for sustained inputs, while the location of the middle zero-crossing of I0 corresponds to a fixed voltage threshold Vth for rapid inputs. We confirm this using numerical simulations: for "rapid" synaptic inputs, spikes are initiated if the somatic potential exceeds Vth, while for slowly varying input Ith must be exceeded. Due to the presence of the large dendritic tree, no charge threshold Qth exists for physiological input. Introducing the temporal average of the somatic membrane potential <Vm> while the cell is spiking repetitively, allows us to define a dynamic I-V relationship Idynamic(infinity)(<Vm>). We find an exponential relationship between <Vm> and the net current sunk by the somatic membrane during spiking (diode-like behavior). The slope of Idynamic(infinity)(<Vm>) allows us to define a dynamic input conductance and a time constant that characterizes how rapidly the cell changes its output firing frequency in response to a change in its input.

Statistical Physics Algorithms That Converge

In recent years there has been significant interest in adapting techniques from statistical physics, in particular mean field theory, to provide deterministic heuristic algorithms for obtaining approximate solutions to optimization problems. Although these algorithms have been shown experimentally to be successful there has been little theoretical analysis of them. In this paper we demonstrate connections between mean field theory methods and other approaches, in particular, barrier function and interior point methods. As an explicit example, we summarize our work on the linear assignment problem. In this previous work we defined a number of algorithms, including deterministic annealing, for solving the assignment problem. We proved convergence, gave bounds on the convergence times, and showed relations to other optimization algorithms.

Three-dimensional illusory contours and surfaces

Under general viewing conditions, objects are often partially camouflaged, obscured or occluded, thereby limiting information about their three-dimensional position, orientation and shape to incomplete and variable image cues. When presented with such partial cues, observers report perceiving 'illusory' contours and surfaces (forms) in regions having no physical image contrast. Here we report that three-dimensional illusory forms share three fundamental properties with 'real' forms: (1) the same forms are perceived using either stereo or motion parallax cues (cue invariance); (2) they retain their shape over changes in position and orientation relative to an observer (view stability); and (3) they can take the shape of general contours and surfaces in three dimensions (morphic generality). We hypothesize that illusory contours and surfaces are manifestations of a previously unnoticed visual process which constructs a representation of three-dimensional position, orientation and shape of objects from available image cues.

The plastic coupled map lattice: A novel image-processing paradigm

Coupled map lattices (CML) can describe many relaxation and optimization algorithms currently used in image processing. We recently introduced the "plastic-CML" as a paradigm to extract (segment) objects in an image. Here, the image is applied by a set of forces to a metal sheet which is allowed to undergo plastic deformation parallel to the applied forces. In this paper we present an analysis of our "plastic-CML" in one and two dimensions, deriving the nature and stability of its stationary solutions. We also detail how to use the CML in image processing, how to set the system parameters and present examples of it at work. We conclude that the plastic-CML is able to segment images with large amounts of noise and large dynamic range of pixel values, and is suitable for a very large scale integration (VLSI) implementation.

The synaptic organization of the prepacemaker nucleus in weakly electric knifefish, Eigenmannia: a quantitative ultrastructural study

Weakly electric knifefish (Eigenmannia sp.) produce continuous electric organ discharges at very constant frequencies. Modulations of the discharges occur during social interactions and are under control of the diencephalic prepacemaker nucleus. Abrupt frequency modulations, or 'chirps', which are observed predominantly during the breeding season, can be elicited by stimulation of neurons in a ventro-lateral portion of the prepacemaker nucleus, the so-called PPn-C. The PPn-C consists of approximately 100 loosely scattered large multipolar neurons which send dendrites into three territories, called 'dorso-medial', 'dorso-lateral', and 'ventral'. In the present ultrastructural investigation, the synaptic organization of these neurons, identified by retrograde labelling with horseradish peroxidase, was studied quantitatively. Somata and dendrites of the PPn-C receive input from two classes of chemical synapses. Class-1 boutons contain predominantly agranular, round vesicles and are believed to be excitatory. Class-2 boutons display predominantly flattened or pleiomorphic vesicles and are probably inhibitory. The action of the agranular vesicles in the synaptic boutons of these two classes may be modulated by the content of large dense-core vesicles. These comprise approximately 1% of the total vesicle population and are found predominantly in regions distant from the active zone of the synaptic bouton. The density of chemical synapses exhibits marked topographic differences. Class-1 boutons occur typically at densities of 3-12 synapses per 100 microns of profile length on dendrites and cell bodies. No significant differences in density of class-1 boutons could be found between distal dendrites of the three territories, proximal dendrites and cell bodies. The density of class-2 synapses, on the other hand, increases significantly from usually less than 1 synapse per 100 microns of profile length on distal dendrites to 2-3 synapses per 100 microns of profile length on proximal dendrites and cell bodies. Such a topographic organization could enable the proximal elements to 'veto' the depolarizing response of distal dendrites to excitatory inputs. The growth of dendrites in the dorso-medial territory during the breeding season, as shown in a previous study, and the concurrent doubling of excitatory input received by class-1 synapses, could overcome the inhibition caused on somata and proximal dendrites by class-2 synapses and thus account for the dramatic increase in the fish's propensity to chirp in the context of sexual maturity.

An overview of neural networks

Some of the world's leading researchers in neural networks submitted their most recent results concerning their research in neural networks to the author for inclusion in this survey. Descriptive accounts of their collective papers are presented as well as a list of sources of information concerning neural networks, such as journals, books, and technical reports. The material is broken into categories related to established areas in computer science, robotics, neural modeling, and engineering.

Multiscale optimization in neural nets

One way to speed up convergence in a large optimization problem is to introduce a smaller, approximate version of the problem at a coarser scale and to alternate between relaxation steps for the fine-scale and coarse-scale problems. Such an optimization method for neural networks governed by quite general objective functions is presented. At the coarse scale, there is a smaller approximating neural net which, like the original net, is nonlinear and has a nonquadratic objective function. The transitions and information flow from fine to coarse scale and back do not disrupt the optimization, and the user need only specify a partition of the original fine-scale variables. Thus, the method can be applied easily to many problems and networks. There is generally about a fivefold improvement in estimated cost under the multiscale method. In the networks to which it was applied, a nontrivial speedup by a constant factor of between two and five was observed, independent of problem size. Further improvements in computational cost are very likely to be available, especially for problem-specific multiscale neural net methods.

A two-dimensional analog VLSI circuit for detecting discontinuities in early vision

A large number of computer vision algorithms for finding intensity edges, computing motion, depth, and color, and recovering the three-dimensional shape of objects have been developed within the framework of minimizing an associated "energy" or "cost" functional. Particularly successful has been the introduction of binary variables coding for discontinuities in intensity, optical flow field, depth, and other variables, allowing image segmentation to occur in these modalities. The associated nonconvex variational functionals can be mapped onto analog, resistive networks, such that the stationary voltage distribution in the network corresponds to a minimum of the functional. The performance of an experimental analog very-large-scale integration (VLSI) circuit implementing the nonlinear resistive network for the problem of two-dimensional surface interpolation in the presence of discontinuities is demonstrated; this circuit is implemented in complementary metal oxide semiconductor technology.

Seeing Chips: Analog VLSI Circuits for Computer Vision

Vision is simple. We open our eyes and, instantly, the world surrounding us is perceived in all its splendor. Yet Artificial Intelligence has been trying with very limited success for over 20 years to endow machines with similar abilities. A large van, filled with computers and driving unguided at a mile per hour across gently sloping hills in Colorado and using a laser-range system to “see” is the most we have accomplished so far. On the other hand, computers can play a decent game of chess or prove simple mathematical theorems. It is ironic that we are unable to reproduce perceptual abilities which we share with most animals while some of the features distinguishing us from even our closest cousins, chimpanzees, can be carried out by machines. Vision is difficult.