How biological vision succeeds in the physical world

The spike gating flow: A hierarchical structure-based spiking neural network for online gesture recognition

Action recognition is an exciting research avenue for artificial intelligence since it may be a game changer in emerging industrial fields such as robotic visions and automobiles. However, current deep learning (DL) faces major challenges for such applications because of the huge computational cost and inefficient learning. Hence, we developed a novel brain-inspired spiking neural network (SNN) based system titled spiking gating flow (SGF) for online action learning. The developed system consists of multiple SGF units which are assembled in a hierarchical manner. A single SGF unit contains three layers: a feature extraction layer, an event-driven layer, and a histogram-based training layer. To demonstrate the capability of the developed system, we employed a standard dynamic vision sensor (DVS) gesture classification as a benchmark. The results indicated that we can achieve 87.5% of accuracy which is comparable with DL, but at a smaller training/inference data number ratio of 1.5:1. Only a single training epoch is required during the learning process. Meanwhile, to the best of our knowledge, this is the highest accuracy among the non-backpropagation based SNNs. Finally, we conclude the few-shot learning (FSL) paradigm of the developed network: 1) a hierarchical structure-based network design involves prior human knowledge; 2) SNNs for content-based global dynamic feature detection.

On the synthesis of visual illusions using deep generative models

Visual illusions expand our understanding of the visual system by imposing constraints in the models in two different ways: i) visual illusions for humans should induce equivalent illusions in the model, and ii) illusions synthesized from the model should be compelling for human viewers too. These constraints are alternative strategies to find good vision models. Following the first research strategy, recent studies have shown that artificial neural network architectures also have human-like illusory percepts when stimulated with classical hand-crafted stimuli designed to fool humans. In this work we focus on the second (less explored) strategy: we propose a framework to synthesize new visual illusions using the optimization abilities of current automatic differentiation techniques. The proposed framework can be used with classical vision models as well as with more recent artificial neural network architectures. This framework, validated by psychophysical experiments, can be used to study the difference between a vision model and the actual human perception and to optimize the vision model to decrease this difference.

Unity Human Eye Model for Gaze Tracking with a Query-Driven Dynamic Vision Sensor

Objective measurement of gaze pattern and eye movement during untethered activity has important applications for neuroscience research and neurological disease detection. Current commercial eye-tracking tools rely on desk-top devices with infrared emitters and conventional frame-based cameras. Although wearable options do exist, the large power-consumption from their conventional cameras limit true long-term mobile usage. The query-driven Dynamic Vision Sensor (qDVS) is a neuromorphic camera which dramatically reduces power consumption by outputting only intensity-change threshold events, as opposed to full frames of intensity data. However, such hardware has not yet been implemented for on-body eye-tracking, but the feasibility can be demonstrated using a mathematical simulator to evaluate the eye-tracking ca-pabilities of the qDVS under controlled conditions. Specifically, a framework utilizing a realistic human eye model in the 3D graphics engine, Unity, is presented to enable the controlled and direct comparison of image-based gaze tracking methods. Eye-tracking based on qDVS frames was compared against two different conventional frame eye-tracking methods - the traditional ellipse pupil-fitting algorithm and a deep learning neural network inference model. Gaze accuracy from qDVS frames achieved an average of 93.2% for movement along the primary horizontal axis (pitch angle) and 93.1 % for movement along the primary vertical axis (yaw angle) under 4 different illumination conditions, demonstrating the feasibility for using qDVS hardware cameras for such applications. The quantitative framework for the direct comparison of eye tracking algorithms presented here is made open-source and can be extended to include other eye parameters, such as pupil dilation, reflection, motion artifact, and more.

The Eye Pupil Adjusts to Illusorily Expanding Holes

Some static patterns evoke the perception of an illusory expanding central region or “hole.” We asked observers to rate the magnitudes of illusory motion or expansion of black holes, and these predicted the degree of dilation of the pupil, measured with an eye tracker. In contrast, when the “holes” were colored (including white), i.e., emitted light, these patterns constricted the pupils, but the subjective expansions were also weaker compared with the black holes. The change rates of pupil diameters were significantly related to the illusory motion phenomenology only with the black holes. These findings can be accounted for within a perceiving-the-present account of visual illusions, where both the illusory motion and the pupillary adjustments represent compensatory mechanisms to the perception of the next moment, based on shared experiences with the ecological regularities of light.

Self-operated stimuli improve subsequent visual motion integration

Evidences of perceptual changes that accompany motor activity have been limited primarily to audition and somatosensation. Here we asked whether motor learning results in changes to visual motion perception. We designed a reaching task in which participants were trained to make movements along several directions, while the visual feedback was provided by an intrinsically ambiguous moving stimulus directly tied to hand motion. We find that training improves coherent motion perception and that changes in movement are correlated with perceptual changes. No perceptual changes are observed in passive training even when observers were provided with an explicit strategy to facilitate single motion perception. A Bayesian model suggests that movement training promotes the fine-tuning of the internal representation of stimulus geometry. These results emphasize the role of sensorimotor interaction in determining the persistent properties in space and time that define a percept.

Global and local interference effects in ensemble encoding are best explained by interactions between summary representations of the mean and the range

Through ensemble encoding, the visual system compresses redundant statistical properties from multiple items into a single summary metric (e.g., average size). Numerous studies have shown that global summary information is extracted quickly, does not require access to single-item representations, and often interferes with reports of single items from the set. Yet a thorough understanding of ensemble processing would benefit from a more extensive investigation at the local level. Thus, the purpose of this study was to provide a more critical inspection of global-local processing in ensemble perception. Taking inspiration from Navon (Cognitive Psychology, 9(3), 353-383, 1977), we employed a novel paradigm that independently manipulates the degree of interference at the global (mean) or local (single item) level of the ensemble. Initial results were consistent with reciprocal interference between global and local ensemble processing. However, further testing revealed that local interference effects were better explained by interference from another summary statistic, the range of the set. Furthermore, participants were unable to disambiguate single items from the ensemble display from other items that were within the ensemble range but, critically, were not actually present in the ensemble. Thus, it appears that local item values are likely inferred based on their relationship to higher-order summary statistics such as the range and the mean. These results conflict with claims that local information is captured alongside global information in summary representations. In such studies, successful identification of set members was not compared with misidentification of items within the range, but which were nevertheless not presented within the set.

Sustained representation of perspectival shape

Arguably the most foundational principle in perception research is that our experience of the world goes beyond the retinal image; we perceive the distal environment itself, not the proximal stimulation it causes. Shape may be the paradigm case of such "unconscious inference": When a coin is rotated in depth, we infer the circular object it truly is, discarding the perspectival ellipse projected on our eyes. But is this really the fate of such perspectival shapes? Or does a tilted coin retain an elliptical appearance even when we know it's circular? This question has generated heated debate from Locke and Hume to the present; but whereas extant arguments rely primarily on introspection, this problem is also open to empirical test. If tilted coins bear a representational similarity to elliptical objects, then a circular coin should, when rotated, impair search for a distal ellipse. Here, nine experiments demonstrate that this is so, suggesting that perspectival shapes persist in the mind far longer than traditionally assumed. Subjects saw search arrays of three-dimensional "coins," and simply had to locate a distally elliptical coin. Surprisingly, rotated circular coins slowed search for elliptical targets, even when subjects clearly knew the rotated coins were circular. This pattern arose with static and dynamic cues, couldn't be explained by strategic responding or unfamiliarity, generalized across shape classes, and occurred even with sustained viewing. Finally, these effects extended beyond artificial displays to real-world objects viewed in naturalistic, full-cue conditions. We conclude that objects have a remarkably persistent dual character: their objective shape "out there," and their perspectival shape "from here."

Deciphering image contrast in object classification deep networks

The ultimate goal of neuroscience is to explain how complex behaviour arises from neuronal activity. A comparable level of complexity also emerges in deep neural networks (DNNs) while exhibiting human-level performance in demanding visual tasks. Unlike in biological systems, all parameters and operations of DNNs are accessible. Therefore, in theory, it should be possible to decipher the exact mechanisms learnt by these artificial networks. Here, we investigate the concept of contrast invariance within the framework of DNNs. We start by discussing how a network can achieve robustness to changes in local and global image contrast. We used a technique from neuroscience-"kernel lesion"-to measure the degree of performance degradation when individual kernels are eliminated from a network. We further compared contrast normalisation, a mechanism used in biological systems, to the strategies that DNNs learn to cope with changes of contrast. The results of our analysis suggest that (i) contrast is a low-level feature for these networks, and it is encoded in the shallow layers; (ii) a handful of kernels appear to have a greater impact on this feature, and their removal causes a substantially larger accuracy loss for low-contrast images; (iii) edges are a distinct visual feature within the internal representation of object classification DNNs.

An Alternative Theory of Binocularity

The fact that seeing with two eyes is universal among vertebrates raises a problem that has long challenged vision scientists: how do animals with overlapping visual fields combine non-identical right and left eye images to achieve fusion and the perception of depth that follows? Most theories address this problem in terms of matching corresponding images on the right and left retinas. Here we suggest an alternative theory of binocular vision based on anatomical correspondence that circumvents the correspondence problem and provides a rationale for ocular dominance.

Colorful glares: Effects of colors on brightness illusions measured with pupillometry

We hypothesized that pupil constrictions to the glare illusion, where converging luminance gradients subjectively enhance the perception of brightness, would be stronger for 'blue' than for other colors. Such an expectation was based on reflections about the ecology of vision, where the experience of dazzling light is common when one happens to look directly at sunlight through some occluders. Thus, we hypothesized that pupil constrictions to 'blue' reflect an ecologically-based expectation of the visual system from the experience of sky's light and color, which also leads to interpret the blue gradients of illusory glare to act as effective cues to impending probable intense light. We therefore manipulated the gradients color of glare illusions and measured changes in subjective brightness of identical shape stimuli. We confirmed that the blue resulted in what was subjectively evaluated as the brightest condition, despite all colored stimuli were equiluminant. This enhanced brightness effect was observed both in a psychophysical adjustment task and in changes in pupil size, where the maximum pupil constriction peak was observed with the 'blue' converging gradients over and above to the pupil response to blue in other conditions (i.e., diverging gradients and homogeneous patches). Moreover, glare-related pupil constrictions for each participant were correlated to each individual's subjective brightness adjustments. Homogenous blue hues also constricted the pupil more than other hues, which represents a pupillometric analog of the Helmholtz-Kohlrausch effect on brightness perception. Together, these findings show that pupillometry constitutes an easy tool to assess individual differences in color brightness perception.

Rationality, perception, and the all-seeing eye

Seeing-perception and vision-is implicitly the fundamental building block of the literature on rationality and cognition. Herbert Simon and Daniel Kahneman's arguments against the omniscience of economic agents-and the concept of bounded rationality-depend critically on a particular view of the nature of perception and vision. We propose that this framework of rationality merely replaces economic omniscience with perceptual omniscience. We show how the cognitive and social sciences feature a pervasive but problematic meta-assumption that is characterized by an "all-seeing eye." We raise concerns about this assumption and discuss different ways in which the all-seeing eye manifests itself in existing research on (bounded) rationality. We first consider the centrality of vision and perception in Simon's pioneering work. We then point to Kahneman's work-particularly his article "Maps of Bounded Rationality"-to illustrate the pervasiveness of an all-seeing view of perception, as manifested in the extensive use of visual examples and illusions. Similar assumptions about perception can be found across a large literature in the cognitive sciences. The central problem is the present emphasis on inverse optics-the objective nature of objects and environments, e.g., size, contrast, and color. This framework ignores the nature of the organism and perceiver. We argue instead that reality is constructed and expressed, and we discuss the species-specificity of perception, as well as perception as a user interface. We draw on vision science as well as the arts to develop an alternative understanding of rationality in the cognitive and social sciences. We conclude with a discussion of the implications of our arguments for the rationality and decision-making literature in cognitive psychology and behavioral economics, along with suggesting some ways forward.

The Eye Pupil's Response to Static and Dynamic Illusions of Luminosity and Darkness

Pupil diameters were recorded with an eye-tracker while participants observed cruciform patterns of gray-scale gradients that evoked illusions of enhanced brightness (glare) or of enhanced darkness. The illusions were either presented as static images or as dynamic animations which initially appeared as a pattern of filled squares that—in a few seconds—gradually changed into gradients until the patterns were identical to the static ones. Gradients could either converge toward the center, resulting in a central region of enhanced, illusory, brightness or darkness, or oriented toward each side of the screen, resulting in the perception of a peripheral ring of illusory brightness or darkness. It was found that pupil responses to these illusions matched both the direction and intensity of perceived changes in light: Glare stimuli resulted in pupil constrictions, and darkness stimuli evoked dilations of the pupils. A second experiment found that gradients of brightness were most effective in constricting the pupils than isoluminant step-luminance, local, variations in luminance. This set of findings suggest that the eye strategically adjusts to reflect in a predictive manner, given that these brightness illusions only suggest a change in luminance when none has occurred, the content within brightness maps of the visual scene.

Perceptual distortions and deceptions: what computers can teach us

The nature of perception has fascinated philosophers for centuries, and has more recently been the focus of research in psychology and neuroscience. Many psychiatric disorders are characterised by perceptual abnormalities, ranging from sensory distortions to illusions and hallucinations. The distinction between normal and abnormal perception is, however, hard to articulate. In this article we argue that the distinction between normal perception and abnormal perception is best seen as a quantitative one, resting on the degree to which the observer's prior expectations influence perceptual inference. We illustrate this point with an example taken from researchers at Google working on computer vision.

Intrinsic spatial knowledge about terrestrial ecology favors the tall for judging distance

Our sense of vision reliably directs and guides our everyday actions, such as reaching and walking. This ability is especially fascinating because the optical images of natural scenes that project into our eyes are insufficient to adequately form a perceptual space. It has been proposed that the brain makes up for this inadequacy by using its intrinsic spatial knowledge. However, it is unclear what constitutes intrinsic spatial knowledge and how it is acquired. We investigated this question and showed evidence of an ecological basis, which uses the statistical spatial relationship between the observer and the terrestrial environment, namely, the ground surface. We found that in dark and reduced-cue environments where intrinsic knowledge has a greater contribution, perceived target location is more accurate when referenced to the ground than to the ceiling. Furthermore, taller observers more accurately localized the target. Superior performance was also observed in the full-cue environment, even when we compensated for the observers' heights by having the taller observer sit on a chair and the shorter observers stand on a box. Although fascinating, this finding dovetails with the prediction of the ecological hypothesis for intrinsic spatial knowledge. It suggests that an individual's accumulated lifetime experiences of being tall and his or her constant interactions with ground-based objects not only determine intrinsic spatial knowledge but also endow him or her with an advantage in spatial ability in the intermediate distance range.

Statistical learning is constrained to less abstract patterns in complex sensory input (but not the least)

The influence of statistical information on behavior (either through learning or adaptation) is quickly becoming foundational to many domains of cognitive psychology and cognitive neuroscience, from language comprehension to visual development. We investigate a central problem impacting these diverse fields: when encountering input with rich statistical information, are there any constraints on learning? This paper examines learning outcomes when adult learners are given statistical information across multiple levels of abstraction simultaneously: from abstract, semantic categories of everyday objects to individual viewpoints on these objects. After revealing statistical learning of abstract, semantic categories with scrambled individual exemplars (Exp. 1), participants viewed pictures where the categories as well as the individual objects predicted picture order (e.g., bird₁—dog₁, bird₂—dog₂). Our findings suggest that participants preferentially encode the relationships between the individual objects, even in the presence of statistical regularities linking semantic categories (Exps. 2 and 3). In a final experiment we investigate whether learners are biased towards learning object-level regularities or simply construct the most detailed model given the data (and therefore best able to predict the specifics of the upcoming stimulus) by investigating whether participants preferentially learn from the statistical regularities linking individual snapshots of objects or the relationship between the objects themselves (e.g., bird_picture₁— dog_picture₁, bird_picture₂—dog_picture₂). We find that participants fail to learn the relationships between individual snapshots, suggesting a bias towards object-level statistical regularities as opposed to merely constructing the most complete model of the input. This work moves beyond the previous existence proofs that statistical learning is possible at both very high and very low levels of abstraction (categories vs. individual objects) and suggests that, at least with the current categories and type of learner, there are biases to pick up on statistical regularities between individual objects even when robust statistical information is present at other levels of abstraction. These findings speak directly to emerging theories about how systems supporting statistical learning and prediction operate in our structure-rich environments. Moreover, the theoretical implications of the current work across multiple domains of study is already clear: statistical learning cannot be assumed to be unconstrained even if statistical learning has previously been established at a given level of abstraction when that information is presented in isolation.

Gravity in the Brain as a Reference for Space and Time Perception

Moving and interacting with the environment require a reference for orientation and a scale for calibration in space and time. There is a wide variety of environmental clues and calibrated frames at different locales, but the reference of gravity is ubiquitous on Earth. The pull of gravity on static objects provides a plummet which, together with the horizontal plane, defines a three-dimensional Cartesian frame for visual images. On the other hand, the gravitational acceleration of falling objects can provide a time-stamp on events, because the motion duration of an object accelerated by gravity over a given path is fixed. Indeed, since ancient times, man has been using plumb bobs for spatial surveying, and water clocks or pendulum clocks for time keeping. Here we review behavioral evidence in favor of the hypothesis that the brain is endowed with mechanisms that exploit the presence of gravity to estimate the spatial orientation and the passage of time. Several visual and non-visual (vestibular, haptic, visceral) cues are merged to estimate the orientation of the visual vertical. However, the relative weight of each cue is not fixed, but depends on the specific task. Next, we show that an internal model of the effects of gravity is combined with multisensory signals to time the interception of falling objects, to time the passage through spatial landmarks during virtual navigation, to assess the duration of a gravitational motion, and to judge the naturalness of periodic motion under gravity.

Perception and Reality: Why a Wholly Empirical Paradigm is Needed to Understand Vision

A central puzzle in vision science is how perceptions that are routinely at odds with physical measurements of real world properties can arise from neural responses that nonetheless lead to effective behaviors. Here we argue that the solution depends on: (1) rejecting the assumption that the goal of vision is to recover, however imperfectly, properties of the world; and (2) replacing it with a paradigm in which perceptions reflect biological utility based on past experience rather than objective features of the environment. Present evidence is consistent with the conclusion that conceiving vision in wholly empirical terms provides a plausible way to understand what we see and why.

Will understanding vision require a wholly empirical paradigm?

Based on electrophysiological and anatomical studies, a prevalent conception is that the visual system recovers features of the world from retinal images to generate perceptions and guide behavior. This paradigm, however, is unable to explain why visual perceptions differ from physical measurements, or how behavior could routinely succeed on this basis. An alternative is that vision does not recover features of the world, but assigns perceptual qualities empirically by associating frequently occurring stimulus patterns with useful responses on the basis of survival and reproductive success. The purpose of the present article is to briefly describe this strategy of vision and the evidence for it.

A conceptual framework of computations in mid-level vision

If a picture is worth a thousand words, as an English idiom goes, what should those words-or, rather, descriptors-capture? What format of image representation would be sufficiently rich if we were to reconstruct the essence of images from their descriptors? In this paper, we set out to develop a conceptual framework that would be: (i) biologically plausible in order to provide a better mechanistic understanding of our visual system; (ii) sufficiently robust to apply in practice on realistic images; and (iii) able to tap into underlying structure of our visual world. We bring forward three key ideas. First, we argue that surface-based representations are constructed based on feature inference from the input in the intermediate processing layers of the visual system. Such representations are computed in a largely pre-semantic (prior to categorization) and pre-attentive manner using multiple cues (orientation, color, polarity, variation in orientation, and so on), and explicitly retain configural relations between features. The constructed surfaces may be partially overlapping to compensate for occlusions and are ordered in depth (figure-ground organization). Second, we propose that such intermediate representations could be formed by a hierarchical computation of similarity between features in local image patches and pooling of highly-similar units, and reestimated via recurrent loops according to the task demands. Finally, we suggest to use datasets composed of realistically rendered artificial objects and surfaces in order to better understand a model's behavior and its limitations.

Properties of artificial neurons that report lightness based on accumulated experience with luminance

The responses of visual neurons in experimental animals have been extensively characterized. To ask whether these responses are consistent with a wholly empirical concept of visual perception, we optimized simple neural networks that responded according to the cumulative frequency of occurrence of local luminance patterns in retinal images. Based on this estimation of accumulated experience, the neuron responses showed classical center-surround receptive fields, luminance gain control and contrast gain control, the key properties of early level visual neurons determined in animal experiments. These results imply that a major purpose of pre-cortical neuronal circuitry is to contend with the inherently uncertain significance of luminance values in natural stimuli.

Properties of artificial networks evolved to contend with natural spectra

Understanding why spectra that are physically the same appear different in different contexts (color contrast), whereas spectra that are physically different appear similar (color constancy) presents a major challenge in vision research. Here, we show that the responses of biologically inspired neural networks evolved on the basis of accumulated experience with spectral stimuli automatically generate contrast and constancy. The results imply that these phenomena are signatures of a strategy that biological vision uses to circumvent the inverse optics problem as it pertains to light spectra, and that double-opponent neurons in early-level vision evolve to serve this purpose. This strategy provides a way of understanding the peculiar relationship between the objective world and subjective color experience, as well as rationalizing the relevant visual circuitry without invoking feature detection or image representation.