Retinal encoding of ultrabrief shape recognition cues

Seeing on the fly: Physiological and behavioral evidence show that space-to-space representation and processing enable fast and efficient performance by the visual system

When we view the world, our eyes saccade quickly between points of interest. Even when fixating a target our eyes are not completely at rest but execute small fixational eye movements (FEMs). That vision is not blurred despite this ever-present jitter has seemingly motivated an increasingly popular theory denying the reliance of the visual system on pure spatial processing in favor of a space-to-time mechanism generated by the eye drifting across the image. Accordingly, FEMs are not detrimental but rather essential to good visibility. However, the space-to-time theory is incompatible with physiological data showing that all information is conveyed by the short neural volleys generated when the eyes land on a target, and with our faithful perception of briefly displayed objects, during which time FEMs have no effect. Another difficulty in rejecting the idea of image representation by the locations and nature of responding cells in favor of a timecode, is that somewhere, somehow, this code must be decoded into a parallel spatial one when reaching perception. Thus, in addition to the implausibility of generating meaningful responses during retinal drift, the space-to-time hypothesis calls for replacing efficient point-to-point parallel transmission with a cumbersome, delayed, space-to-time-to-space process. A novel physiological framework is presented here wherein the ability of the visual system to quickly process information is mediated by the short, powerful neural volleys generated by the landing saccades. These volleys are necessary and sufficient for normal perception without FEMs contribution. This mechanism enables our excellent perception of brief stimuli and explains that visibility is not blurred by FEMs because they do not generate useful information.

Evaluating spatiotemporal integration of shape cues

Prior work has shown that humans can successfully identify letters that are constructed with a sparse array of dots, wherein the dot pattern reflects the strokes that would normally be used to fashion a given letter. In the present work the dots were briefly displayed, one at a time in sequence, varying the spatial order in which they were shown. A forward sequence was spatially ordered as though one were passing a stroke across the dots to connect them. Experiments compared this baseline condition to the following three conditions: a) the dot sequence was spatially ordered, but in the reverse direction from how letter strokes might normally be written; b) the dots in each stroke of the letter were displayed in a random order; c) the sequence of displayed dots were chosen for display from any location in the letter. Significant differences were found between the baseline condition and all three of the comparison conditions, with letter recognition being far worse for the random conditions than for conditions that provided consistent spatial ordering of dot sequences. These findings show that spatial order is critical for integration of shape cues that have been sequentially displayed.

New encoding concepts for shape recognition are needed

Models designed to explain how shapes are perceived and stored by the nervous system commonly emphasize encoding of contour features, especially orientation, curvature, and linear extent. A number of experiments from my laboratory provide evidence that contours deliver a multitude of location markers, and shapes can be identified when relatively few of the markers are displayed. The emphasis on filtering for orientation and other contour features has directed attention away from full and effective examination of how the location information is registered and used for summarizing shapes. Neural network (connectionist) models try to deal with location information by modifying linkage among neuronal populations through training trials. Connections that are initially diffuse and not useful in achieving recognition get eliminated or changed in strength, resulting in selective response to a given shape. But results from my laboratory, reviewed here, demonstrate that unknown shapes that are displayed only once can be identified using a matching task. These findings show that our visual system can immediately encode shape information with no requirement for training trials. This encoding might be accomplished by neuronal circuits in the retina.

Visual encoding of partial unknown shape boundaries

Prior research has found that known shapes and letters can be recognized from a sparse sampling of dots that mark locations on their boundaries. Further, unknown shapes that are displayed only once can be identified by a matching protocol, and here also, above-chance performance requires very few boundary markers. The present work examines whether partial boundaries can be identified under similar low-information conditions. Several experiments were conducted that used a match-recognition task, with initial display of a target shape followed quickly by a comparison shape. The comparison shape was either derived from the target shape or was based on a different shape, and the respondent was asked for a matching judgment, i.e., did it "match" the target shape. Stimulus treatments included establishing how density affected the probability of a correct decision, followed by assessment of how much positioning of boundary dots affected this probability. Results indicate that correct judgments were possible when partial boundaries were displayed with a sparse sampling of dots. We argue for a process that quickly registers the locations of boundary markers and distills that information into a shape summary that can be used to identify the shape even when only a portion of the boundary is represented.

Evaluating persistence of shape information using a matching protocol

Many laboratories have studied persistence of shape information, the goal being to better understand how the visual system mediates recognition of objects. Most have asked for recognition of known shapes, e.g., letters of the alphabet, or recall from an array. Recognition of known shapes requires access to long-term memory, so it is not possible to know whether the experiment is assessing short-term encoding and working memory mechanisms, or has encountered limitations on retrieval from memory stores. Here we have used an inventory of unknown shapes, wherein a string of discrete dots forms the boundary of each shape. Each was displayed as a target only once to a given respondent, with recognition being tested using a matching task. Analysis based on signal detection theory was used to provide an unbiased estimate of the probability of correct decisions about whether comparison shapes matched target shapes. Four experiments were conducted, which found the following: a) Shapes were identified with a high probability of being correct with dot densities ranging from 20% to 4%. Performance dropped only about 10% across this density range. b) Shape identification levels remained very high with up to 500 milliseconds of target and comparison shape separation. c) With one-at-a-time display of target dots, varying the total time for a given display, the proportion of correct decisions dropped only about 10% even with a total display time of 500 milliseconds. d) With display of two complementary target subsets, also varying the total time of each display, there was a dramatic decline of proportion correct that reached chance levels by 500 milliseconds. The greater rate of decline for the two-pulse condition may be due to a mechanism that registers when the number of dots is sufficient to create a shape summary. Once a summary is produced, the temporal window that allows shape information to be added may be more limited.

Shapes displayed with durations in the microsecond range do not obey Bloch's law of temporal summation

Shape patterns were displayed with simultaneous brief flashes from a light-emitting diode array. Flash durations in the microsecond range and luminous intensities were adjusted to vary the degree of successful shape recognition. Four experiments were conducted to test whether Bloch's law would apply in this task. Bloch's law holds that for very brief flashes the perceptual threshold is determined by the total number of photons being delivered, i.e., there is reciprocity of intensity and duration. The present results did not find that effectiveness of flashes was based on the total quantity of photons, as predicted by Bloch's law. Additionally, the evidence points to a visual mechanism that has ultra-high temporal precision that either registers the rate of photon flux or the duration of flashes.

Evaluating the contribution of shape attributes to recognition using the minimal transient discrete cue protocol

Subjects were tested for their ability to identify objects that were represented by an array of dots that marked the major contours, usually only the outer boundary. Each dot was briefly flashed to make its position known, and a major variable was the time interval that was required to flash all the dots for a given shape. Recognition declined as the total time for display of the dot inventory was increased. Each shape was shown to a given subject only once and it was either recognized -- named - or not. Although the recorded response was binary, a large number of subjects was tested, which made it possible to derive regression functions and thus specify an intercept and slope for each shape. Shapes differed substantially in their slopes, which is likely due to the amount of redundant information provided by neighboring dots. Indices of shape attributes were also derived, specifically Attneave's indices of complexity, mean curvature, inflection count, and symmetry. Three of the four shape attributes were significantly related to intercept and slope levels, but none made a substantial contribution. This suggests that these attributes are not essential properties that define shapes and allow for recognition.

Early category-specific cortical activation revealed by visual stimulus inversion

Visual categorization may already start within the first 100-ms after stimulus onset, in contrast with the long-held view that during this early stage all complex stimuli are processed equally and that category-specific cortical activation occurs only at later stages. The neural basis of this proposed early stage of high-level analysis is however poorly understood. To address this question we used magnetoencephalography and anatomically-constrained distributed source modeling to monitor brain activity with millisecond-resolution while subjects performed an orientation task on the upright and upside-down presented images of three different stimulus categories: faces, houses and bodies. Significant inversion effects were found for all three stimulus categories between 70-100-ms after picture onset with a highly category-specific cortical distribution. Differential responses between upright and inverted faces were found in well-established face-selective areas of the inferior occipital cortex and right fusiform gyrus. In addition, early category-specific inversion effects were found well beyond visual areas. Our results provide the first direct evidence that category-specific processing in high-level category-sensitive cortical areas already takes place within the first 100-ms of visual processing, significantly earlier than previously thought, and suggests the existence of fast category-specific neocortical routes in the human brain.

A motion illusion reveals mechanisms of perceptual stabilization

Visual illusions are valuable tools for the scientific examination of the mechanisms underlying perception. In the peripheral drift illusion special drift patterns appear to move although they are static. During fixation small involuntary eye movements generate retinal image slips which need to be suppressed for stable perception. Here we show that the peripheral drift illusion reveals the mechanisms of perceptual stabilization associated with these micromovements. In a series of experiments we found that illusory motion was only observed in the peripheral visual field. The strength of illusory motion varied with the degree of micromovements. However, drift patterns presented in the central (but not the peripheral) visual field modulated the strength of illusory peripheral motion. Moreover, although central drift patterns were not perceived as moving, they elicited illusory motion of neutral peripheral patterns. Central drift patterns modulated illusory peripheral motion even when micromovements remained constant. Interestingly, perceptual stabilization was only affected by static drift patterns, but not by real motion signals. Our findings suggest that perceptual instabilities caused by fixational eye movements are corrected by a mechanism that relies on visual rather than extraretinal (proprioceptive or motor) signals, and that drift patterns systematically bias this compensatory mechanism. These mechanisms may be revealed by utilizing static visual patterns that give rise to the peripheral drift illusion, but remain undetected with other patterns. Accordingly, the peripheral drift illusion is of unique value for examining processes of perceptual stabilization.

Additional evidence that contour attributes are not essential cues for object recognition

It is believed that certain contour attributes, specifically orientation, curvature and linear extent, provide essential cues for object (shape) recognition. The present experiment examined this hypothesis by comparing stimulus conditions that differentially provided such cues. A spaced array of dots was used to mark the outside boundary of namable objects, and subsets were chosen that contained either contiguous strings of dots or randomly positioned dots. These subsets were briefly and successively displayed using an MTDC information persistence paradigm. Across the major range of temporal separation of the subsets, it was found that contiguity of boundary dots did not provide more effective shape recognition cues. This is at odds with the concept that encoding and recognition of shapes is predicated on the encoding of contour attributes such as orientation, curvature and linear extent.