Visual attention to surfaces in three-dimensional space

Concurrent attention to hetero-depth surfaces in 3-D visual space is governed by theta rhythm

When simultaneously confronted with multiple attentional targets, visual system employs a time-multiplexing approach in which each target alternates for prioritized access, a mechanism broadly known as rhythmic attentional sampling. For the past decade, rhythmic attentional sampling has received mounting support from converging behavioral and neural findings. However, so compelling are these findings that a critical test ground has been long overshadowed, namely the 3-D visual space where attention is complicated by extraction of the spatial layout of surfaces extending beyond 2-D planes. It remains unknown how attentional deployment to multiple targets is accomplished in the 3-D space. Here, we provided a time-resolved portrait of the behavioral and neural dynamics when participants concurrently attended to two surfaces defined by motion-depth conjunctions. To characterize the moment-to-moment attentional modulation effects, we measured perceptual sensitivity to the hetero-depth surface motions on a fine temporal scale and reconstructed their neural representations using a time-resolved multivariate inverted encoding model. We found that the perceptual sensitivity to the two surface motions rhythmically fluctuated over time at ~4 Hz, with one's enhancement closely tracked by the other's diminishment. Moreover, the behavioral pattern was coupled with an ongoing periodic alternation in strength between the two surface motion representations in the same frequency. Together, our findings provide the first converging evidence of an attentional "pendulum" that rhythmically traverses different stereoscopic depth planes and are indicative of a ubiquitous attentional time multiplexor based on theta rhythm in the 3-D visual space.

Representational structures as a unifying framework for attention

Our visual system consciously processes only a subset of the incoming information. Selective attention allows us to prioritize relevant inputs, and can be allocated to features, locations, and objects. Recent advances in feature-based attention suggest that several selection principles are shared across these domains and that many differences between the effects of attention on perceptual processing can be explained by differences in the underlying representational structures. Moving forward, it can thus be useful to assess how attention changes the structure of the representational spaces over which it operates, which include the spatial organization, feature maps, and object-based coding in visual cortex. This will ultimately add to our understanding of how attention changes the flow of visual information processing more broadly.

PTVR - A software in Python to make virtual reality experiments easier to build and more reproducible

Researchers increasingly use virtual reality (VR) to perform behavioral experiments, especially in vision science. These experiments are usually programmed directly in so-called game engines that are extremely powerful. However, this process is tricky and time-consuming as it requires solid knowledge of game engines. Consequently, the anticipated prohibitive effort discourages many researchers who want to engage in VR. This paper introduces the Perception Toolbox for Virtual Reality (PTVR) library, allowing visual perception studies in VR to be created using high-level Python script programming. A crucial consequence of using a script is that an experiment can be described by a single, easy-to-read piece of code, thus improving VR studies' transparency, reproducibility, and reusability. We built our library upon a seminal open-source library released in 2018 that we have considerably developed since then. This paper aims to provide a comprehensive overview of the PTVR software for the first time. We introduce the main objects and features of PTVR and some general concepts related to the three-dimensional (3D) world. This new library should dramatically reduce the difficulty of programming experiments in VR and elicit a whole new set of visual perception studies with high ecological validity.

Predicting the attention of others

As social animals, people are highly sensitive to the attention of others. Seeing someone else gaze at an object automatically draws one's own attention to that object. Monitoring the attention of others aids in reconstructing their emotions, beliefs, and intentions and may play a crucial role in social alignment. Recently, however, it has been suggested that the human brain constructs a predictive model of other people's attention that is far more involved than a moment-by-moment monitoring of gaze direction. The hypothesized model learns the statistical patterns in other people's attention and extrapolates how attention is likely to move. Here, we tested the hypothesis of a predictive model of attention. Subjects saw movies of attention displayed as a bright spot shifting around a scene. Subjects were able to correctly distinguish natural attention sequences (based on eye tracking of prior participants) from altered sequences (e.g., played backward or in a scrambled order). Even when the attention spot moved around a blank background, subjects could distinguish natural from scrambled sequences, suggesting a sensitivity to the spatial-temporal statistics of attention. Subjects also showed an ability to recognize the attention patterns of different individuals. These results suggest that people possess a sophisticated model of the normal statistics of attention and can identify deviations from the model. Monitoring attention is therefore more than simply registering where someone else's eyes are pointing. It involves predictive modeling, which may contribute to our remarkable social ability to predict the mind states and behavior of others.

Orienting attention across binocular disparity

The spatial distribution of covert visual attention following an exogenous cue is often described as a spotlight, which disregards depth. Here, we study the orienting of attention across binocular disparity, a key depth cue in primates. A small Gabor patch target was displayed at ±12-arcmin horizontal offset in each eye independently, resulting in four possible 3D locations. With some latency relative to target onset (0-300 ms), an attentional cue was displayed at one of five binocular locations, resulting in various combinations of relative azimuth (horizontal position) and disparity (depth). Observers' task was to discriminate the orientation of the target. Observers' performance decreased as the relative azimuth between the cue and the target increased. Performance also decreased with the difference in disparity, even when the azimuth remained constant. Performance varied with the delay between the cue and the target and was maximal between 100 and 150 ms. The orienting of attention in azimuth and depth followed the same time course. We mapped the 3D shape of attentional focus over time and found that the spatial envelope was approximately a Gaussian modulated in time. These results could not be explained by monocular confounds nor by eye movements. We conclude that exogenous cues direct attention not only to their visual direction but also to their depth and that binocular disparity is sufficient to define that depth. The identical time course and interaction between azimuth and depth suggest a shared mechanism, and therefore that visual attention to spatial location is an intrinsically 3D process.

Spatial attention based on 2D location and relative depth order modulates visual working memory in a 3D environment

The attentional effect on visual working memory (VWM) has been a heated research topic in the past two decades. Studies show that VWM performance for an attended memory item can be improved by cueing its two-dimensional (2D) spatial location during retention. However, few studies have investigated the effect of attentional selection on VWM in a three-dimensional setting, and it remains unknown whether depth information can produce beneficial attentional effects on 2D visual representations similar to 2D spatial information. Here we conducted four experiments, displaying memory items at various stereoscopic depth planes, and examined the retro-cue effects of four types of cues - a cue would either indicate the 2D or depth location of a memory item, and either in the form of physical (directly pointing to a location) or symbolic (numerically mapping onto a location) cues. We found that retro-cue benefits were only observed for cues directly pointing to a 2D location, whereas a null effect was observed for cues directly pointing to a depth location. However, there was a retro-cue effect when cueing the relative depth order, though the effect was weaker than that for cueing the 2D location. The selective effect on VWM based on 2D spatial attention is different from depth-based attention, and the divergence suggests that an object representation is primarily bound with its 2D spatial location, weakly bound with its depth order but not with its metric depth location. This indicates that attentional selection based on memory for depth, particularly metric depth, is ineffective.

The flickering spotlight of visual attention: Characterizing abnormal object-based attention in schizophrenia

Schizophrenia is associated with deficits in both object perception and visual attention. However, few studies in schizophrenia have investigated object-based attention, which is dissociable from other forms of visuospatial attention. Recent research in healthy populations has shown that the 'spotlight' of sustained visual attention flickers in a rhythmic, oscillatory fashion at specific frequencies in the 4-12 Hz range. In healthy samples, this oscillatory signature has been used to investigate spatiotemporal dynamics of object-based attention, showing that the attentional spotlight spreads to uncued locations within cued objects, and also periodically alternates focus between cued and uncued objects. In this study, we adapted a performance-based visual object cueing task to investigate object-based attention in individuals with a schizophrenia diagnosis and healthy controls. In controls, spatiotemporal patterns of object-based attention closely resembled those reported in previous studies of healthy individuals. In the schizophrenia group, the oscillatory signature of attention also appeared in the location of the cue and on uncued objects, similar to the effects in controls. Indeed, the oscillatory signature of attention at the spatial location of the cue was stronger in the schizophrenia group than in controls. However, attention did not spread across the cued object in schizophrenia; rather, attention appeared to remain hyperfocused at the spatial location of the cue. These findings provide the first evidence that visual attention has oscillatory characteristics in schizophrenia, as in the general population. The results also show that the fundamental process of attentional spreading which underlies object-based attention is abnormal in schizophrenia.

The early attentional pancake: Minimal selection in depth for rapid attentional cueing

There have been conflicting findings on the degree to which rapidly deployed visual attention is selective for depth, and this issue has important implications for attention models. Previous findings have attempted to find depth-based cueing effects on such attention using reaction time (RT) measures for stimuli presented in stereo goggles with a display screen. Results stemming from such approaches have been mixed, depending on whether target/distractor discrimination was required. To help clarify the existence of such depth effects, we have developed a paradigm that measures accuracy rather than RT in an immersive virtual-reality environment, providing a more appropriate context of depth. Three modified Posner Cueing paradigms were run to test for depth-specific rapid attentional selectivity. Participants fixated a cross while attempting to identify a rapidly masked black letter preceded by a red cue that could be valid in depth, side, or both. In Experiment 1a, a potent cueing effect was found for lateral cueing validity, but a weak effect was found for depth despite an extreme difference in virtual depth (1 vs. 300 m). In Experiment 1b, a near-replication of 1a, the lateral effect replicated while the depth effect did not. Finally, in Experiment 2, to increase the depth cue's effectiveness, the letter matched the cue's color, and the presentation duration was increased; however, again only a minimal depth-based cueing effect - no greater than that of Experiment 1a - was observed. Thus, we conclude that rapidly deployed attention is driven largely by spatiotopic rather than depth-based information.

The Short-Term Retention of Depth

We review research on the visual working memory for information portrayed by items arranged in depth (i.e., distance to the observer) within peri-personal space. Most items lose their metric depths within half a second, even though their identities and spatial positions are retained. The paradoxical loss of depth information may arise because visual working memory retains the depth of a single object for the purpose of actions such as pointing or grasping which usually apply to only one thing at a time.

Increasing perceptual separateness affects working memory for depth - re-allocation of attention from boundaries to the fixated center

For decades, working memory (WM) has been a heated research topic in the field of cognitive psychology. However, most studies on WM presented visual stimuli on a two-dimensional plane, rarely involving depth perception. Several previous studies have investigated how depth information is stored in WM, and found that WM for depth is even more limited in capacity and the memory performance is poor compared to visual WM. In the present study, we used a change detection task to investigate whether dissociating memory items by different visual features, thereby to increase their perceptual separateness, can improve WM performance for depth. Memory items presented at various depth planes were bound with different colors (Experiments 1 and 3) or sizes (Experiment 2). The memory performance for depth locations of visual stimuli with homogeneous and heterogeneous appearances were tested and compared. The results showed a consistent pattern that although separating items with various feature values did not affect the overall memory performance, the manipulation significantly improved memory performance for the middle depth locations but impaired the performance for the boundary locations when observers fixated at the center of the whole depth volume. The memory benefits of feature separation can be attributed to enhanced individuation of memory items, therefore facilitating a more balanced allocation of attention and memory resources.

Visual Search in 3D: Effects of Monoscopic and Stereoscopic Cues to Depth on the Validity of Feature Integration Theory and Perceptual Load Theory

Previous research has successfully used feature integration theory to operationalise the predictions of Perceptual Load Theory, while simultaneously testing the predictions of both models. Building on this work, we test the extent to which these models hold up in a 3D world. In two experiments, participants responded to a target stimulus within an array of shapes whose apparent depth was manipulated using a combination of monoscopic and stereoscopic cues. The search task was designed to test the predictions of (a) feature integration theory, as the target was identified by a single feature or a conjunction of features and embedded in search arrays of varying size, and (b) perceptual load theory, as the task included congruent and incongruent distractors presented alongside search tasks imposing high or low perceptual load. Findings from both experiments upheld the predictions of feature integration theory, regardless of 2D/3D condition. Longer search times in conditions with a combination of monoscopic and stereoscopic depth cues suggests that binding features into three-dimensional objects requires greater attentional effort. This additional effort should have implications for perceptual load theory, yet our findings did not uphold its predictions; the effect of incongruent distractors did not differ between conjunction search trials (conceptualised as high perceptual load) and feature search trials (low perceptual load). Individual differences in susceptibility to the effects of perceptual load were evident and likely explain the absence of load effects. Overall, our findings suggest that feature integration theory may be useful for predicting attentional performance in a 3D world.

Selective attention to real-world objects drives their emotional appraisal

Attentional manipulations have been shown to influence subsequent evaluations of objects and images. For example, images used as distractors in a visual search task are subsequently rated more negatively than are target images. One powerful manipulation of attention occurs when we plan and execute movements toward objects in our environment. Here, in two experiments, we show that selective attention to real-world objects subsequently improves emotional appraisal of those objects-an effect we term "target appreciation." Participants were presented with abstract images on three-dimensional objects, and were cued to either reach and grasp one of the two objects, or to respond to the cued object with a keyboard. Images presented on target objects were appraised more positively when compared with novel images. In contrast, images associated with obstacles or distractor objects were not appraised differently than novel images, despite the attentional suppression thought to be required to successfully avoid or ignore these objects. We speculate that this automatic appreciation of the objects of selective attention may be adaptive for organisms acting in complex environments.

Relation matters: relative depth order is stored in working memory for depth

Working memory is considered as a cognitive memory buffer for temporarily holding, processing, and manipulating information. Although working memory for verbal and visual information has been studied extensively in the past literature, few studies have systematically investigated how depth information is stored in working memory. Here, we show that the memory performance for detecting changes in stereoscopic depth is low when there is no change in relative depth order, and the performance is reliably better when depth order is changed. Increasing the magnitude of change only improves memory performance when depth order is kept constant. However, if depth order is changed, the performance remains high, even with a small change magnitude. Our findings suggest that relative depth order is a better indicator for working memory performance than absolute metric depth. The memory representation for individual depth is not independent, but inherently relational, revealing a fundamental organizing principle for depth information in the visual system.

Understanding of Curved Corridor Scenes Based on Projection of Spatial Right-angles

Helping mobile robots understand curved corridor scenes has considerable value in computer vision. However, due to the diversity of curved corridor scenes, such as curved structures that do not satisfy Manhattan assumption, understanding them remains a challenge. Curved non-Manhattan structures can be seen as compositions of spatial right angles projected into two dimensional projections, which may help us estimate their original posture in 3D scenes. In this paper, we presented an approach for mobile robots to understand curved corridor scenes including Manhattan and curved non-Manhattan structures, from a single image. Angle projections can be assigned to different clusters via geometric inference. Then coplanar structures can be estimated. Fold structures consisting of coplanar structures can be estimated, and curved non-Manhattan structures can be approximately represented by fold structures. Based on understanding curved non-Manhattan structures, the method is practical and efficient for a navigating mobile robot in curved corridor scenes. The algorithm requires no prior training or knowledge of the camera's internal parameters. With geometric features from a monocular camera, the method is robust to calibration errors and image noise. We compared the estimated curved layout against the ground truth and measured the percentage of pixels that were incorrectly classified. The experimental results showed that the algorithm can successfully understand curved corridor scenes including both Manhattan and curved non-Manhattan structures, meeting the requirements of robot navigation in a curved corridor environment.

Exogenous spatial attention shortens perceived depth

Although spatial attention has been found to alter the subjective appearance of visual stimuli in several perceptual dimensions, no research has explored whether exogenous spatial attention can affect depth perception, which is a fundamental dimension in perception that allows us to effectively interact with the environment. Here, we used an experimental paradigm adapted from Gobell and Carrasco (Psychological Science, 16[8], 644-651, 2005) to investigate this question. A peripheral cue preceding two line stimuli was used to direct exogenous attention to either location of the two lines. The two lines were separated by a certain relative disparity, and participants were asked to judge the perceived depth of two lines while attention was manipulated. We found that a farther stereoscopic depth at the attended location was perceived to be equally distant as a nearer depth at the unattended location. No such effect was found in a control experiment that employed a postcue paradigm, suggesting that our findings could not be attributed to response bias. Therefore, our study shows that exogenous spatial attention shortens perceived depth. The apparent change in stereoscopic depth may be regulated by a mechanism involving direct neural enhancement on those tuned to disparity, or be modulated by an attentional effect on apparent contrast. This finding shows that attention can change not only visual appearance but also the perceived spatial relation between an object and an observer.

Object-based attention generalizes to multisurface objects

When a part of an object is cued, targets presented in other locations on the same object are detected more rapidly and accurately than are targets on other objects. Often in object-based attention experiments, cues and targets appear not only on the same object but also on the same surface. In four psychophysical experiments, we examined whether the "object" of attentional selection was the entire object or one of its surfaces. In Experiment 1, facilitation effects were found for targets on uncued, adjacent surfaces on the same object, even when the cued and uncued surfaces were oriented differently in depth. This suggests that the "object-based" benefits of attention are not restricted to individual surfaces. Experiments 2a and 2b examined the interaction of perceptual grouping and object-based attention. In both experiments, cuing benefits extended across objects when the surfaces of those objects could be grouped, but the effects were not as strong as in Experiment 1, where the surfaces belonged to the same object. The cuing effect was strengthened in Experiment 3 by connecting the cued and target surfaces with an intermediate surface, making them appear to all belong to the same object. Together, the experiments suggest that the objects of attention do not necessarily map onto discrete physical objects defined by bounded surfaces. Instead, attentional selection can be allocated to perceptual groups of surfaces and objects in the same way as it can to a location or to groups of features that define a single object.

Figure-Ground Organization in Natural Scenes: Performance of a Recurrent Neural Model Compared with Neurons of Area V2

A crucial step in understanding visual input is its organization into meaningful components, in particular object contours and partially occluded background structures. This requires that all contours are assigned to either the foreground or the background (border ownership assignment). While earlier studies showed that neurons in primate extrastriate cortex signal border ownership for simple geometric shapes, recent studies show consistent border ownership coding also for complex natural scenes. In order to understand how the brain performs this task, we developed a biologically plausible recurrent neural network that is fully image computable. Our model uses local edge detector ( B ) cells and grouping ( G ) cells whose activity represents proto-objects based on the integration of local feature information. G cells send modulatory feedback connections to those B cells that caused their activation, making the B cells border ownership selective. We found close agreement between our model and neurophysiological results in terms of the timing of border ownership signals (BOSs) as well as the consistency of BOSs across scenes. We also benchmarked our model on the Berkeley Segmentation Dataset and achieved performance comparable to recent state-of-the-art computer vision approaches. Our proposed model provides insight into the cortical mechanisms of figure-ground organization.

Saturation and brightness modulate the effect of depth on visual working memory

Although previous studies show inconsistent results regarding the effect of depth perception on visual working memory (VWM), a recent finding shows that perceptually closer-in-depth items are better remembered than farther items when combining the congruent disparity and relative size cues. In this study, we employed a similar change detection paradigm to investigate the effects of saturation and brightness, alone or in combination with binocular disparity, on VWM. By varying the appearance of the memory items, we aimed to manipulate the visual salience as well as to simulate the aerial perspective cue that induces depth perception. We found that the change detection accuracy was significantly improved for brighter and more saturated items, but not for items solely with higher saturation. Additionally, combining saturation with the congruent disparity cue significantly improved memory performance for perceptually closer items over farther items. Conflicting the disparity cue with saturation eliminated the memory benefit for the closer items. These results indicate that saturation and brightness could modulate the effect of depth on VWM, and both visual salience and depth perception affect VWM possibly through a common underlying mechanism of setting priority for attentional selection.

The influence of relevant and irrelevant stereoscopic depth cues: Depth information does not always capture attention

Previous research reported ambiguous findings regarding the relationship of visuospatial attention and (stereoscopic) depth information. Some studies indicate that attention can be focused on a distinct depth plane, while other investigations revealed attentional capture from irrelevant items located in other, unattended depth planes. To evaluate whether task relevance of depth information modulates the deployment of attentional resources across depth planes, the additional singleton paradigm was adapted: Singletons defined by depth (i.e., displayed behind or in front of a central depth plane) or color (green against gray) were presented among neutral items and served as targets or (irrelevant) distractors. When participants were instructed to search for a color target, no attentional capture from irrelevant depth distractors was observed. In contrast, it took substantially longer to search for depth targets when an irrelevant distractor was presented simultaneously. Color distractors as well as depth distractors caused attentional capture, independent of the distractors' relative depth position (i.e., in front of or behind the target). However, slight differences in task performance were obtained depending on whether or not participants fixated within the target depth plane. Thus, the current findings indicate that attentional resources in general are uniformly distributed across different depth planes. Although task relevant depth singletons clearly affect the attentional system, this information might be processed subsequent to other stimulus features.

Separating memoranda in depth increases visual working memory performance

Visual working memory is the mechanism supporting the continued maintenance of information after sensory inputs are removed. Although the capacity of visual working memory is limited, memoranda that are spaced farther apart on a 2-D display are easier to remember, potentially because neural representations are more distinct within retinotopically organized areas of visual cortex during memory encoding, maintenance, or retrieval. The impact on memory of spatial separability in depth is less clear, even though depth information is essential to guiding interactions with objects in the environment. On one account, separating memoranda in depth may facilitate performance if interference between items is reduced. However, depth information must be inferred indirectly from the 2-D retinal image, and less is known about how visual cortex represents depth. Thus, an alternative possibility is that separation in depth does not attenuate between-items interference; it may even impair performance, as attention must be distributed across a larger volume of 3-D space. We tested these alternatives using a stereo display while participants remembered the colors of stimuli presented either near or far in the 2-D plane or in depth. Increasing separation in-plane and in depth both enhanced performance. Furthermore, participants who were better able to utilize stereo depth cues showed larger benefits when memoranda were separated in depth, particularly for large memory arrays. The observation that spatial separation in the inferred 3-D structure of the environment improves memory performance, as is the case in 2-D environments, suggests that separating memoranda in depth might reduce neural competition by utilizing cortically separable resources.

Binocular disparity can augment the capacity of vision without affecting subjective experience of depth

Binocular disparity results in a tangible subjective experience of three-dimensional world, but whether disparity also augments objective perceptual performance remains debated. We hypothesized that the improved coding of depth enabled by binocular disparity allows participants to individuate more objects at a glance as the objects can be more efficiently differentiated from each other and the background. We asked participants to enumerate objects in briefly presented naturalistic (Experiment 1) and artificial (Experiment 2) scenes in immersive virtual reality. This type of enumeration task yields well-documented capacity limits where up to 3-4 items can be enumerated rapidly and accurately, known as subitizing. Our results show that although binocular disparity did not yield a large general improvement in enumeration accuracy or reaction times, it improved participants' ability to process the items right after the limit of perceptual capacity. Binocular disparity also sped-up response times by 27 ms on average when artificial stimuli (cubes) were used. Interestingly, the influence of disparity on subjectively experienced depth revealed a clearly different pattern than the influence of disparity on objective performance. This suggests that the functional and subjective sides of stereopsis can be dissociated. Altogether our results suggest that binocular disparity may increase the number of items the visual system can simultaneously process. This may help animals to better resolve and track objects in complex, cluttered visual environments.

Visual Navigation Using Projection of Spatial Right-Angle In Indoor Environment

Helping robots understand indoor scenes has considerable value in computer vision. However, due to the diversity of indoor scenes, understanding them remains a big challenge. There are many spatial right-angles in indoor scenes. These spatial right-angles are projected into diverse 2D projections. These projections can be considered a composition of a pair of lines (line-pairs). Given the vanishing points (VPs), line segments can be assigned to 1 of 3 main orthogonal directions. The line-pairs (intersection of 2 lines), such that each of them converges to a different VP, are likely to be the projection of a spatial right-angle onto the image plane. These projections may enable us to estimate their original orientation and position in 3D scenes. In this paper, we presented a method to efficiently understand indoor scenes from a single image, without training or any knowledge of the camera's internal calibration. Through geometric inference of line-pairs, it is possible to find these spatial right-angle projections. Then, these projections can be assigned to different clusters, and the line that lies in the neighbor-cluster helps us estimate the layout of the indoor scene. The proposed approach required no prior training. We compared the room layout estimated by our algorithm against the room box ground truth, measuring the percentage of pixels that were correctly classified. These experiments showed that our method estimated not only room layout, but also details of the indoor scene.

Shifts of spatial attention in perceived 3-D space

Previous studies have shown that spatial attention can shift in three-dimensional (3-D) space determined by binocular disparity. Using Posner′s precueing paradigm, the current work examined whether attentional selection occurs in perceived 3-D space defined by occlusion. Experiment 1 showed that shifts of spatial attention induced by central cues between two surfaces in the left and right visual fields did not differ between the conditions when the two surfaces were located at the same or different perceptual depth. In contrast, Experiment 2 found that peripheral cues generated a stronger cue validity effect when the two surfaces were perceived at a different rather than at the same perceptual depth. The results suggest that exogenous but not endogenous attention operates in perceived 3-D space.

A recurrent neural model for proto-object based contour integration and figure-ground segregation

Visual processing of objects makes use of both feedforward and feedback streams of information. However, the nature of feedback signals is largely unknown, as is the identity of the neuronal populations in lower visual areas that receive them. Here, we develop a recurrent neural model to address these questions in the context of contour integration and figure-ground segregation. A key feature of our model is the use of grouping neurons whose activity represents tentative objects ("proto-objects") based on the integration of local feature information. Grouping neurons receive input from an organized set of local feature neurons, and project modulatory feedback to those same neurons. Additionally, inhibition at both the local feature level and the object representation level biases the interpretation of the visual scene in agreement with principles from Gestalt psychology. Our model explains several sets of neurophysiological results (Zhou et al. Journal of Neuroscience, 20(17), 6594-6611 2000; Qiu et al. Nature Neuroscience, 10(11), 1492-1499 2007; Chen et al. Neuron, 82(3), 682-694 2014), and makes testable predictions about the influence of neuronal feedback and attentional selection on neural responses across different visual areas. Our model also provides a framework for understanding how object-based attention is able to select both objects and the features associated with them.

Change detection in pictorial and solid scenes: The role of depth of field

This paper investigates the influence of depth of field on change detection in both pictorial and solid scenes. In this work, a within-subjects experiment is conducted using a flicker paradigm, with which the hit rate and response time for change detection are obtained. The results show that depth of field has effects on change detection: the hit rate is smaller and response time is longer in the scene with small depth of field than in the scene with large depth of field or uniform blur. It is concluded that when depth of field is small and binocular disparity is not zero in a picture, the influence of depth of field on change detection is more significant than binocular disparity. This conclusion leads to the result that the change in the sharp area is detected easier and faster than in the area that is closer to the observer.

Evidence for the effect of depth on visual working memory

Visual working memory (VWM) is a cognitive memory buffer for temporarily holding, processing, and manipulating visual information. Previous studies have demonstrated mixed results of the effect of depth perception on VWM, with some showing a beneficial effect while others not. In this study, we employed an adapted change detection paradigm to investigate the effects of two depth cues, binocular disparity and relative size. The memory array consisted of a set of pseudo-randomly positioned colored items, and the task was to judge whether the test item was changed compared to the memory item after a retention interval. We found that presenting the items in stereoscopic depth alone hardly affected VWM performance. When combining the two coherent depth cues, a significant larger VWM capacity of the perceptually closer-in-depth items was observed than that of the farther items, but the capacity for the two-depth-planes condition was not significantly different from that for the one-plane condition. Conflicting the two depth cues resulted in cancelling the beneficial effect of presenting items at a closer depth plane. The results indicate that depth perception could affect VWM, and the visual system may have an advantage in maintaining closer-in-depth objects in working memory.

Stereoscopic Segmentation Cues Improve Visual Timing Performance in Spatiotemporally Cluttered Environments

Recently, Cass and Van der Burg demonstrated that temporal order judgment (TOJ) precision could be profoundly impaired by the mere presence of dynamic visual clutter elsewhere in the visual field. This study examines whether presenting target and distractor objects in different depth planes might ameliorate this remote temporal camouflage (RTC) effect. TOJ thresholds were measured under static and dynamic (flickering) distractor conditions. In Experiment 1, targets were presented at zero, crossed, or uncrossed disparity, with distractors fixed at zero disparity. Thresholds were significantly elevated under dynamic compared with static contextual conditions, replicating the RTC effect. Crossed but not uncrossed disparity targets improved performance in dynamic distractor contexts, which otherwise produce substantial RTC. In Experiment 2, the assignment of disparity was reversed: targets fixed at zero disparity; distractors crossed, uncrossed, or zero. Under these conditions, thresholds improved significantly in the nonzero distractor disparity conditions. These results indicate that presenting target and distractor objects in different planes can significantly improve TOJ performance in dynamic conditions. In Experiment 3, targets were each presented with a different sign of disparity (e.g., one crossed and the other uncrossed), with no resulting performance benefits. Results suggest that disparity can be used to alleviate the performance-diminishing effects of RTC, but only if both targets constitute a single and unique disparity-defined surface.

Attention to Multiple Objects Facilitates Their Integration in Prefrontal and Parietal Cortex

Selective attention is known to interact with perceptual organization. In visual scenes, individual objects that are distinct and discriminable may occur on their own, or in groups such as a stack of books. The main objective of this study is to probe the neural interaction that occurs between individual objects when attention is directed toward one or more objects. Here we record steady-state visual evoked potentials via electrocorticography to directly assess the responses to individual stimuli and to their interaction. When human participants attend to two adjacent stimuli, prefrontal and parietal cortex shows a selective enhancement of only the neural interaction between stimuli, but not the responses to individual stimuli. When only one stimulus is attended, the neural response to that stimulus is selectively enhanced in prefrontal and parietal cortex. In contrast, early visual areas generally manifest responses to individual stimuli and to their interaction regardless of attentional task, although a subset of the responses is modulated similarly to prefrontal and parietal cortex. Thus, the neural representation of the visual scene as one progresses up the cortical hierarchy becomes more highly task-specific and represents either individual stimuli or their interaction, depending on the behavioral goal. Attention to multiple objects facilitates an integration of objects akin to perceptual grouping.SIGNIFICANCE STATEMENT Individual objects in a visual scene are seen as distinct entities or as parts of a whole. Here we examine how attention to multiple objects affects their neural representation. Previous studies measured single-cell or fMRI responses and obtained only aggregate measures that combined the activity to individual stimuli as well as their potential interaction. Here, we directly measure electrocorticographic steady-state responses corresponding to individual objects and to their interaction using a frequency-tagging technique. Attention to two stimuli increases the interaction component that is a hallmark for perceptual integration of stimuli. Furthermore, this stimulus-specific interaction is represented in prefrontal and parietal cortex in a task-dependent manner.

Problems with visual statistical learning in developmental dyslexia

Previous research shows that dyslexic readers are impaired in their recognition of faces and other complex objects, and show hypoactivation in ventral visual stream regions that support word and object recognition. Responses of these brain regions are shaped by visual statistical learning. If such learning is compromised, people should be less sensitive to statistically likely feature combinations in words and other objects, and impaired visual word and object recognition should be expected. We therefore tested whether people with dyslexia showed diminished capability for visual statistical learning. Matched dyslexic and typical readers participated in tests of visual statistical learning of pairs of novel shapes that frequently appeared together. Dyslexic readers on average recognized fewer pairs than typical readers, indicating some problems with visual statistical learning. These group differences were not accounted for by differences in intelligence, ability to remember individual shapes, or spatial attention paid to the stimuli, but other attentional problems could play a mediating role. Deficiencies in visual statistical learning may in some cases prevent appropriate experience-driven shaping of neuronal responses in the ventral visual stream, hampering visual word and object recognition.

Attentional Control in 3-D Space

Two experiments investigated the nature of attention in 3-D space. In Experiment 1, the hypothesis that attention can be localized to a depth plane was tested. Observers searched for a red line in two arrays of green lines. The arrays of lines were near in 2-D space but separated in depth. Search for the target was faster when the depth plane where the target would appear was cued, indicating attention can be localized in depth. A second experiment tested the hypothesis that attending to a location in depth would reduce the effect of a distracter at other depth locations. In this experiment, search for a tilted red line was faster when a distracting vertical line was present at another depth than when it was present at the same depth as the target. Implications for display design using depth information is discussed.

Visual Prediction Error Spreads Across Object Features in Human Visual Cortex

Visual cognition is thought to rely heavily on contextual expectations. Accordingly, previous studies have revealed distinct neural signatures for expected versus unexpected stimuli in visual cortex. However, it is presently unknown how the brain combines multiple concurrent stimulus expectations such as those we have for different features of a familiar object. To understand how an unexpected object feature affects the simultaneous processing of other expected feature(s), we combined human fMRI with a task that independently manipulated expectations for color and motion features of moving-dot stimuli. Behavioral data and neural signals from visual cortex were then interrogated to adjudicate between three possible ways in which prediction error (surprise) in the processing of one feature might affect the concurrent processing of another, expected feature: (1) feature processing may be independent; (2) surprise might "spread" from the unexpected to the expected feature, rendering the entire object unexpected; or (3) pairing a surprising feature with an expected feature might promote the inference that the two features are not in fact part of the same object. To formalize these rival hypotheses, we implemented them in a simple computational model of multifeature expectations. Across a range of analyses, behavior and visual neural signals consistently supported a model that assumes a mixing of prediction error signals across features: surprise in one object feature spreads to its other feature(s), thus rendering the entire object unexpected. These results reveal neurocomputational principles of multifeature expectations and indicate that objects are the unit of selection for predictive vision.

SIGNIFICANCE STATEMENT

We address a key question in predictive visual cognition: how does the brain combine multiple concurrent expectations for different features of a single object such as its color and motion trajectory? By combining a behavioral protocol that independently varies expectation of (and attention to) multiple object features with computational modeling and fMRI, we demonstrate that behavior and fMRI activity patterns in visual cortex are best accounted for by a model in which prediction error in one object feature spreads to other object features. These results demonstrate how predictive vision forms object-level expectations out of multiple independent features.

Surface Assignment Modulates Object Formation for Visual Short-Term Memory

Recent results have suggested that the operational units of visual short-term memory (VSTM) are whole objects, rather than features or the total amount of information to be remembered. Here, for the first time, the influence of surface assignment on object formation for VSTM was investigated. The observers had to memorize the features of four briefly presented (300 ms) two-part objects, followed by a mask and a cue indicating which object to report on. The experiments contrasted whether there were any apparent depth differences between the two parts of each object, and whether observers had to report on only one or both features of the post-cued target object. Depth differences induced with stereoscopic disparity, and with a pictorial depth cue (simple interposition of object features), interfered strongly with performance when both features of an object needed to be memorized, but aided performance when only a single feature needed to be remembered. Furthermore, there was considerable within-feature interference consistent with some previous findings, but contradicting others. The potential implications for conceptions of VSTM are discussed in the light of two hypothesized stages: an early feature-based stage, as well as a higher-level object-based stage where the depth manipulations exert their effects. The results argue for a strong modulatory influence of surface assignment on object formation for a VSTM task.

Testing the Coplanar Ratio Hypothesis of Lightness Perception

What determines an object's lightness remains unclear, but it is generally thought that the ratios of its luminance to the luminance of other objects in a scene play a crucial role because these ratios allow the relative reflectance of each object to be estimated, providing all the objects are under the same illumination. Because objects that lie in the same plane are typically illuminated equally, it has been suggested that it is the luminance ratios between coplanar objects that primarily determine lightness (Gilchrist, 1977 Science195 185–187; Gilchrist et al, 1999 Psychological Review106 795–834). An alternative hypothesis is that perceived illumination differences can affect lightness directly. As the studies that provided evidence for the coplanar ratio hypothesis always varied the illumination and the coplanar relationships simultaneously, it is unclear which hypothesis is correct. I measured the influence of each factor separately and found that the perceived illumination differences have a greater effect on lightness.

How Does the Cerebral Cortex Work? Development, Learning, Attention, and 3-D Vision by Laminar Circuits of Visual Cortex

A key goal of behavioral and cognitive neuroscience is to link brain mechanisms to behavioral functions. The present article describes recent progress toward explaining how the visual cortex sees. Visual cortex, like many parts of perceptual and cognitive neocortex, is organized into six main layers of cells, as well as characteristic sublamina. Here it is proposed how these layered circuits help to realize processes of development, learning, perceptual grouping, attention, and 3-D vision through a combination of bottom-up, horizontal, and top-down interactions. A main theme is that the mechanisms which enable development and learning to occur in a stable way imply properties of adult behavior. These results thus begin to unify three fields: infant cortical development, adult cortical neurophysiology and anatomy, and adult visual perception. The identified cortical mechanisms promise to generalize to explain how other perceptual and cognitive processes work.

Stimulus competition mediates the joint effects of spatial and feature-based attention

Distinct attentional mechanisms enhance the sensory processing of visual stimuli that appear at task-relevant locations and have task-relevant features. We used a combination of psychophysics and computational modeling to investigate how these two types of attention--spatial and feature based--interact to modulate sensitivity when combined in one task. Observers monitored overlapping groups of dots for a target change in color saturation, which they had to localize as being in the upper or lower visual hemifield. Pre-cues indicated the target's most likely location (left/right), color (red/green), or both location and color. We measured sensitivity (d') for every combination of the location cue and the color cue, each of which could be valid, neutral, or invalid. When three competing saturation changes occurred simultaneously with the target change, there was a clear interaction: The spatial cueing effect was strongest for the cued color, and the color cueing effect was strongest at the cued location. In a second experiment, only the target dot group changed saturation, such that stimulus competition was low. The resulting cueing effects were statistically independent and additive: The color cueing effect was equally strong at attended and unattended locations. We account for these data with a computational model in which spatial and feature-based attention independently modulate the gain of sensory responses, consistent with measurements of cortical activity. Multiple responses then compete via divisive normalization. Sufficient competition creates interactions between the two cueing effects, although the attentional systems are themselves independent. This model helps reconcile seemingly disparate behavioral and physiological findings.

A proto-object based saliency model in three-dimensional space

Most models of visual saliency operate on two-dimensional images, using elementary image features such as intensity, color, or orientation. The human visual system, however, needs to function in complex three-dimensional environments, where depth information is often available and may be used to guide the bottom-up attentional selection process. In this report we extend a model of proto-object based saliency to include depth information and evaluate its performance on three separate three-dimensional eye tracking datasets. Our results show that the additional depth information provides a small, but statistically significant, improvement in the model's ability to predict perceptual saliency (eye fixations) in natural scenes. The computational mechanisms of our model have direct neural correlates, and our results provide further evidence that proto-objects help to establish perceptual organization of the scene.

Visual search in depth: The neural correlates of segmenting a display into relevant and irrelevant three-dimensional regions

Visual perception is facilitated by the ability to selectively attend to relevant parts of the world and to ignore irrelevant regions or features. In visual search tasks, viewers are able to segment displays into relevant and irrelevant items based on a number of factors including the colour, motion, and temporal onset of the target and distractors. Understanding the process by which viewers prioritise relevant parts of a display can provide insights into the effect of top-down control on visual perception. Here, we investigate the behavioural and neural correlates of segmenting a display according to the expected three-dimensional (3D) location of a target. We ask whether this segmentation is based on low-level visual features (e.g. common depth or common surface) or on higher-order representations of 3D regions. Similar response-time benefits and neural activity were obtained when items fell on common surfaces or within depth-defined volumes, and when displays were vertical (such that items shared a common depth/disparity) or were tilted in depth. These similarities indicate that segmenting items according to their 3D location is based on attending to a 3D region, rather than a specific depth or surface. Segmenting the items in depth was mainly associated with increased activation in depth-sensitive parietal regions rather than in depth-sensitive visual regions. We conclude that segmenting items in depth is primarily achieved via higher-order, cue invariant representations rather than through filtering in lower-level perceptual regions.

Binocular fusion and invariant category learning due to predictive remapping during scanning of a depthful scene with eye movements

How does the brain maintain stable fusion of 3D scenes when the eyes move? Every eye movement causes each retinal position to process a different set of scenic features, and thus the brain needs to binocularly fuse new combinations of features at each position after an eye movement. Despite these breaks in retinotopic fusion due to each movement, previously fused representations of a scene in depth often appear stable. The 3D ARTSCAN neural model proposes how the brain does this by unifying concepts about how multiple cortical areas in the What and Where cortical streams interact to coordinate processes of 3D boundary and surface perception, spatial attention, invariant object category learning, predictive remapping, eye movement control, and learned coordinate transformations. The model explains data from single neuron and psychophysical studies of covert visual attention shifts prior to eye movements. The model further clarifies how perceptual, attentional, and cognitive interactions among multiple brain regions (LGN, V1, V2, V3A, V4, MT, MST, PPC, LIP, ITp, ITa, SC) may accomplish predictive remapping as part of the process whereby view-invariant object categories are learned. These results build upon earlier neural models of 3D vision and figure-ground separation and the learning of invariant object categories as the eyes freely scan a scene. A key process concerns how an object's surface representation generates a form-fitting distribution of spatial attention, or attentional shroud, in parietal cortex that helps maintain the stability of multiple perceptual and cognitive processes. Predictive eye movement signals maintain the stability of the shroud, as well as of binocularly fused perceptual boundaries and surface representations.

Feature attention for binocular disparity in primate area MT depends on tuning strength

Attending to a stimulus modulates the responses of sensory neurons that represent features of that stimulus, a phenomenon named "feature attention." For example, attending to a stimulus containing upward motion enhances the responses of upward-preferring direction-selective neurons in the middle temporal area (MT) and suppresses the responses of downward-preferring neurons, even when the attended stimulus is outside of the spatial receptive fields of the recorded neurons (Treue S, Martinez-Trujillo JC. Nature 399: 575-579, 1999). This modulation renders the representation of sensory information across a neuronal population more selective for the features present in the attended stimulus (Martinez-Trujillo JC, Treue S. Curr Biol 14: 744-751, 2004). We hypothesized that if feature attention modulates neurons according to their tuning preferences, it should also be sensitive to their tuning strength, which is the magnitude of the difference in responses to preferred and null stimuli. We measured how the effects of feature attention on MT neurons in rhesus monkeys (Macaca mulatta) depended on the relationship between features-in our case, direction of motion and binocular disparity-of the attended stimulus and a neuron's tuning for those features. We found that, as for direction, attention to stimuli containing binocular disparity cues modulated the responses of MT neurons and that the magnitude of the modulation depended on both a neuron's tuning preferences and its tuning strength. Our results suggest that modulation by feature attention may depend not just on which features a neuron represents but also on how well the neuron represents those features.

A computer vision system for rapid search inspired by surface-based attention mechanisms from human perception

Humans are highly efficient at visual search tasks by focusing selective attention on a small but relevant region of a visual scene. Recent results from biological vision suggest that surfaces of distinct physical objects form the basic units of this attentional process. The aim of this paper is to demonstrate how such surface-based attention mechanisms can speed up a computer vision system for visual search. The system uses fast perceptual grouping of depth cues to represent the visual world at the level of surfaces. This representation is stored in short-term memory and updated over time. A top-down guided attention mechanism sequentially selects one of the surfaces for detailed inspection by a recognition module. We show that the proposed attention framework requires little computational overhead (about 11 ms), but enables the system to operate in real-time and leads to a substantial increase in search efficiency.

Pupil size reflects the focus of feature-based attention

We measured pupil size in adult human subjects while they selectively attended to one of two surfaces, bright and dark, defined by coherently moving dots. The two surfaces were presented at the same location; therefore, subjects could select the cued surface only on the basis of its features. With no luminance change in the stimulus, we find that pupil size was smaller when the bright surface was attended and larger when the dark surface was attended: an effect of feature-based (or surface-based) attention. With the same surfaces at nonoverlapping locations, we find a similar effect of spatial attention. The pupil size modulation cannot be accounted for by differences in eye position and by other variables known to affect pupil size such as task difficulty, accommodation, or the mere anticipation (imagery) of bright/dark stimuli. We conclude that pupil size reflects not just luminance or cognitive state, but the interaction between the two: it reflects which luminance level in the visual scene is relevant for the task at hand.

Effects of spatial and feature attention on disparity-rendered structure-from-motion stimuli in the human visual cortex

An important advance in the study of visual attention has been the identification of a non-spatial component of attention that enhances the response to similar features or objects across the visual field. Here we test whether this non-spatial component can co-select individual features that are perceptually bound into a coherent object. We combined human psychophysics and functional magnetic resonance imaging (fMRI) to demonstrate the ability to co-select individual features from perceptually coherent objects. Our study used binocular disparity and visual motion to define disparity structure-from-motion (dSFM) stimuli. Although the spatial attention system induced strong modulations of the fMRI response in visual regions, the non-spatial system’s ability to co-select features of the dSFM stimulus was less pronounced and variable across subjects. Our results demonstrate that feature and global feature attention effects are variable across participants, suggesting that the feature attention system may be limited in its ability to automatically select features within the attended object. Careful comparison of the task design suggests that even minor differences in the perceptual task may be critical in revealing the presence of global feature attention.

Surface-based constraints on target selection and distractor rejection: evidence from preview search

In preview search when an observer ignores an early appearing set of distractors, there can subsequently be impeded detection of new targets that share the colour of this preview. This "negative carry-over effect" has been attributed to an active inhibitory process targeted against the old items and inadvertently their features. Here we extend negative carry-over effects to the case of stereoscopically defined surfaces of coplanar elements without common features. In Experiment 1 observers previewed distractors in one surface (1000ms), before being presented with the target and new distractors divided over the old and a new surface either above or below the old one. Participants were slower and less efficient to detect targets in the old surface. In Experiment 2 in both the first and second display the items were divided over two planes in the proportion 66/33% such that no new planes appeared following the preview, and there was no majority of items in any one plane in the final combined display. The results showed that participants were slower to detect the target when it occurred in the old majority surface. Experiment 3 held constant the 2D properties of the stimuli while varying the presence of binocular depth cues. The carry-over effect only occurred in the presence of binocular depth cues, ruling out any account of the results in terms of 2-D cues. The results suggest well formed surfaces in addition to simple features may be targets for inhibition in search.

A model of proto-object based saliency

Organisms use the process of selective attention to optimally allocate their computational resources to the instantaneously most relevant subsets of a visual scene, ensuring that they can parse the scene in real time. Many models of bottom-up attentional selection assume that elementary image features, like intensity, color and orientation, attract attention. Gestalt psychologists, however, argue that humans perceive whole objects before they analyze individual features. This is supported by recent psychophysical studies that show that objects predict eye-fixations better than features. In this report we present a neurally inspired algorithm of object based, bottom-up attention. The model rivals the performance of state of the art non-biologically plausible feature based algorithms (and outperforms biologically plausible feature based algorithms) in its ability to predict perceptual saliency (eye fixations and subjective interest points) in natural scenes. The model achieves this by computing saliency as a function of proto-objects that establish the perceptual organization of the scene. All computational mechanisms of the algorithm have direct neural correlates, and our results provide evidence for the interface theory of attention.

Visual unimodal grouping mediates auditory attentional bias in visuo-spatial working memory

Audiovisual links in spatial attention have been reported in many previous studies. However, the effectiveness of auditory spatial cues in biasing the information encoding into visuo-spatial working memory (VSWM) is still relatively unknown. In this study, we addressed this issue by combining a cuing paradigm with a change detection task in VSWM. Moreover, we manipulated the perceptual organization of the to-be-remembered visual stimuli. We hypothesized that the auditory effect on VSWM would depend on the perceptual association between the auditory cue and the visual probe. Results showed, for the first time, a significant auditory attentional bias in VSWM. However, the effect was observed only when the to-be-remembered visual stimuli were organized in two distinctive visual objects. We propose that these results shed new light on audio-visual crossmodal links in spatial attention suggesting that, apart from the spatio-temporal contingency, the likelihood of perceptual association between the auditory cue and the visual target can have a large impact on crossmodal attentional biases.