How the forest interacts with the trees: Multiscale shape integration explains global and local processing

Visual homogeneity computations in the brain enable solving property-based visual tasks

Most visual tasks involve looking for specific object features. But we also often perform property-based tasks where we look for specific property in an image, such as finding an odd item, deciding if two items are same, or if an object has symmetry. How do we solve such tasks? These tasks do not fit into standard models of decision making because their underlying feature space and decision process is unclear. Using well-known principles governing multiple object representations, we show that displays with repeating elements can be distinguished from heterogeneous displays using a property we define as visual homogeneity. In behavior, visual homogeneity predicted response times on visual search, same-different and symmetry tasks. Brain imaging during visual search and symmetry tasks revealed that visual homogeneity was localized to a region in the object-selective cortex. Thus, property-based visual tasks are solved in a localized region in the brain by computing visual homogeneity.

The time course of categorical and perceptual similarity effects in visual search

During visual search for objects (e.g., an apple), the surrounding distractor objects may share perceptual (tennis ball), categorical (banana), or both (peach) properties with the target. Previous studies showed that the perceptual similarity between target and distractor objects influences visual search. Here, we tested whether categorical target-distractor similarity also influences visual search, and how this influence depends on perceptual similarity. By orthogonally manipulating categorical and perceptual target-distractor similarity, we could investigate how and when the two similarities interactively affect search performance and neural correlates of spatial attention (N2pc) using electroencephalography (EEG). Behavioral results showed that categorical target-distractor similarity interacted with perceptual target-distractor similarity, such that the effect of categorical similarity was strongest when target and distractor objects were perceptually similar. EEG results showed that perceptual similarity influenced the early part of the N2pc (200-250 ms after stimulus onset), while categorical similarity influenced the later part (250-300 ms). Mirroring the behavioral results, categorical similarity interacted with perceptual similarity during this later time window, with categorical effects only observed for perceptually similar target-distractor pairs. Together, these results provide evidence for hierarchical processing in visual search: categorical properties influence spatial attention only when perceptual properties are insufficient to guide attention to the target. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Trailblazers in Neuroscience: Using compositionality to understand how parts combine in whole objects

A fundamental question for any visual system is whether its image representation can be understood in terms of its components. Decomposing any image into components is challenging because there are many possible decompositions with no common dictionary, and enumerating them leads to a combinatorial explosion. Even in perception, many objects are readily seen as containing parts, but there are many exceptions. These exceptions include objects that are not perceived as containing parts, properties like symmetry that cannot be localized to any single part, and also special categories like words and faces whose perception is widely believed to be holistic. Here, I describe a novel approach we have used to address these issues and evaluate compositionality at the behavioral and neural levels. The key design principle is to create a large number of objects by combining a small number of pre-defined components in all possible ways. This allows for building component-based models that explain whole objects using a combination of these components. Importantly, any systematic error in model fits can be used to detect the presence of emergent or holistic properties. Using this approach, we have found that whole object representations are surprisingly predictable from their components, that some components are preferred to others in perception, and that emergent properties can be discovered or explained using compositional models. Thus, compositionality is a powerful approach for understanding how whole objects relate to their parts.

Qualitative similarities and differences in visual object representations between brains and deep networks

Deep neural networks have revolutionized computer vision, and their object representations across layers match coarsely with visual cortical areas in the brain. However, whether these representations exhibit qualitative patterns seen in human perception or brain representations remains unresolved. Here, we recast well-known perceptual and neural phenomena in terms of distance comparisons, and ask whether they are present in feedforward deep neural networks trained for object recognition. Some phenomena were present in randomly initialized networks, such as the global advantage effect, sparseness, and relative size. Many others were present after object recognition training, such as the Thatcher effect, mirror confusion, Weber's law, relative size, multiple object normalization and correlated sparseness. Yet other phenomena were absent in trained networks, such as 3D shape processing, surface invariance, occlusion, natural parts and the global advantage. These findings indicate sufficient conditions for the emergence of these phenomena in brains and deep networks, and offer clues to the properties that could be incorporated to improve deep networks.