Scene complexity modulates degree of feedback activity during object detection in natural scenes

Human Visual Cortex and Deep Convolutional Neural Network Care Deeply about Object Background

Deep convolutional neural networks (DCNNs) are able to partially predict brain activity during object categorization tasks, but factors contributing to this predictive power are not fully understood. Our study aimed to investigate the factors contributing to the predictive power of DCNNs in object categorization tasks. We compared the activity of four DCNN architectures with EEG recordings obtained from 62 human participants during an object categorization task. Previous physiological studies on object categorization have highlighted the importance of figure-ground segregation-the ability to distinguish objects from their backgrounds. Therefore, we investigated whether figure-ground segregation could explain the predictive power of DCNNs. Using a stimulus set consisting of identical target objects embedded in different backgrounds, we examined the influence of object background versus object category within both EEG and DCNN activity. Crucially, the recombination of naturalistic objects and experimentally controlled backgrounds creates a challenging and naturalistic task, while retaining experimental control. Our results showed that early EEG activity (< 100 msec) and early DCNN layers represent object background rather than object category. We also found that the ability of DCNNs to predict EEG activity is primarily influenced by how both systems process object backgrounds, rather than object categories. We demonstrated the role of figure-ground segregation as a potential prerequisite for recognition of object features, by contrasting the activations of trained and untrained (i.e., random weights) DCNNs. These findings suggest that both human visual cortex and DCNNs prioritize the segregation of object backgrounds and target objects to perform object categorization. Altogether, our study provides new insights into the mechanisms underlying object categorization as we demonstrated that both human visual cortex and DCNNs care deeply about object background.

Perceptual reorganization from prior knowledge emerges late in childhood

Human vision relies heavily on prior knowledge. Here, we show for the first time that prior-knowledge-induced reshaping of visual inputs emerges gradually in late childhood. To isolate the effects of prior knowledge on perception, we presented 4- to 12-year-olds and adults with two-tone images - hard-to-recognize degraded photos. In adults, seeing the original photo triggers perceptual reorganization, causing mandatory recognition of the two-tone version. This involves top-down signaling from higher-order brain areas to early visual cortex. We show that children younger than 7-9 years do not experience this knowledge-guided shift, despite viewing the original photo immediately before each two-tone. To assess computations underlying this development, we compared human performance to three neural networks with varying architectures. The best-performing model behaved much like 4- to 5-year-olds, displaying feature-based rather than holistic processing strategies. The reconciliation of prior knowledge with sensory input undergoes a striking age-related shift, which may underpin the development of many perceptual abilities.

Independent spatiotemporal effects of spatial attention and background clutter on human object location representations

Spatial attention helps us to efficiently localize objects in cluttered environments. However, the processing stage at which spatial attention modulates object location representations remains unclear. Here we investigated this question identifying processing stages in time and space in an EEG and fMRI experiment respectively. As both object location representations and attentional effects have been shown to depend on the background on which objects appear, we included object background as an experimental factor. During the experiments, human participants viewed images of objects appearing in different locations on blank or cluttered backgrounds while either performing a task on fixation or on the periphery to direct their covert spatial attention away or towards the objects. We used multivariate classification to assess object location information. Consistent across the EEG and fMRI experiment, we show that spatial attention modulated location representations during late processing stages (>150 ms, in middle and high ventral visual stream areas) independent of background condition. Our results clarify the processing stage at which attention modulates object location representations in the ventral visual stream and show that attentional modulation is a cognitive process separate from recurrent processes related to the processing of objects on cluttered backgrounds.

Deeper neural network models better reflect how humans cope with contrast variation in object recognition

Visual inputs are far from ideal in everyday situations such as in the fog where the contrasts of input stimuli are low. However, human perception remains relatively robust to contrast variations. To provide insights about the underlying mechanisms of contrast invariance, we addressed two questions. Do contrast effects disappear along the visual hierarchy? Do later stages of the visual hierarchy contribute to contrast invariance? We ran a behavioral experiment where we manipulated the level of stimulus contrast and the involvement of higher-level visual areas through immediate and delayed backward masking of the stimulus. Backward masking led to significant drop in performance in our visual categorization task, supporting the role of higher-level visual areas in contrast invariance. To obtain mechanistic insights, we ran the same categorization task on three state-of the-art computational models of human vision each with a different depth in visual hierarchy. We found contrast effects all along the visual hierarchy, no matter how far into the hierarchy. Moreover, that final layers of deeper hierarchical models, which had been shown to be best models of final stages of the visual system, coped with contrast effects more effectively. These results suggest that, while contrast effects reach the final stages of the hierarchy, those stages play a significant role in compensating for contrast variations in the visual system.

Fronto—Parietal Regions Predict Transient Emotional States in Emotion Modulated Response Inhibition via Low Frequency and Beta Oscillations

The current study evaluated the impact of task-relevant emotion on inhibitory control while focusing on midline cortical regions rather than brain asymmetry. Single-trial time-frequency analysis of electroencephalography recordings linked with response execution and response inhibition was done while thirty-four participants performed the emotion modulated stop-signal task. To evaluate individual differences across decision-making processes involved in inhibitory control, a hierarchical drift-diffusion model was used to fit data from Go-trials for each of the 34 participants. Response threshold in the early processing stage for happy and disgust emotions could be distinguished from the later processing stage at the mid-parietal and mid-frontal regions, respectively, by the single-trial power increments in low frequency (delta and theta) bands. Beta desynchronization in the mid-frontal region was specific for differentiating disgust from neutral emotion in the early as well as later processing stages. The findings are interpreted based on the influence of emotional stimuli on early perceptual processing originating as a bottom-up process in the mid-parietal region and later proceeding to the mid-frontal region responsible for cognitive control processing, which resulted in enhanced inhibitory performance. The results show the importance of mid-frontal and mid-parietal regions in single-trial dynamics of inhibitory control processing.

How much can we differentiate at a brief glance: revealing the truer limit in conscious contents through the massive report paradigm (MRP)

Upon a brief glance, how well can we differentiate what we see from what we do not? Previous studies answered this question as 'poorly'. This is in stark contrast with our everyday experience. Here, we consider the possibility that previous restriction in stimulus variability and response alternatives reduced what participants could express from what they consciously experienced. We introduce a novel massive report paradigm that probes the ability to differentiate what we see from what we do not. In each trial, participants viewed a natural scene image and judged whether a small image patch was a part of the original image. To examine the limit of discriminability, we also included subtler changes in the image as modification of objects. Neither the images nor patches were repeated per participant. Our results showed that participants were highly accurate (accuracy greater than 80%) in differentiating patches from the viewed images from patches that are not present. Additionally, the differentiation between original and modified objects was influenced by object sizes and/or the congruence between objects and the scene gists. Our massive report paradigm opens a door to quantitatively measure the limit of immense informativeness of a moment of consciousness.

The Neural Responses of Visual Complexity in the Oddball Paradigm: An ERP Study

This research measured human neural responses to images of different visual complexity levels using the oddball paradigm to explore the neurocognitive responses of complexity perception in visual processing. In the task, 24 participants (12 females) were required to react to images with high complexity for all stimuli. We hypothesized that high-complexity stimuli would induce early visual and attentional processing effects and may elicit the visual mismatch negativity responses and the emergence of error-related negativity. Our results showed that the amplitude of P1 and N1 were unaffected by complexity in the early visual processing. Under the target stimuli, both N2 and P3b components were reported, suggesting that the N2 component was sensitive to the complexity deviation, and the attentional processing related to complexity may be derived from the occipital zone according to the feature of the P3b component. In addition, compared with the low-complexity stimulus, the high-complexity stimulus aroused a larger amplitude of the visual mismatch negativity. The detected error negativity (Ne) component reflected the error detection of the participants’ mismatch between visual complexity and psychological expectations.

The spatiotemporal neural dynamics of object location representations in the human brain

To interact with objects in complex environments, we must know what they are and where they are in spite of challenging viewing conditions. Here, we investigated where, how and when representations of object location and category emerge in the human brain when objects appear on cluttered natural scene images using a combination of functional magnetic resonance imaging, electroencephalography and computational models. We found location representations to emerge along the ventral visual stream towards lateral occipital complex, mirrored by gradual emergence in deep neural networks. Time-resolved analysis suggested that computing object location representations involves recurrent processing in high-level visual cortex. Object category representations also emerged gradually along the ventral visual stream, with evidence for recurrent computations. These results resolve the spatiotemporal dynamics of the ventral visual stream that give rise to representations of where and what objects are present in a scene under challenging viewing conditions.

Dynamic Electrode-to-Image (DETI) mapping reveals the human brain's spatiotemporal code of visual information

A number of neuroimaging techniques have been employed to understand how visual information is transformed along the visual pathway. Although each technique has spatial and temporal limitations, they can each provide important insights into the visual code. While the BOLD signal of fMRI can be quite informative, the visual code is not static and this can be obscured by fMRI’s poor temporal resolution. In this study, we leveraged the high temporal resolution of EEG to develop an encoding technique based on the distribution of responses generated by a population of real-world scenes. This approach maps neural signals to each pixel within a given image and reveals location-specific transformations of the visual code, providing a spatiotemporal signature for the image at each electrode. Our analyses of the mapping results revealed that scenes undergo a series of nonuniform transformations that prioritize different spatial frequencies at different regions of scenes over time. This mapping technique offers a potential avenue for future studies to explore how dynamic feedforward and recurrent processes inform and refine high-level representations of our visual world.

The visual information that we sample from our environment undergoes a series of neural modifications, with each modification state (or visual code) consisting of a unique distribution of responses across neurons along the visual pathway. However, current noninvasive neuroimaging techniques provide an account of that code that is coarse with respect to time or space. Here, we present dynamic electrode-to-image (DETI) mapping, an analysis technique that capitalizes on the high temporal resolution of EEG to map neural signals to each pixel within a given image to reveal location-specific modifications of the visual code. The DETI technique reveals maps of features that are associated with the neural signal at each pixel and at each time point. DETI mapping shows that real-world scenes undergo a series of nonuniform modifications over both space and time. Specifically, we find that the visual code varies in a location-specific manner, likely reflecting that neural processing prioritizes different features at different image locations over time. DETI mapping therefore offers a potential avenue for future studies to explore how each modification state informs and refines the conceptual meaning of our visual world.

On the Necessity of Recurrent Processing during Object Recognition: It Depends on the Need for Scene Segmentation

Although feedforward activity may suffice for recognizing objects in isolation, additional visual operations that aid object recognition might be needed for real-world scenes. One such additional operation is figure-ground segmentation, extracting the relevant features and locations of the target object while ignoring irrelevant features. In this study of 60 human participants (female and male), we show objects on backgrounds of increasing complexity to investigate whether recurrent computations are increasingly important for segmenting objects from more complex backgrounds. Three lines of evidence show that recurrent processing is critical for recognition of objects embedded in complex scenes. First, behavioral results indicated a greater reduction in performance after masking objects presented on more complex backgrounds, with the degree of impairment increasing with increasing background complexity. Second, electroencephalography (EEG) measurements showed clear differences in the evoked response potentials between conditions around time points beyond feedforward activity, and exploratory object decoding analyses based on the EEG signal indicated later decoding onsets for objects embedded in more complex backgrounds. Third, deep convolutional neural network performance confirmed this interpretation. Feedforward and less deep networks showed a higher degree of impairment in recognition for objects in complex backgrounds compared with recurrent and deeper networks. Together, these results support the notion that recurrent computations drive figure-ground segmentation of objects in complex scenes.SIGNIFICANCE STATEMENT The incredible speed of object recognition suggests that it relies purely on a fast feedforward buildup of perceptual activity. However, this view is contradicted by studies showing that disruption of recurrent processing leads to decreased object recognition performance. Here, we resolve this issue by showing that how object recognition is resolved and whether recurrent processing is crucial depends on the context in which it is presented. For objects presented in isolation or in simple environments, feedforward activity could be sufficient for successful object recognition. However, when the environment is more complex, additional processing seems necessary to select the elements that belong to the object and by that segregate them from the background.

Singularity and consciousness: A neuropsychological contribution

In common sense experience based on introspection, consciousness is singular. There is only one ‘me’ and that is the one that is conscious. This means that ‘singularity’ is a defining aspect of ‘consciousness’. However, the three main theories of consciousness, Integrated Information, Global Workspace and Recurrent Processing theory, are generally not very clear on this issue. These theories have traditionally relied heavily on neuropsychological observations and have interpreted various disorders, such as anosognosia, neglect and split‐brain as impairments in conscious awareness without any reference to ‘the singularity’. In this review, we will re‐examine the theoretical implications of these impairments in conscious awareness and propose a new way how to conceptualize consciousness of singularity. We will argue that the subjective feeling of singularity can coexist with several disunified conscious experiences. Singularity awareness may only come into existence due to environmental response constraints. That is, perceptual, language, memory, attentional and motor processes may largely proceed unintegrated in parallel, whereas a sense of unity only arises when organisms need to respond coherently constrained by the affordances of the environment. Next, we examine from this perspective psychiatric disorders and psycho‐active drugs. Finally, we present a first attempt to test this hypothesis with a resting state imaging experiment in a split‐brain patient. The results suggest that there is substantial coherence of activation across the two hemispheres. These data show that a complete lesioning of the corpus callosum does not, in general, alter the resting state networks of the brain. Thus, we propose that we have separate systems in the brain that generate distributed conscious. The sense of singularity, the experience of a ‘Me‐ness’, emerges in the interaction between the world and response‐planning systems, and this leads to coherent activation in the different functional networks across the cortex.

All in Good Time: Long-Lasting Postdictive Effects Reveal Discrete Perception

Is consciousness a continuous stream of percepts or is it discrete, occurring only at certain moments in time? This question has puzzled philosophers, psychologists, and neuroscientists for centuries. Both hypotheses have fallen repeatedly in and out of favor. Here, we review recent studies exploring long-lasting postdictive effects and show that the results favor a two-stage discrete model, in which substantial periods of continuous unconscious processing precede discrete conscious percepts. We propose that such a model marries the advantages of both continuous and discrete models and resolves centuries old debates about perception and consciousness.

Depth in convolutional neural networks solves scene segmentation

Feed-forward deep convolutional neural networks (DCNNs) are, under specific conditions, matching and even surpassing human performance in object recognition in natural scenes. This performance suggests that the analysis of a loose collection of image features could support the recognition of natural object categories, without dedicated systems to solve specific visual subtasks. Research in humans however suggests that while feedforward activity may suffice for sparse scenes with isolated objects, additional visual operations ('routines') that aid the recognition process (e.g. segmentation or grouping) are needed for more complex scenes. Linking human visual processing to performance of DCNNs with increasing depth, we here explored if, how, and when object information is differentiated from the backgrounds they appear on. To this end, we controlled the information in both objects and backgrounds, as well as the relationship between them by adding noise, manipulating background congruence and systematically occluding parts of the image. Results indicate that with an increase in network depth, there is an increase in the distinction between object- and background information. For more shallow networks, results indicated a benefit of training on segmented objects. Overall, these results indicate that, de facto, scene segmentation can be performed by a network of sufficient depth. We conclude that the human brain could perform scene segmentation in the context of object identification without an explicit mechanism, by selecting or "binding" features that belong to the object and ignoring other features, in a manner similar to a very deep convolutional neural network.

Low-level image statistics in natural scenes influence perceptual decision-making

A fundamental component of interacting with our environment is gathering and interpretation of sensory information. When investigating how perceptual information influences decision-making, most researchers have relied on manipulated or unnatural information as perceptual input, resulting in findings that may not generalize to real-world scenes. Unlike simplified, artificial stimuli, real-world scenes contain low-level regularities that are informative about the structural complexity, which the brain could exploit. In this study, participants performed an animal detection task on low, medium or high complexity scenes as determined by two biologically plausible natural scene statistics, contrast energy (CE) or spatial coherence (SC). In experiment 1, stimuli were sampled such that CE and SC both influenced scene complexity. Diffusion modelling showed that the speed of information processing was affected by low-level scene complexity. Experiment 2a/b refined these observations by showing how isolated manipulation of SC resulted in weaker but comparable effects, with an additional change in response boundary, whereas manipulation of only CE had no effect. Overall, performance was best for scenes with intermediate complexity. Our systematic definition quantifies how natural scene complexity interacts with decision-making. We speculate that CE and SC serve as an indication to adjust perceptual decision-making based on the complexity of the input.