Parsing rooms: the role of the PPA and RSC in perceiving object relations and spatial layout

Distinguishing the roles of edge, color, and other surface information in basic and superordinate scene representation

The human brain possesses a remarkable ability to recognize scenes depicted in line drawings, despite that these drawings contain only edge information. It remains unclear how the brain uses this information alongside surface information in scene recognition. Here, we combined electroencephalogram (EEG) and multivariate pattern analysis (MVPA) methods to distinguish the roles of edge, color, and other surface information in scene representation at the basic category level and superordinate naturalness level over time. The time-resolved decoding results indicated that edge information in line drawings is both sufficient and more effective than in color photographs and grayscale images at the superordinate naturalness level. Meanwhile, color and other surface information are exclusively involved in neural representation at the basic category level. The time generalization analysis further revealed that edge information is crucial for representation at both levels of abstraction. These findings highlight the distinct roles of edge, color, and other surface information in dynamic neural scene processing, shedding light on how the human brain represents scene information at different levels of abstraction.

Task-modulated neural responses in scene-selective regions of the human brain

The study of scene perception is crucial to the understanding of how one interprets and interacts with their environment, and how the environment impacts various cognitive functions. The literature so far has mainly focused on the impact of low-level and categorical properties of scenes and how they are represented in the scene-selective regions in the brain, PPA, RSC, and OPA. However, higher-level scene perception and the impact of behavioral goals is a developing research area. Moreover, the selection of the stimuli has not been systematic and mainly focused on outdoor environments. In this fMRI experiment, we adopted multiple behavioral tasks, selected real-life indoor stimuli with a systematic categorization approach, and used various multivariate analysis techniques to explain the neural modulation of scene perception in the scene-selective regions of the human brain. Participants (N = 21) performed categorization and approach-avoidance tasks during fMRI scans while they were viewing scenes from built environment categories based on different affordances ((i)access and (ii)circulation elements, (iii)restrooms and (iv)eating/seating areas). ROI-based classification analysis revealed that the OPA was significantly successful in decoding scene category regardless of the task, and that the task condition affected category decoding performances of all the scene-selective regions. Model-based representational similarity analysis (RSA) revealed that the activity patterns in scene-selective regions are best explained by task. These results contribute to the literature by extending the task and stimulus content of scene perception research, and uncovering the impact of behavioral goals on the scene-selective regions of the brain.

Alpha-frequency feedback to early visual cortex orchestrates coherent naturalistic vision

During naturalistic vision, the brain generates coherent percepts by integrating sensory inputs scattered across the visual field. Here, we asked whether this integration process is mediated by rhythmic cortical feedback. In electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) experiments, we experimentally manipulated integrative processing by changing the spatiotemporal coherence of naturalistic videos presented across visual hemifields. Our EEG data revealed that information about incoherent videos is coded in feedforward-related gamma activity while information about coherent videos is coded in feedback-related alpha activity, indicating that integration is indeed mediated by rhythmic activity. Our fMRI data identified scene-selective cortex and human middle temporal complex (hMT) as likely sources of this feedback. Analytically combining our EEG and fMRI data further revealed that feedback-related representations in the alpha band shape the earliest stages of visual processing in cortex. Together, our findings indicate that the construction of coherent visual experiences relies on cortical feedback rhythms that fully traverse the visual hierarchy.

Scene-selective brain regions respond to embedded objects of a scene

Objects are fundamental to scene understanding. Scenes are defined by embedded objects and how we interact with them. Paradoxically, scene processing in the brain is typically discussed in contrast to object processing. Using the BOLD5000 dataset (Chang et al., 2019), we examined whether objects within a scene predicted the neural representation of scenes, as measured by functional magnetic resonance imaging in humans. Stimuli included 1,179 unique scenes across 18 semantic categories. Object composition of scenes were compared across scene exemplars in different semantic scene categories, and separately, in exemplars of the same scene category. Neural representations in scene- and object-preferring brain regions were significantly related to which objects were in a scene, with the effect at times stronger in the scene-preferring regions. The object model accounted for more variance when comparing scenes within the same semantic category to scenes from different categories. Here, we demonstrate the function of scene-preferring regions includes the processing of objects. This suggests visual processing regions may be better characterized by the processes, which are engaged when interacting with the stimulus kind, such as processing groups of objects in scenes, or processing a single object in our foreground, rather than the stimulus kind itself.

Effects of spatial boundaries on episodic memory development

Children's spatial mapping starts out particularly sensitive to 3D wall-like boundaries and develops over early childhood to flexibly include other boundary types. This study investigated whether spatial boundaries influence children's episodic memory, as in adults, and whether this effect is modulated by boundary type. Eighty-one Korean children (34 girls, 36-84 months old) re-enacted a sequence of three discrete hiding events within a space containing one of three boundaries: 3D wall, aligned objects, or 2D line. Children's memory of events occurring on one side of the boundary developed earlier than those that crossed the boundary. At first, this interaction only applied to the 3D wall and extended to other boundary types with age, suggesting that children's changing spatial representations influence their episodic memory development.

Parts and Wholes in Scene Processing

During natural vision, our brains are constantly exposed to complex, but regularly structured environments. Real-world scenes are defined by typical part-whole relationships, where the meaning of the whole scene emerges from configurations of localized information present in individual parts of the scene. Such typical part-whole relationships suggest that information from individual scene parts is not processed independently, but that there are mutual influences between the parts and the whole during scene analysis. Here, we review recent research that used a straightforward, but effective approach to study such mutual influences: By dissecting scenes into multiple arbitrary pieces, these studies provide new insights into how the processing of whole scenes is shaped by their constituent parts and, conversely, how the processing of individual parts is determined by their role within the whole scene. We highlight three facets of this research: First, we discuss studies demonstrating that the spatial configuration of multiple scene parts has a profound impact on the neural processing of the whole scene. Second, we review work showing that cortical responses to individual scene parts are shaped by the context in which these parts typically appear within the environment. Third, we discuss studies demonstrating that missing scene parts are interpolated from the surrounding scene context. Bridging these findings, we argue that efficient scene processing relies on an active use of the scene's part-whole structure, where the visual brain matches scene inputs with internal models of what the world should look like.

Long term fMRI adaptation depends on adapter response in face-selective cortex

Repetition suppression (RS) reflects a neural attenuation during repeated stimulation. We used fMRI and the subsequent memory paradigm to test the predictive coding hypothesis for RS during visual memory processing by investigating the interaction between RS and differences due to memory in category-selective cortex (FFA, pSTS, PPA, and RSC). Fifty-six participants encoded face and house stimuli twice, followed by an immediate and delayed (48 h) recognition memory assessment. Linear Mixed Model analyses with repetition, subsequent recognition performance, and their interaction as fixed effects revealed that absolute RS during encoding interacts with probability of future remembrance in face-selective cortex. This effect was not observed for relative RS, i.e. when controlled for adapter-response. The findings also reveal an association between adapter response and RS, both for short and long term (48h) intervals, after controlling for the mathematical dependence between both measures. These combined findings are challenging for predictive coding models of visual memory and are more compatible with adapter-related and familiarity accounts.

Rapid contextualization of fragmented scene information in the human visual system

Real-world environments are extremely rich in visual information. At any given moment in time, only a fraction of this information is available to the eyes and the brain, rendering naturalistic vision a collection of incomplete snapshots. Previous research suggests that in order to successfully contextualize this fragmented information, the visual system sorts inputs according to spatial schemata, that is knowledge about the typical composition of the visual world. Here, we used a large set of 840 different natural scene fragments to investigate whether this sorting mechanism can operate across the diverse visual environments encountered during real-world vision. We recorded brain activity using electroencephalography (EEG) while participants viewed incomplete scene fragments at fixation. Using representational similarity analysis on the EEG data, we tracked the fragments' cortical representations across time. We found that the fragments' typical vertical location within the environment (top or bottom) predicted their cortical representations, indexing a sorting of information according to spatial schemata. The fragments' cortical representations were most strongly organized by their vertical location at around 200 ​ms after image onset, suggesting rapid perceptual sorting of information according to spatial schemata. In control analyses, we show that this sorting is flexible with respect to visual features: it is neither explained by commonalities between visually similar indoor and outdoor scenes, nor by the feature organization emerging from a deep neural network trained on scene categorization. Demonstrating such a flexible sorting across a wide range of visually diverse scenes suggests a contextualization mechanism suitable for complex and variable real-world environments.

Real-world structure facilitates the rapid emergence of scene category information in visual brain signals

In everyday life, our visual surroundings are not arranged randomly but structured in predictable ways. Although previous studies have shown that the visual system is sensitive to such structural regularities, it remains unclear whether the presence of an intact structure in a scene also facilitates the cortical analysis of the scene's categorical content. To address this question, we conducted an EEG experiment during which participants viewed natural scene images that were either "intact" (with their quadrants arranged in typical positions) or "jumbled" (with their quadrants arranged into atypical positions). We then used multivariate pattern analysis to decode the scenes' category from the EEG signals (e.g., whether the participant had seen a church or a supermarket). The category of intact scenes could be decoded rapidly within the first 100 ms of visual processing. Critically, within 200 ms of processing, category decoding was more pronounced for the intact scenes compared with the jumbled scenes, suggesting that the presence of real-world structure facilitates the extraction of scene category information. No such effect was found when the scenes were presented upside down, indicating that the facilitation of neural category information is indeed linked to a scene's adherence to typical real-world structure rather than to differences in visual features between intact and jumbled scenes. Our results demonstrate that early stages of categorical analysis in the visual system exhibit tuning to the structure of the world that may facilitate the rapid extraction of behaviorally relevant information from rich natural environments.NEW & NOTEWORTHY Natural scenes are structured, with different types of information appearing in predictable locations. Here, we use EEG decoding to show that the visual brain uses this structure to efficiently analyze scene content. During early visual processing, the category of a scene (e.g., a church vs. a supermarket) could be more accurately decoded from EEG signals when the scene adhered to its typical spatial structure compared with when it did not.