Luminance and Contrast of Images in the THINGS Database

The Second Database of Emotional Videos from Ottawa (DEVO-2): Over 1300 brief video clips rated on valence, arousal, impact, and familiarity

We introduce an updated set of video clips for research on emotion and its relations with perception, cognition, and behavior. These 1380 brief video clips each portray realistic episodes. They were selected to portray situations that vary widely in emotion. Undergraduate students rated the videos on emotional valence, arousal, impact, and familiarity. As expected, valence and arousal ratings were related in a U-shaped function, and arousal and impact were positively linearly associated with one another. Ratings of familiarity were near zero on average, verifying that the clips came from obscure sources. k-means cluster analysis revealed that they could be grouped by valence, arousal, and impact for selecting subsets for future studies. We also provide estimates of motion, luminance, contrast, and visual complexity to facilitate selection. The videos can be used in similar ways to static images but have the advantage of being dynamic and thus more ecologically valid.

Priors for natural image statistics inform confidence in perceptual decisions

Decision confidence plays a critical role in humans' ability to make adaptive decisions in a noisy perceptual world. Despite its importance, there is currently little consensus about the computations underlying confidence judgements in perceptual decisions. To better understand these mechanisms, we addressed the extent to which confidence is informed by a naturalistic prior distribution. Contrary to previous research, we did not require participants to internalise parameters of an arbitrary prior distribution. We instead used a novel psychophysical paradigm leveraging probability distributions of low-level image features in natural scenes, which are well-known to influence perception. Participants reported the subjective upright of naturalistic image patches, targets, and then reported their confidence in their orientation responses. We used computational modelling to relate the statistics of the low-level features in the targets to the average distribution of these features across many naturalistic images, a prior. Our results showed that participants' perceptual and importantly, their confidence judgments aligned with an internalised prior for image statistics. Overall, our study highlights the importance of naturalistic task designs that capitalise on existing, long-term priors to further understand the computational basis of confidence.

The visual experience dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video

We introduce the Visual Experience Dataset (VEDB), a compilation of more than 240 hours of egocentric video combined with gaze- and head-tracking data that offer an unprecedented view of the visual world as experienced by human observers. The dataset consists of 717 sessions, recorded by 56 observers ranging from 7 to 46 years of age. This article outlines the data collection, processing, and labeling protocols undertaken to ensure a representative sample and discusses the potential sources of error or bias within the dataset. The VEDB's potential applications are vast, including improving gaze-tracking methodologies, assessing spatiotemporal image statistics, and refining deep neural networks for scene and activity recognition. The VEDB is accessible through established open science platforms and is intended to be a living dataset with plans for expansion and community contributions. It is released with an emphasis on ethical considerations, such as participant privacy and the mitigation of potential biases. By providing a dataset grounded in real-world experiences and accompanied by extensive metadata and supporting code, the authors invite the research community to use and contribute to the VEDB, facilitating a richer understanding of visual perception and behavior in naturalistic settings.

The Time-Course of Food Representation in the Human Brain

Humans make decisions about food every day. The visual system provides important information that forms a basis for these food decisions. Although previous research has focused on visual object and category representations in the brain, it is still unclear how visually presented food is encoded by the brain. Here, we investigate the time-course of food representations in the brain. We used time-resolved multivariate analyses of electroencephalography (EEG) data, obtained from human participants (both sexes), to determine which food features are represented in the brain and whether focused attention is needed for this. We recorded EEG while participants engaged in two different tasks. In one task, the stimuli were task relevant, whereas in the other task, the stimuli were not task relevant. Our findings indicate that the brain can differentiate between food and nonfood items from ∼112 ms after the stimulus onset. The neural signal at later latencies contained information about food naturalness, how much the food was transformed, as well as the perceived caloric content. This information was present regardless of the task. Information about whether food is immediately ready to eat, however, was only present when the food was task relevant and presented at a slow presentation rate. Furthermore, the recorded brain activity correlated with the behavioral responses in an odd-item-out task. The fast representation of these food features, along with the finding that this information is used to guide food categorization decision-making, suggests that these features are important dimensions along which the representation of foods is organized.

Higher-contrast images are better remembered during naturalistic encoding

It is unclear whether memory for images of poorer visibility (as low contrast or small size) will be lower due to weak signals elicited in early visual processing stages, or perhaps better since their processing may entail top-down processes (as effort and attention) associated with deeper encoding. We have recently shown that during naturalistic encoding (free viewing without task-related modulations), for image sizes between 3°-24°, bigger images stimulating more visual system processing resources at early processing stages are better remembered. Similar to size, higher contrast leads to higher activity in early visual processing. Therefore, here we hypothesized that during naturalistic encoding, at critical visibility ranges, higher contrast images will lead to higher signal-to-noise ratio and better signal quality flowing downstream and will thus be better remembered. Indeed, we found that during naturalistic encoding higher contrast images were remembered better than lower contrast ones (~ 15% higher accuracy, ~ 1.58 times better) for images at 7.5-60 RMS contrast range. Although image contrast and size modulate early visual processing very differently, our results further substantiate that at poor visibility ranges, during naturalistic non-instructed visual behavior, physical image dimensions (contributing to image visibility) impact image memory.

Contribution of low-level image statistics to EEG decoding of semantic content in multivariate and univariate models with feature optimization

Spatio-temporal patterns of evoked brain activity contain information that can be used to decode and categorize the semantic content of visual stimuli. However, this procedure can be biased by low-level image features independently of the semantic content present in the stimuli, prompting the need to understand the robustness of different models regarding these confounding factors. In this study, we trained machine learning models to distinguish between concepts included in the publicly available THINGS-EEG dataset using electroencephalography (EEG) data acquired during a rapid serial visual presentation paradigm. We investigated the contribution of low-level image features to decoding accuracy in a multivariate model, utilizing broadband data from all EEG channels. Additionally, we explored a univariate model obtained through data-driven feature selection applied to the spatial and frequency domains. While the univariate models exhibited better decoding accuracy, their predictions were less robust to the confounding effect of low-level image statistics. Notably, some of the models maintained their accuracy even after random replacement of the training dataset with semantically unrelated samples that presented similar low-level content. In conclusion, our findings suggest that model optimization impacts sensitivity to confounding factors, regardless of the resulting classification performance. Therefore, the choice of EEG features for semantic decoding should ideally be informed by criteria beyond classifier performance, such as the neurobiological mechanisms under study.

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Visual Representations: Insights from Neural Decoding

Patterns of brain activity contain meaningful information about the perceived world. Recent decades have welcomed a new era in neural analyses, with computational techniques from machine learning applied to neural data to decode information represented in the brain. In this article, we review how decoding approaches have advanced our understanding of visual representations and discuss efforts to characterize both the complexity and the behavioral relevance of these representations. We outline the current consensus regarding the spatiotemporal structure of visual representations and review recent findings that suggest that visual representations are at once robust to perturbations, yet sensitive to different mental states. Beyond representations of the physical world, recent decoding work has shone a light on how the brain instantiates internally generated states, for example, during imagery and prediction. Going forward, decoding has remarkable potential to assess the functional relevance of visual representations for human behavior, reveal how representations change across development and during aging, and uncover their presentation in various mental disorders.

Neural tuning instantiates prior expectations in the human visual system

Perception is often modelled as a process of active inference, whereby prior expectations are combined with noisy sensory measurements to estimate the structure of the world. This mathematical framework has proven critical to understanding perception, cognition, motor control, and social interaction. While theoretical work has shown how priors can be computed from environmental statistics, their neural instantiation could be realised through multiple competing encoding schemes. Using a data-driven approach, here we extract the brain's representation of visual orientation and compare this with simulations from different sensory coding schemes. We found that the tuning of the human visual system is highly conditional on stimulus-specific variations in a way that is not predicted by previous proposals. We further show that the adopted encoding scheme effectively embeds an environmental prior for natural image statistics within the sensory measurement, providing the functional architecture necessary for optimal inference in the earliest stages of cortical processing.

Levels of orientation bias differ across digital content categories: Implications for visual perception

With the continued growth of digital device use, a greater portion of the visual world experienced daily by many people has shifted towards digital environments. The "oblique effect" denotes a bias for horizontal and vertical (canonical) contours over oblique contours, which is derived from a disproportionate exposure to canonical content. Carpentered environments have been shown to possess proportionally more canonical than oblique contours, leading to perceptual bias in those who live in "built" environments. Likewise, there is potential for orientation sensitivity to be shaped by frequent exposure to digital content. The potential influence of digital content on the oblique effect was investigated by measuring the degree of orientation anisotropy from a range of digital scenes using Fourier analysis. Content from popular cartoons, video games, and social communication websites was compared to real-life nature, suburban, and urban scenes. Findings suggest that digital content varies widely in orientation anisotropy, but pixelated video games and social communication websites were found to exhibit a degree of orientation anisotropy substantially exceeding that observed in all measured categories of real-world environments. Therefore, the potential may exist for digital content to induce an even greater shift in orientation bias than has been observed in previous research. This potential, and implications of such a shift, is discussed.