The adult face-diet: A naturalistic observation study

Facial expression categorization predominantly relies on mid-spatial frequencies

Facial expressions are crucial in human communication. Recent decades have seen growing interest in understanding the role of spatial frequencies (SFs) in emotion perception in others. While some studies have suggested a preferential treatment of low versus high SFs, the optimal SFs for recognizing basic facial expressions remain elusive. This study, conducted on Western participants, addresses this gap using two complementary methods: a data-driven method (Exp. 1) without arbitrary SF cut-offs, and a more naturalistic method (Exp. 2) simulating variations in viewing distance. Results generally showed a preponderant role of low over high SFs, but particularly stress that facial expression categorization mostly relies on mid-range SF content (i.e. ∼6-13 cycles per face), often overlooked in previous studies. Optimal performance was observed at short to medium viewing distances (1.2-2.4 m), declining sharply with increased distance, precisely when mid-range SFs were no longer available. Additionally, our data suggest variations in SF tuning profiles across basic facial expressions and nuanced contributions from low and mid SFs in facial expression processing. Most importantly, it suggests that any method that removes mid-SF content has the downfall of offering an incomplete account of SFs diagnosticity for facial expression recognition.

Learning to recognize unfamiliar faces from fine-phonetic detail in visual speech

How speech is realized varies across talkers but can be somewhat consistent within a talker. Humans are sensitive to these idiosyncrasies when perceiving auditory speech, but also, in face-to-face communications, when perceiving their visual speech. Our recent work has shown that humans can also use talker idiosyncrasies seen in how talkers produce sentences to rapidly learn to recognize unfamiliar talkers, suggesting that visual speech information can be used for speech perception and talker recognition. However, in learning from sentences, learners may focus only on global information about the talker, such as talker-specific realizations of prosody and rate. The present study tested whether human perceivers can learn the identity of the talker based solely on fine-phonetic detail in the dynamic realization of visual speech alone. Participants learned to identify talkers from point-light displays showing them uttering isolated words. These point-light displays isolated the dynamic speech information, while discarding static information about the talker's face. No sound was presented. Feedback was given only during training. Test included point-light displays of familiar words from training and of novel words. Participants learned to recognize two and four talkers from the word-level dynamics of visual speech from very little exposure. The established representations allowed talker recognition independent of linguistic content-that is, even from novel words. Spoken words therefore contain sufficient indexical information in their fine-phonetic detail for perceivers to acquire dynamic facial representations for unfamiliar talkers that allows generalization across words. Dynamic representations of talking faces are formed for the recognition of unfamiliar faces.

Preferred fixation position and gaze location: Two factors modulating the composite face effect

Humans consistently land their first saccade to a face at a preferred fixation location (PFL). Humans also typically process faces as wholes, as evidenced by perceptual effects such as the composite face effect (CFE). However, not known is whether an individual's tendency to process faces as wholes varies with their gaze patterns on the face. Here, we investigated variation of the CFE with the PFL. We compared the strength of the CFE for two groups of observers who were screened to have their PFLs either higher up, closer to the eyes, or lower on the face, closer to the tip of the nose. During the task, observers maintained their gaze at either their own group's mean PFL or at the other group's mean PFL. We found that the top half of the face elicits a stronger CFE than the bottom half. Further, the strength of the CFE was modulated by the distance of the PFL from the eyes, such that individuals with a PFL closer to the eyes had a stronger CFE than those with a PFL closer to the mouth. Finally, the top-half CFE for both upper-lookers and lower-lookers was abolished when they fixated at a non-preferred location on the face. Our findings show that the CFE relies on internal face representations shaped by the long-term use of a consistent oculomotor strategy to view faces.

The visual experience dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video

We introduce the Visual Experience Dataset (VEDB), a compilation of more than 240 hours of egocentric video combined with gaze- and head-tracking data that offer an unprecedented view of the visual world as experienced by human observers. The dataset consists of 717 sessions, recorded by 56 observers ranging from 7 to 46 years of age. This article outlines the data collection, processing, and labeling protocols undertaken to ensure a representative sample and discusses the potential sources of error or bias within the dataset. The VEDB's potential applications are vast, including improving gaze-tracking methodologies, assessing spatiotemporal image statistics, and refining deep neural networks for scene and activity recognition. The VEDB is accessible through established open science platforms and is intended to be a living dataset with plans for expansion and community contributions. It is released with an emphasis on ethical considerations, such as participant privacy and the mitigation of potential biases. By providing a dataset grounded in real-world experiences and accompanied by extensive metadata and supporting code, the authors invite the research community to use and contribute to the VEDB, facilitating a richer understanding of visual perception and behavior in naturalistic settings.

Atypical daily visual exposure to faces in adults with autism spectrum disorder

Expert face processes are refined and tuned through a protracted development. Exposure statistics of the daily visual experience of neurotypical adults (the face diet) show substantial exposure to familiar faces. People with autism spectrum disorder (ASD) do not show the same expertise with faces as their non-autistic counterparts. This may be due to an impoverished visual experience with faces, according to experiential models of autism. Here, we present the first empirical report on the day-to-day visual experience of the faces of adults with ASD. Our results, based on over 360 h of first-person perspective footage of daily exposure, show striking qualitative and quantitative differences in the ASD face diet compared with those of neurotypical observers, which is best characterized by a pattern of reduced and atypical exposure to familiar faces in ASD. Specifically, duration of exposure to familiar faces was lower in ASD, and faces were viewed from farther distances and from viewpoints that were biased toward profile pose. Our results provide strong evidence that individuals with ASD may not be getting the experience needed for the typical development of expert face processes.

A preference to look closer to the eyes is associated with a position-invariant face neural code

When looking at faces, humans invariably move their eyes to a consistent preferred first fixation location on the face. While most people have the preferred fixation location just below the eyes, a minority have it between the nose-tip and mouth. Not much is known about whether these long-term differences in the preferred fixation location are associated with distinct neural representations of faces. To study this, we used a gaze-contingent face adaptation aftereffect paradigm to test in two groups of observers, one with their mean preferred fixation location closer to the eyes (upper lookers) and the other closer to the mouth (lower lookers). In this task, participants were required to maintain their gaze at either their own group's mean preferred fixation location or that of the other group during adaptation and testing. The two possible fixation locations were 3.6° apart on the face. We measured the face adaptation aftereffects when the adaptation and testing happened while participants maintained fixation at either the same or different locations on the face. Both groups showed equally strong adaptation effects when the adaptation and testing happened at the same fixation location. Crucially, only the upper lookers showed a partial transfer of the FAE across the two fixation locations, when adaptation occurred at the eyes. Lower lookers showed no spatial transfer of the FAE irrespective of the adaptation position. Given the classic finding that neural tuning is increasingly position invariant as one moves higher in the visual hierarchy, this result suggests that differences in the preferred fixation location are associated with distinct neural representations of faces.

Are people with autism prosopagnosic?

Difficulties in various face processing tasks have been well documented in autism spectrum disorder (ASD). Several meta-analyses and numerous case-control studies have indicated that this population experiences a moderate degree of impairment, with a small percentage of studies failing to detect any impairment. One possible account of this mixed pattern of findings is heterogeneity in face processing abilities stemming from the presence of a subpopulation of prosopagnosic individuals with ASD alongside those with normal face processing skills. Samples randomly drawn from such a population, especially relatively smaller ones, would vary in the proportion of participants with prosopagnosia, resulting in a wide range of group-level deficits from mild (or none) to severe across studies. We test this prosopagnosic subpopulation hypothesis by examining three groups of participants: adults with ASD, adults with developmental prosopagnosia (DP), and a comparison group. Our results show that the prosopagnosic subpopulation hypothesis does not account for the face impairments in the broader autism spectrum. ASD observers show a continuous and graded, rather than categorical, heterogeneity that span a range of face processing skills including many with mild to moderate deficits, inconsistent with a prosopagnosic subtype account. We suggest that pathogenic origins of face deficits for at least some with ASD differ from those of DP.

The development of upright face perception depends on evolved orientation-specific mechanisms and experience

Here we examine whether our impressive ability to perceive upright faces arises from evolved orientation-specific mechanisms, our extensive experience with upright faces, or both factors. To do so, we tested Claudio, a man with a congenital joint disorder causing his head to be rotated back so that it is positioned between his shoulder blades. As a result, Claudio has seen more faces reversed in orientation to his own face than matched to it. Controls exhibited large inversion effects on all tasks, but Claudio performed similarly with upright and inverted faces in both detection and identity-matching tasks, indicating these abilities are the product of evolved mechanisms and experience. In contrast, he showed clear upright superiority when detecting "Thatcherized" faces (faces with vertically flipped features), suggesting experience plays a greater role in this judgment. Together, these findings indicate that both evolved orientation-specific mechanisms and experience contribute to our proficiency with upright faces.

Recognizing distant faces

As an 'early alerting' sense, one of the primary tasks for the human visual system is to recognize distant objects. In the specific context of facial identification, this ecologically important task has received surprisingly little attention. Most studies have investigated facial recognition at short, fixed distances. Under these conditions, the photometric and configural information related to the eyes, nose and mouth are typically found to be primary determinants of facial identity. Here we characterize face recognition performance as a function of viewing distance and investigate whether the primacy of the internal features continues to hold across increasing viewing distances. We find that exploring the distance dimension reveals a qualitatively different salience distribution across a face. Observers' recognition performance significantly exceeds that obtained with the internal facial physiognomy, and also exceeds the computed union of performances with internal and external features alone, suggesting that in addition to the mutual configuration of the eyes, nose and mouth, it is the relationships between these features and external head contours that are crucial for recognition. We have also conducted computational studies with convolutional neural networks trained on the task of face recognition to examine whether this representational bias could emerge spontaneously through exposure to faces. The results provide partial support for this possibility while also highlighting important differences between the human and artificial system. These findings have implications for the nature of facial representations useful for a visual system, whether human or machine, for recognition over large and varying distances.

Neural sensitivity to faces is increased by immersion into a novel ethnic environment: Evidence from ERPs

Previous reports suggest that East-Asians may show larger face-elicited N170 components in the ERP as compared to Caucasian participants. Since the N170 can be modulated by perceptual expertise, such group differences may be accounted for by differential experience, for example, with logographic versus alphabetic scripts (script system hypothesis) or by exposure to abundant novel faces during the immersion into a new social and/or ethnic environment (social immersion hypothesis). We conducted experiments in Hong Kong and Berlin, recording ERPs in a series of one-back tasks, using same- and other-ethnicity face stimuli in upright and inverted orientation and doodle stimuli. In Hong Kong we tested local Chinese residents and foreign guest students who could not read the logographic script; in Berlin we tested German residents who could not read the logographic script and foreign Chinese visitors. In both experiments, we found significantly larger N170 amplitudes to faces, regardless of ethnicity, in the foreign than in the local groups. Moreover, this effect did not depend on stimulus orientation, suggesting that the N170 group differences do not reflect differences in configural visual processing. A group of short-term German residents in Berlin did not differ in N170 amplitude from long-term residents. Together, these findings indicate that the extensive confrontation with novel other-ethnicity faces during immersion in a foreign culture may enhance the neural response to faces, reflecting the short-term plasticity of the underlying neural system.

The roles of gaze and head orientation in face categorization during rapid serial visual presentation

Little is known about how perceived gaze direction and head orientation may influence human categorization of visual stimuli as faces. To address this question, a sequence of unsegmented natural images, each containing a random face or a non-face object, was presented in rapid succession (stimulus duration: 91.7 ms per image) during which human observers were instructed to respond immediately to every face presentation. Faces differed in gaze and head orientation in 7 combinations - full-front views with perceived gaze (1) directed to the observer, (2) averted to the left, or (3) averted to the right, left ¾ side views with (4) direct gaze or (5) averted gaze, and right ¾ side views with (6) direct gaze or (7) averted gaze - were presented randomly throughout the sequence. We found highly accurate and rapid behavioural responses to all kinds of faces. Crucially, both perceived gaze direction and head orientation had comparable, non-interactive effects on response times, where direct gaze was responded faster than averted gaze by 48 ms and full-front view faster than ¾ side view also by 48 ms on average. Presentations of full-front faces with direct gaze led to an additive speed advantage of 96 ms to ¾ faces with averted gaze. The results reveal that the effects of perceived gaze direction and head orientation on the speed of face categorization probably depend on the degree of social relevance of the face to the viewer.

Does automatic human face categorization depend on head orientation?

Whether human categorization of visual stimuli as faces is optimal for full-front views, best revealing diagnostic features but lacking depth cues, remains largely unknown. To address this question, we presented 16 human observers with unsegmented natural images of different living and non-living objects at a fast rate (f = 12 Hz), with natural face images appearing at f/9 = 1.33 Hz. Faces posing all full-front or at ¾ side view angles appeared in separate sequences. Robust frequency-tagged 1.33 Hz (and harmonic) occipito-temporal electroencephalographic (EEG) responses reflecting face-selective neural activity did not differ in overall amplitude between full-front and ¾ side views. Despite this, alternating between full-front and ¾ side views within a sequence led to significant responses at specific harmonics of .67 Hz (f/18), objectively isolating view-dependent face-selective responses over occipito-temporal regions. Critically, a time-domain analysis showed that these view-dependent face-selective responses reflected only an earlier response to full-front than ¾ side views by 8-13 ms. Overall, these findings indicate that the face-selective neural representation is as robust for ¾ side faces as for full-front faces in the human brain, but full-front views provide a slightly earlier processing-time advantage as compared to rotated face views.

Size Effects in the Recognition of Blurry Faces

Spatial frequencies critical for recognition of faces are scale-dependent. Progressively coarser features of the face are utilized at smaller sizes, despite the availability of finer features. Blur removes fine details in an image, disrupting the finer features utilized for recognition at large sizes. At smaller sizes, observers utilize coarser features, and thus, recognition may be less impacted by blur. This coupling between size and critical spatial frequencies allows us to predict a regime in which observers tolerate blur better with decreasing image sizes within a range of moderate face sizes. We tested recognition of famous faces in four conditions: large-intact, small-intact, large-blurry, and small-blurry. Observers showed high recognition performance in both intact conditions. Blur significantly disrupted recognition, yet accuracy was significantly and consistently higher in the small-blurry compared to the large-blurry condition. These results are suggestive of an inherent scale-dependent mechanism that, at certain sizes, negatively impacts recognition of blurry images.

Face perception: A brief journey through recent discoveries and current directions

Faces are a rich source of information about the people around us. Identity, state of mind, emotions, intentions, age, gender, ethnic background, attractiveness and a host of other attributes about an individual can be gleaned from a face. When face perception fails, dramatic psycho-social consequences can follow at the individual level, as in the case of prosopagnosic parents who are unable to recognize their children at school pick-up. At the species level, social interaction patterns are shaped by human face perception abilities. The computational feat of recognizing faces and facial attributes, and the challenges overcome by the human brain to achieve this feat, have fascinated generations of vision researchers. In this paper, we present a brief overview of some of the milestones of discovery as well as outline a selected set of current directions and open questions on this topic.