Familiarity and Within-Person Facial Variability: The Importance of the Internal and External Features

High familiar faces have both eye recognition and holistic processing advantages

People recognize familiar faces better than unfamiliar faces. However, it remains unknown whether familiarity affects part-based and/or holistic processing. Wang et al., Frontiers in Psychology, 6, 559 (2015), Vision Research, 157, 89-96 (2019) found both enhanced part-based and holistic processing in eye relative to mouth regions (i.e., in a region-selective manner) for own-race and own-species faces, i.e., faces with more experience. Here, we examined the role of face familiarity in eyes (part-based, region-selective) and holistic processing. Face familiarity was tested at three levels: high-familiar (faces of students from the same department and the same class who attended almost all courses together), low-familiar (faces of students from the same department but different classes who attended some courses together), and unfamiliar (faces of schoolmates from different departments who seldom attended the same courses). Using the old/new task in Experiment 1, we found that participants recognized eyes of high-familiar faces better than low-familiar and unfamiliar ones, while similar performance was observed for mouths, indicating a region-selective, eyes familiarity effect. Using the "Perceptual field" paradigm in Experiment 2, we observed a stronger inversion effect for high-familiar faces, a weaker inversion effect for low-familiar faces, but a non-significant inversion effect for unfamiliar faces, indicating that face familiarity plays a role in holistic processing. Taken together, our results suggest that familiarity, like other experience-based variables (e.g., race and species), can improve both eye processing and holistic processing.

The impact of stability in appearance on the development of facial representations

The way faces become familiar and what information is represented as familiarity develops has puzzled researchers in the field of human face recognition for decades. In this paper, we present three experiments serving as proof of concept for a cost-efficient mechanism of face learning describing how facial representations form over time and accounting for recognition errors. We propose that the encoding of facial information is dynamic and modulated by the intrinsic stability in individual faces' appearance. We drew on a robust and ecological method using a proxy of exposure to famous faces in the real world and manipulated test images to assess the prediction that recognition of famous faces is affected by their relative stability in appearance. We consistently show that stable facial appearances (like Tom Cruise's) facilitate recognition in early stages of familiarisation but that performance does not improve much over time. In contrast, variations in appearance (like Jared Leto's) hinder recognition at first but improve performance with further media exposure. This pattern of results is consistent with the proposed cost-efficient face learning mechanism whereby facial representations build on a foundation of large-scale diagnostic information and refine over time if needed. When coarse information loses its diagnostic value through the experience of variations in appearance across encounters, diagnostic facial details and/or their spatial relationships must receive more weights, leading to refined representations that are more discriminative and reliable than representations of stable faces.

From Pictures to the People in Them: Averaging Within-Person Variability Leads to Face Familiarization

Familiar faces can be confidently recognized despite sometimes radical changes in their appearance. Exposure to within-person variability-differences in facial characteristics over successive encounters-contributes to face familiarization. Research also suggests that viewers create mental averages of the different views of faces they encounter while learning them. Averaging over within-person variability is thus a promising mechanism for face familiarization. In Experiment 1, 153 Canadian undergraduates (88 female; age: M = 21 years, SD = 5.24) learned six target identities from eight different photos of each target interspersed among 32 distractor identities. Face-matching accuracy improved similarly irrespective of awareness of the target's identity, confirming that target faces presented among distractors can be learned incidentally. In Experiment 2, 170 Canadian undergraduates (125 female; age: M = 22.6 years, SD = 6.02) were tested using a novel indirect measure of learning. The results show that viewers update a mental average of a person's face as it becomes learned. Our findings are the first to show how averaging within-person variability over time leads to face familiarization.

The importance of internal and external features in recognizing faces that vary in familiarity and race

Familiar and unfamiliar faces are recognized in fundamentally different ways. One way in which recognition differs is in terms of the features that facilitate recognition: previous studies have shown that familiar face recognition depends more on internal facial features (i.e., eyes, nose and mouth), whereas unfamiliar face recognition depends more on external facial features (i.e., hair, ears and contour). However, very few studies have examined the recognition of faces that vary in both familiarity and race, and the reliance on different facial features, whilst also using faces that incorporate natural within-person variability. In the current study, we used an online version of the card sorting task to assess adults’ (n = 258) recognition of faces that varied in familiarity and race when presented with either the whole face, internal features only, or external features only. Adults better recognized familiar faces than unfamiliar faces in both the whole face and the internal features only conditions, but not in the external features only condition. Reasons why adults did not show an own-race advantage in recognition are discussed.

The Episodic Prototypes Model (EPM): On the nature and genesis of facial representations

Faces undergo massive changes over time and life events. We need a mental representation which is flexible enough to cope with the existing visual varieties, but which is also stable enough to be the basis for valid recognition. Two main theoretical frameworks exist to describe facial representations: prototype models assuming one central item comprising all visual experiences of a face, and exemplar models assuming single representations of each visual experience of a face. We introduce a much more ecological valid model dealing with episodic prototypes (the Episodic Prototypes Model—EPM), where faces are represented by a low number of prototypes that refer to specific Episodes of Life (EoL, e.g., early adulthood, mature age) during which the facial appearance shows only moderate variation. Such an episodic view of mental representation allows for efficient storage, as the number of needed prototypes is relatively low, and it allows for the needed variation within a prototype that keeps the everyday and steadily ongoing changes across a certain period of time. Studies 1–3 provide evidence that facial representations are highly dependent on temporal aspects which is in accord with EoL, and that individual learning history generates the structure and content of respective prototypes. In Study 4, we used implicit measures (RT) in a face verification task to investigate the postulated power of the EPM. We could demonstrate that episodic prototypes clearly outperformed visual depictions of exhaustive prototypes, supporting the general idea of our approach.

The Importance of Internal and External Features in Matching Own and Other Race Faces

Research has shown that we are better at discriminating between faces that are our own race, and much less accurate with faces of another race. When the external features of faces were removed, this reduced the accuracy for recognizing other-races faces, more than own-race faces, suggesting that the external features (hair, face shape) are especially important for the recognition of other-race faces. The aim of the current study was to determine whether external features were more useful in matching other-race faces, and whether this was the case for Western and Eastern viewers. The current study employed a face matching task with Caucasian (U.K.) and Asian (Chinese) participants and found that responses were more accurate for own-race faces, and for whole faces when compared with faces where the internal or external features had been removed. Removing the external features of other-race faces increased the own-race bias for Chinese and U.K. participants, demonstrating the importance of viewing whole faces, including the external features when matching other-race faces.

Forgetting faces over a week: investigating self-reported face recognition ability and personality

Background

Although face recognition is now well studied, few researchers have considered the nature of forgetting over longer time periods. Here, I investigated how newly learned faces were recognised over the course of one week. In addition, I considered whether self-reported face recognition ability, as well as Big Five personality dimensions, were able to predict actual performance in a recognition task.

Methods

In this experiment (N = 570), faces were learned through short video interviews, and these identities were later presented in a recognition test (using previously unseen images) after no delay, six hours, twelve hours, one day, or seven days.

Results

The majority of forgetting took place within the first 24 hours, with no significant decrease after that timepoint. Further, self-reported face recognition abilities were moderately predictive of performance, while extraversion showed a small, negative association with performance. In both cases, these associations remained consistent across delay conditions.

Discussion

The current work begins to address important questions regarding face recognition using longitudinal, real-world time intervals, focussing on participant insight into their own abilities, and the process of forgetting more generally.

Assessing the effect of facial disguises on forensic facial comparison by morphological analysis

Disguises are commonly used to mask a person's facial appearance in areas under closed-circuit television (CCTV) surveillance. While many studies attempted to understand the effects of disguises, such as hats and glasses, on facial recognition, limited studies have looked at disguises in forensic facial comparison. The aim of this study was to compare the outcomes of forensic facial comparison by morphological analysis (MA) in a CCTV sample with sunglasses and brimmed caps. The sample was obtained from the Wits Face Database and organized into 81 face pools of one target facial image wearing a disguise (cap or sunglasses) and 10 potential matching images. MA was conducted across face pools, and confusion matrices were used to assess the outcomes. Surprisingly, sunglasses had limited effect on MA performance both in accuracy (90.4%) and in reliability (κ = 0.798), while caps markedly decreased both accuracy (68.1%) and reliability (κ = 0.639). Error rates were associated primarily with false negatives in both samples (caps: 42.4%; sunglasses: 16.1%) despite the sample distribution favoring false-positive errors, which were very low (caps: 0.6%; sunglasses: 0%). Similarly to other studies, hats and caps were more harmful to correct identification when compared to sunglasses, which actually resulted in better accuracy than regular CCTV recordings. The effect of brimmed caps on accuracy was attributed to the overall loss of facial information caused. On training analysts, it may be helpful to instruct purposefully avoiding overreliance on easily disguised facial features, as other regions of the face also contain substantial feature information.

Contrast Negation Impairs Sorting Unfamiliar Faces by Identity: A Comparison With Original (Contrast-Positive) and Stretched Images

Recognition of unfamiliar faces is difficult in part due to variations in expressions, angles, and image quality. Studies suggest shape and surface properties play varied roles in face learning, and identification of unfamiliar faces uses diagnostic pigmentation/surface reflectance relative to shape information. Here, participants sorted photo-cards of unfamiliar faces by identity, which were shown in their original, stretched, and contrast-negated forms, to examine the utility of diagnostic shape and surface properties in sorting unfamiliar faces by identity. In four experiments, we varied the presentation order of conditions (contrast-negated first or original first with stretched second across experiments) and whether the same or different photo-cards were seen across conditions. Stretching the images did not impair performance in any measures relative to other conditions. Contrast negation generally exacerbated poor sorting by identity compared with the other conditions. However, seeing the contrast-negated photo-cards last mitigated some of the effects of contrast negation. Together, results suggest an important role for surface properties such as pigmentation and reflectance for sorting by identity and add to literatures on informational content and appearance variability in discrimination of facial identity.

Unitization of internal and external features contributes to associative recognition for faces: Evidence from modulations of the FN400

Associative recognition requires discriminating between old items and conjunction lures constructed by recombining elements from two different study items. This task can be solved not only by recollection but also by familiarity if the to-be-remembered stimuli are perceived as a unitized representation. In two event-related potential (ERP) studies, we provide evidence for the integration of internal and external facial features by showing that the early frontal old-new effect (considered a correlate of familiarity) is modulated by the specific combination of facial features. Participants studied faces consisting of internal features (eyes, eyebrows, nose, and mouth) paired with external features (hair, head shape, and ears). During the testing phase, intact, recombined, and new faces were presented. Recombined faces consisted of internal and external features taken from two different studied faces. The results showed that at the frontal sites, during the time window from 300 to 500 ms, ERPs to intact faces were more positive than those to new and recombined faces; the latter two did not differ from one another. The late parietal effect was observed only after a more extended study phase in Experiment 2. We take the modulation of the early frontal old-new effect as evidence for the contribution of familiarity to associative recognition for combinations of internal and external facial features.

The Own-Race Bias for Face Recognition in a Multiracial Society

The own-race bias (ORB) is a reliable phenomenon across cultural and racial groups where unfamiliar faces from other races are usually remembered more poorly than own-race faces (Meissner and Brigham, 2001). By adopting a yes-no recognition paradigm, we found that ORB was pronounced across race groups (Malaysian-Malay, Malaysian-Chinese, Malaysian-Indian, and Western-Caucasian) when faces were presented with only internal features (Experiment 1), implying that growing up in a profoundly multiracial society does not necessarily eliminate ORB. Using a procedure identical to Experiment 1, we observed a significantly greater increment in recognition performance for other-race faces than for own-race faces when the external features (e.g. facial contour and hairline) were presented along with the internal features (Experiment 2)-this abolished ORB. Contrary to assumptions based on the contact hypothesis, participants' self-reported amount of interracial contact on a social contact questionnaire did not significantly predict the magnitude of ORB. Overall, our findings suggest that the level of exposure to other-race faces accounts for only a small part of ORB. In addition, the present results also support the notion that different neural mechanisms may be involved in processing own- and other-race faces, with internal features of own-race faces being processed more effectively, whereas external features dominate representations of other-race faces.

Decomposing the composite face effect: Evidence for non-holistic processing based on the ex-Gaussian distribution

Composite faces fuse the top and bottom halves from two different faces to create a powerful illusion of a novel face. It has been argued that composite faces are processed holistically, namely that the constituent face parts are perceived as a template, rather than independent features. This study sought to uncover the locus of the composite face effect by relating its empirical reaction time distributions to theoretical ex-Gaussian parameters. The results showed that the composite face effect for unfamiliar (Experiment 1) and familiar (Experiment 2) faces is generated by pure changes in the exponential component of the ex-Gaussian distribution. This held true for both partial and complete design measures. The exponential component has been attributed to working memory and attentional processes. The results suggest the involvement of attentional and working memory processes in the composite face effect and in the perception of faces in general. They cast doubts on the holistic nature of face processing. The results also provide important constraints on future computational theories of the effect.

Discrimination and recognition of faces with changed configuration

Subtle metric differences in facial configuration, such as between-person variation in the distances between the eyes, have been used widely in psychology to explain face recognition. However, these studies of configuration have typically utilized unfamiliar faces rather than the familiar faces that the process of recognition ultimately seeks to explain. This study investigates whether face recognition relies on the metric information presumed in configural theory, by manipulating the interocular distance in both unfamiliar and familiar faces. In Experiment 1, observers were asked to detect which face in a pair was presented with its configuration intact. In Experiment 2, this discrimination task was repeated with faces presented individually, and observers were also asked to make familiarity categorizations to the same stimuli. In both experiments, familiarity determined detection of faces in their original configuration, and also enhanced identity categorization in Experiment 2. However, discrimination of configuration was generally low. In turn, recognition accuracy was generally high irrespective of configuration condition. Moreover, observers most sensitive to configuration during discrimination did not appear to rely on this information for recognition of familiar faces. These results demonstrate that configuration theory provides limited explanatory power for the recognition of familiar faces.

Optimal features for auditory categorization

Humans and vocal animals use vocalizations to communicate with members of their species. A necessary function of auditory perception is to generalize across the high variability inherent in vocalization production and classify them into behaviorally distinct categories ('words' or 'call types'). Here, we demonstrate that detecting mid-level features in calls achieves production-invariant classification. Starting from randomly chosen marmoset call features, we use a greedy search algorithm to determine the most informative and least redundant features necessary for call classification. High classification performance is achieved using only 10-20 features per call type. Predictions of tuning properties of putative feature-selective neurons accurately match some observed auditory cortical responses. This feature-based approach also succeeds for call categorization in other species, and for other complex classification tasks such as caller identification. Our results suggest that high-level neural representations of sounds are based on task-dependent features optimized for specific computational goals.

The Effects of Blur and Inversion on the Recognition of Ambient Face Images

When viewing unfamiliar faces that vary in expressions, angles, and image quality, observers make many recognition errors. Specifically, in unconstrained identity-sorting tasks, observers struggle to cope with variation across different images of the same person while succeeding at telling different people apart. The use of ambient face images in this simple card-sorting task reveals the magnitude of these face recognition errors and suggests a useful platform to reexamine the nature of face processing using naturalistic stimuli. In the present study, we chose to investigate the impact of two basic stimulus manipulations (image blur and face inversion) on identity sorting with ambient images. Although these manipulations are both known to affect face processing when well-controlled, frontally viewed face images are used, examining how they affect performance for ambient images is an important step toward linking the large body of research using controlled face images to more ecologically valid viewing conditions. Briefly, we observed a high cost of image blur regardless of blur magnitude, and a strong inversion effect that affected observers’ sensitivity to extrapersonal variability but did not affect the number of unique identities they estimated were present in the set of images presented to them.

Distinct neural processes for the perception of familiar versus unfamiliar faces along the visual hierarchy revealed by EEG

Humans recognize faces with ease, despite the complexity of the task and of the visual system which underlies it. Different spatial regions, including both the core and extended face processing networks, and distinct temporal stages of processing have been implicated in face recognition, but there is ongoing controversy regarding the extent to which the mechanisms for recognizing a familiar face differ from those for an unfamiliar face. Here, we used electroencephalogram (EEG) and flicker SSVEP, a high signal-to-noise approach, and searchlight decoding methods to elucidate the mechanisms mediating the processing of familiar and unfamiliar faces in the time domain. Familiar and unfamiliar faces were presented periodically at 15 Hz, 6 Hz and 3.75 Hz either upright or inverted in separate blocks, with the rationale that faster frequencies require shorter processing times per image and tap into fundamentally different levels of visual processing. The 15 Hz trials, likely to reflect early visual processing, exhibited enhanced neural responses for familiar over unfamiliar face trials, but only when the faces were upright. In contrast, decoding methods revealed similar classification accuracies for upright and inverted faces for both familiar and unfamiliar faces. For the 6 Hz frequency, familiar faces had lower amplitude responses than unfamiliar faces, and decoding familiarity was more accurate for upright compared with inverted faces. Finally, the 3.75 Hz frequency revealed no main effects of familiarity, but decoding showed significant correlations with behavioral ratings of face familiarity, suggesting that activity evoked by this slow presentation frequency reflected higher-level, cognitive aspects of familiarity processing. This three-way dissociation between frequencies reveals that fundamentally different stages of the visual hierarchy are modulated by face familiarity. The combination of experimental and analytical approaches used here represent a novel method for elucidating spatio-temporal characteristics within the visual system.

Unfamiliar face matching with photographs of infants and children

BACKGROUND

Infants and children travel using passports that are typically valid for five years (e.g. Canada, United Kingdom, United States and Australia). These individuals may also need to be identified using images taken from videos and other sources in forensic situations including child exploitation cases. However, few researchers have examined how useful these images are as a means of identification.

METHODS

We investigated the effectiveness of photo identification for infants and children using a face matching task, where participants were presented with two images simultaneously and asked whether the images depicted the same child or two different children. In Experiment 1, both images showed an infant (<1 year old), whereas in Experiment 2, one image again showed an infant but the second image of the child was taken at 4-5 years of age. In Experiments 3a and 3b, we asked participants to complete shortened versions of both these tasks (selecting the most difficult trials) as well as the short version Glasgow face matching test. Finally, in Experiment 4, we investigated whether information regarding the sex of the infants and children could be accurately perceived from the images.

RESULTS

In Experiment 1, we found low levels of performance (72% accuracy) for matching two infant photos. For Experiment 2, performance was lower still (64% accuracy) when infant and child images were presented, given the significant changes in appearance that occur over the first five years of life. In Experiments 3a and 3b, when participants completed both these tasks, as well as a measure of adult face matching ability, we found lowest performance for the two infant tasks, along with mixed evidence of within-person correlations in sensitivities across all three tasks. The use of only same-sex pairings on mismatch trials, in comparison with random pairings, had little effect on performance measures. In Experiment 4, accuracy when judging the sex of infants was at chance levels for one image set and above chance (although still low) for the other set. As expected, participants were able to judge the sex of children (aged 4-5) from their faces.

DISCUSSION

Identity matching with infant and child images resulted in low levels of performance, which were significantly worse than for an adult face matching task. Taken together, the results of the experiments presented here provide evidence that child facial photographs are ineffective for use in real-world identification.

Variation and Likeness in Ambient Artistic Portraiture

An artist-led exploration of portrait accuracy and likeness involved 12 Artists producing 12 portraits referencing a life-size 3D print of the same Sitter. The works were assessed during a public exhibition, and the resulting likeness assessments were compared to portrait accuracy as measured using geometric morphometrics (statistical shape analysis). Our results are that, independently of the assessors' prior familiarity with the Sitter's face, the likeness judgements tended to be higher for less morphologically accurate portraits. The two highest rated were the portrait that most exaggerated the Sitter's distinctive features, and a portrait that was a more accurate (but not the most accurate) depiction. In keeping with research showing photograph likeness assessments involve recognition, we found familiar assessors rated the two highest ranked portraits even higher than those with some or no familiarity. In contrast, those lacking prior familiarity with the Sitter's face showed greater favour for the portrait with the highest morphological accuracy, and therefore most likely engaged in face-matching with the exhibited 3D print. Furthermore, our research indicates that abstraction in portraiture may not enhance likeness, and we found that when our 12 highly diverse portraits were statistically averaged, this resulted in a portrait that is more morphologically accurate than any of the individual artworks comprising the average.