Learning faces from variability

Multiple images captured from a single encounter do not promote face learning

Viewing multiple images of a newly encountered face improves recognition of that identity in new instances. Studies examining face learning have presented high-variability (HV) images that incorporate changes that occur from moment-to-moment (e.g., head orientation and expression) and over time (e.g., lighting, hairstyle, and health). We examined whether low-variability (LV) images (i.e., images that incorporate only moment-to-moment changes) also promote generalisation of learning such that novel instances are recognised. Participants viewed a single image, six LV images, or six HV images of a target identity before being asked to recognise novel images of that identity in a face matching task (training stimuli remained visible) or a memory task (training stimuli were removed). In Experiment 1 (n = 71), participants indicated which image(s) in 8-image arrays belonged to the target identity. In Experiment 2 (n = 73), participants indicated whether sequentially presented images belonged to the target identity. Relative to the single-image condition, sensitivity to identity improved and response biases were less conservative in the HV condition; we found no evidence of generalisation of learning in the LV condition regardless of testing protocol. Our findings suggest that day-to-day variability in appearance plays an essential role in acquiring expertise with a novel face.

When stimulus variability accelerates the learning of task knowledge in adults and school-aged children

Experience with instances that vary in their surface features helps individuals to form abstract task knowledge, leading to transfer of that knowledge to novel contexts. The current study sought to examine the role of this variability effect in how adults and school-aged children learn to engage cognitive control. We focused on the engagement of cognitive control in advance (proactive control) and in response to conflicts (reactive control) in a cued task-switching paradigm, and conducted four preregistered online experiments with adults (Experiment 1A: N = 100, Experiment 1B: N = 105) and 9- to 10-year-olds (Experiment 2A: N = 98, Experiment 2B: N = 97). It was shown that prior task experience of engaging reactive control makes both adults and 9- to 10-year-olds respond more slowly in a subsequent similar-structured condition with different stimuli in which proactive control could have been engaged. 9- to 10-year-olds (Experiment 2B) exhibited more negative transfer of a reactive control mode when uninformative cue and pre-target stimuli, which do not convey task-relevant information, were changed in each block, compared with when they were fixed. Furthermore, adults showed suggestive evidence of the variability effect both when cue and target stimuli were varied (Experiment 1A) and when uninformative cue and pre-target stimuli were varied (Experiment 1B). The collective findings of these experiments provide important insights into the contribution of stimulus variability to the engagement of cognitive control.

Using deep neural networks to disentangle visual and semantic information in human perception and memory

Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual-semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual-semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

The role of visual imagery in face recognition and the construction of facial composites. Evidence from Aphantasia

People with aphantasia have a markedly impaired ability to form visual images in the mind's eye. Here, by testing people with and without aphantasia, we examine the relationship between visual imagery and face processing. We show that aphantasics have weaker face recognition than people with visual imagery, using both self-report (Prosopagnosia Index) and behavioural measures (Cambridge Face Memory Test). However, aphantasics nonetheless have a fully intact ability to construct facial composites from memory (i.e., composites produced using EFIT6 by aphantasics and imagers were rated as equally accurate in terms of their resemblance to a target face). Additionally, we show that aphantasics were less able than imagers to see the resemblance between composites and a target face, suggestive of potential issues with face matching (perception). Finally, we show that holistic and featural methods of composite construction using EFIT6 produce equally accurate composites. Our results suggest that face recognition, but not face composite construction, is facilitated by the ability to represent visual properties as 'pictures in the mind'. Our findings have implications for the study of aphantasia, and also for forensic settings, where face composite systems are commonly used to aid criminal investigations.

Catching wanted people at the border: prospective person memory and face matching in border control decisions

Border control officers may be on the lookout for wanted people while they verify that travellers match their passport photos. We developed a novel experimental paradigm to investigate whether people are more likely to report that someone is wanted if they also believe that person is using a fraudulent passport. In two experiments, undergraduate students assumed the role of a border control officer and completed multiple "shifts" of a face matching task designed to simulate a passport verification check. Before each shift participants viewed posters of wanted people and were instructed to report any sightings if a wanted person appeared in any of the images during the passport check. Participants were more likely to say an individual was wanted if they also believed the person did not match their passport image. In addition, the accuracy of wanted person sightings was reduced for trials with nonmatching passports compared to trials with matching passports. This suggests wanted people with matching passports were easier to spot because participants had an additional image to compare with their memory of the person in the wanted poster.

The impact of stability in appearance on the development of facial representations

The way faces become familiar and what information is represented as familiarity develops has puzzled researchers in the field of human face recognition for decades. In this paper, we present three experiments serving as proof of concept for a cost-efficient mechanism of face learning describing how facial representations form over time and accounting for recognition errors. We propose that the encoding of facial information is dynamic and modulated by the intrinsic stability in individual faces' appearance. We drew on a robust and ecological method using a proxy of exposure to famous faces in the real world and manipulated test images to assess the prediction that recognition of famous faces is affected by their relative stability in appearance. We consistently show that stable facial appearances (like Tom Cruise's) facilitate recognition in early stages of familiarisation but that performance does not improve much over time. In contrast, variations in appearance (like Jared Leto's) hinder recognition at first but improve performance with further media exposure. This pattern of results is consistent with the proposed cost-efficient face learning mechanism whereby facial representations build on a foundation of large-scale diagnostic information and refine over time if needed. When coarse information loses its diagnostic value through the experience of variations in appearance across encounters, diagnostic facial details and/or their spatial relationships must receive more weights, leading to refined representations that are more discriminative and reliable than representations of stable faces.

Developing familiarity during the first eight months of knowing a person: A longitudinal EEG study on face and identity learning

It is well-established that familiar and unfamiliar faces are processed differently, but surprisingly little is known about how familiarity builds up over time and how novel faces gradually become represented in the brain. Here, we used event-related brain potentials (ERPs) in a pre-registered, longitudinal study to examine the neural processes accompanying face and identity learning during the first eight months of knowing a person. Specifically, we examined how increasing real-life familiarity affects visual recognition (N250 Familiarity Effect) and the integration of person-related knowledge (Sustained Familiarity Effect, SFE). Sixteen first-year undergraduates were tested in three sessions, approximately one, five, and eight months after the start of the academic year, with highly variable "ambient" images of a new friend they had met at university and of an unfamiliar person. We observed clear ERP familiarity effects for the new friend after one month of familiarity. While there was an increase in the N250 effect over the course of the study, no change in the SFE was observed. These results suggest that visual face representations develop faster relative to the integration of identity-specific knowledge.

Not the norm: Face likeness is not the same as similarity to familiar face prototypes

Face images depicting the same individual can differ substantially from one another. Ecological variation in pose, expression, lighting, and other sources of appearance variability complicates the recognition and matching of unfamiliar faces, but acquired familiarity leads to the ability to cope with these challenges. Among the many ways that face of the same individual can vary, some images are judged to be better likenesses of familiar individuals than others. Simply put, these images look more like the individual under consideration than others. But what does it mean for an image to be a better likeness than another? Does likeness entail typicality, or is it something distinct from this? We examined the relationship between the likeness of face images and the similarity of those images to average images of target individuals using a set of famous faces selected for reciprocal familiarity/unfamiliarity across US and UK participants. We found that though likeness judgments are correlated with similarity-to-prototype judgments made by both familiar and unfamiliar participants, this correlation was smaller than the correlation between similarity judgments made by different participant groups. This implies that while familiarity weakens the relationship between likeness and similarity-to-prototype judgments, it does not change similarity-to-prototype judgments to the same degree.

Provoked overt recognition in acquired prosopagnosia using multiple different images of famous faces

Provoked overt recognition refers to the fact that patients with acquired prosopagnosia can sometimes recognize faces when presented in arrays of individuals from the same category (e.g., actors or politicians). We ask whether a prosopagnosic patient might experience recognition when presented with multiple different images of the same face simultaneously. Over two sessions, patient Herschel, a 66-year-old British man with acquired prosopagnosia, viewed face images individually or in arrays. On several occasions he failed to recognize single photos of an individual but successfully identified that person when the same photos were presented together. For example, Herschel failed to recognize any individual images of King Charles or Paul McCartney but recognised both in arrays of the same photos. Like reports based on category membership, overt recognition was transient and inconsistent. These findings are discussed in terms of models of covert recognition, alongside more recent research on within-person variability for face perception.

Unfamiliar face matching ability predicts the slope of face learning

We provide the first examination of individual differences in the efficiency of face learning. Investigating individual differences in face learning can illuminate potential mechanisms and provide greater understanding of why certain individuals might be more efficient face learners. Participants completed two unfamiliar face matching tasks and a learning task in which learning was assessed after viewing 1, 3, 6, and 9 images of to-be-learned identities. Individual differences in the slope of face learning (i.e., increases in sensitivity to identity) were predicted by the ability to discriminate between matched (same-identity) vs. mismatched (different-identity) pairs of wholly unfamiliar faces. A Dual Process Signal Detection model showed that three parameters increased with learning: Familiarity (an unconscious type of memory that varies in strength), recollection-old (conscious recognition of a learned identity), and recollection-new (conscious/confident rejection of novel identities). Good (vs. poor) matchers had higher Recollection-Old scores throughout learning and showed a steeper increase in Recollection-New. We conclude that good matchers are better able to capitalize on exposure to within-person variability in appearance, an effect that is attributable to their conscious memory for both learned and novel faces. These results have applied implications and will inform contemporary and traditional models of face identification.

Stable individual differences in unfamiliar face identification: Evidence from simultaneous and sequential matching tasks

Matching identity in images of unfamiliar faces is difficult: Images of the same person can look different and images of different people can look similar. Recent studies have capitalized on individual differences in the ability to distinguish match (same ID) vs. mismatch (different IDs) face pairs to inform models of face recognition. We addressed two significant gaps in the literature by examining the stability of individual differences in both sensitivity to identity and response bias. In Study 1, 210 participants completed a battery of four tasks in each of two sessions separated by one week. Tasks varied in protocol (same/different, lineup, sorting) and stimulus characteristics (low vs. high within-person variability in appearance). In Study 2, 148 participants completed a battery of three tasks in a single session. Stimuli were presented simultaneously on some trials and sequentially on others, introducing short-term memory demands. Principal components analysis revealed two components that were stable across time and tasks: sensitivity to identity and bias. Analyses of response times suggest that individual differences in bias reflect decision-making processes. We discuss the implications of our findings in applied settings and for models of face recognition.

How quickly do we learn new faces in everyday life? Neurophysiological evidence for face identity learning after a brief real-life encounter

Faces learnt in a single experimental session elicit a familiarity effect in event-related brain potentials (ERPs), with more negative amplitudes for newly learnt relative to unfamiliar faces in the N250 component. However, no ERP study has examined face learning following a brief real-life encounter, and it is not clear how long it takes to learn new faces in such ecologically more valid conditions. To investigate these questions, the present study examined whether robust image-independent representations, as reflected in the N250 familiarity effect, could be established after a brief unconstrained social interaction by analysing the ERPs elicited by highly variable images of the newly learnt identity and an unfamiliar person. Significant N250 familiarity effects were observed after a 30-min (Experiment 1) and a 10-min (Experiment 2) encounter, and a trend was observed after 5 min of learning (Experiment 3), demonstrating that 5-10 min of exposure were sufficient for the initial establishment of image-independent representations. Additionally, the magnitude of the effects reported after 10 and 30 min was comparable suggesting that the first 10 min of a social encounter might be crucial, with extra 20 min from the same encounter not adding further benefit for the initial formation of robust face representations.

From Pictures to the People in Them: Averaging Within-Person Variability Leads to Face Familiarization

Familiar faces can be confidently recognized despite sometimes radical changes in their appearance. Exposure to within-person variability-differences in facial characteristics over successive encounters-contributes to face familiarization. Research also suggests that viewers create mental averages of the different views of faces they encounter while learning them. Averaging over within-person variability is thus a promising mechanism for face familiarization. In Experiment 1, 153 Canadian undergraduates (88 female; age: M = 21 years, SD = 5.24) learned six target identities from eight different photos of each target interspersed among 32 distractor identities. Face-matching accuracy improved similarly irrespective of awareness of the target's identity, confirming that target faces presented among distractors can be learned incidentally. In Experiment 2, 170 Canadian undergraduates (125 female; age: M = 22.6 years, SD = 6.02) were tested using a novel indirect measure of learning. The results show that viewers update a mental average of a person's face as it becomes learned. Our findings are the first to show how averaging within-person variability over time leads to face familiarization.

Two face masks are better than one: congruency effects in face matching

Although the positive effects of congruency between stimuli are well replicated in face memory paradigms, mixed findings have been found in face matching. Due to the current COVID-19 pandemic, face masks are now very common during daily life outdoor activities. Thus, the present study aims to further explore congruency effects in matching faces partially occluded by surgical masks. Observers performed a face matching task consisting of pairs of faces presented in full view (i.e., full-view condition), pairs of faces in which only one of the faces had a mask (i.e., one-mask condition), and pairs of faces in which both faces had a mask (i.e., two-mask condition). Although face masks disrupted performance in identity match and identity mismatch trials, in match trials, we found better performance in the two-mask condition compared to the one-mask condition. This finding highlights the importance of congruency between stimuli on face matching when telling faces together.

Perceptual similarity modulates effects of learning from variability on face recognition

Face recognition is a challenging classification task that humans perform effortlessly for familiar faces. Recent studies have emphasized the importance of exposure to high variability appearances of the same identity to perform this task. However, these studies did not explicitly measure the perceptual similarity between the learned images and the images presented at test, which may account for the advantage of learning from high variability. Particularly, randomly selected test images are more likely to be perceptually similar to learned high variability images, and dissimilar to learned low variability images. Here we dissociated effects of learning from variability and study-test perceptual similarity, by collecting human similarity ratings for the study and test images. Using these measures, we independently manipulated the variability between the learning images and their perceptual similarity to the test images. Different groups of participants learned face identities from a low or high variability set of images. The learning phase was followed by a face matching test (Experiment 1) or a face recognition task (Experiment 2) that presented novel images of the learned identities that were perceptually dissimilar or similar to the learned images. Results of both experiments show that perceptual similarity between study and test, rather than image variability at learning per se, predicts face recognition. We conclude that learning from high variability improves face recognition for perceptually similar but not for perceptually dissimilar images. These findings may not be specific to faces and should be similarly evaluated for other domains.

Social-encoding benefit in face recognition is generalized to other-race faces

Faces are visual stimuli that convey rich social information. Previous experiments found better recognition for faces that were evaluated based on their social traits than on their perceptual features during encoding. Here, we ask whether this social-encoding benefit in face recognition is also found for categories of faces that we have no previous social experience with, such as other-race faces. To answer this question, we first explored whether social and perceptual evaluations for other-race faces are consistent and valid. We then asked whether social evaluations during encoding improve recognition for other-race faces. Results show that social and perceptual evaluations of own- and other-race faces were valid. We also found high agreement in social and perceptual evaluations across individuals from different races. This indicates that evaluations of other-race faces are not random but meaningful. Furthermore, we found that social evaluations facilitated face recognition regardless of race, demonstrating a social-encoding benefit for both own- and other-race faces. Our findings highlight the role of social information in face recognition and show how it can be used to improve recognition of categories of faces that are hard to recognize due to lack of experience with them.

Children's ability to recognize their parent's face improves with age

Adults are experts at recognizing familiar faces across images that incorporate natural within-person variability in appearance (i.e., ambient images). Little is known about children's ability to do so. In the current study, we investigated whether 4- to 7-year-olds (n = 56) could recognize images of their own parent-a person with whom children have had abundant exposure in a variety of different contexts. Children were asked to identify images of their parent that were intermixed with images of other people. We included images of each parent taken both before and after their child was born to manipulate how close the images were to the child's own experience. When viewing before-birth images, 4- and 5-year-olds were less sensitive to identity than were older children; sensitivity did not differ when viewing images taken after the child was born. These findings suggest that with even the most familiar face, 4- and 5-year-olds have difficulty recognizing instances that go beyond their direct experience. We discuss two factors that may contribute to the prolonged development of familiar face recognition.

Picture this: Photographers no better than controls for recognizing unfamiliar faces

With the exception of super recognizers and forensic examiners, people make a surprising number of errors when deciding whether photographs of unfamiliar faces belong to the same person or different people. Training protocols designed to improve professionals’ (e.g., passport officers) performance often include photography. We evaluated the influence of life-time photography experience on the ability to distinguish matched versus mismatched face pairs. Expert photographers were not more sensitive to identity than hobbyists or novices—despite specializing in human subjects; Hobbyists were more liberal (more same responses) than Experts. We conclude that photography experience is not a route to expertise.

The pairs training effect in unfamiliar face matching

A wealth of studies have shown that humans are remarkably poor at determining whether two face images show the same person or not (face matching). Given the prevalence of photo-ID, and the fact that people employed to check photo-ID are typically unfamiliar with the person pictured, there is a need to improve unfamiliar face matching accuracy. One method of improvement is to have participants complete the task in a pair, which results in subsequent improvements in the low performer (“the pairs training effect”). Here, we sought to replicate the original finding, to test the longevity of the pairs training effect, and to shed light on the potential underlying mechanisms. In two experiments, we replicated the pairs training effect and showed it is maintained after a delay (Experiment 1). We found no differences between high and low performers in confidence (Experiment 1) or response times (Experiment 2), and the content of the pairs’ discussions (Experiment 2) did not explain the results. The pairs training effect in unfamiliar face matching is robust, but the mechanisms underlying the effects remain as yet unexplained.

Face learning via brief real-world social interactions includes changes in face-selective brain areas and hippocampus

Making new acquaintances requires learning to recognise previously unfamiliar faces. In the current study, we investigated this process by staging real-world social interactions between actors and the participants. Participants completed a face-matching behavioural task in which they matched photographs of the actors (whom they had yet to meet), or faces similar to the actors (henceforth called foils). Participants were then scanned using functional magnetic resonance imaging (fMRI) while viewing photographs of actors and foils. Immediately after exiting the scanner, participants met the actors for the first time and interacted with them for 10 min. On subsequent days, participants completed a second behavioural experiment and then a second fMRI scan. Prior to each session, actors again interacted with the participants for 10 min. Behavioural results showed that social interactions improved performance accuracy when matching actor photographs, but not foil photographs. The fMRI analysis revealed a difference in the neural response to actor photographs and foil photographs across all regions of interest (ROIs) only after social interactions had occurred. Our results demonstrate that short social interactions were sufficient to learn and discriminate previously unfamiliar individuals. Moreover, these learning effects were present in brain areas involved in face processing and memory.

Impaired grouping of ambient facial images in autism

Ambient facial images depict individuals from a variety of viewing angles, with a range of poses and expressions, under different lighting conditions. Exposure to ambient images is thought to help observers form robust representations of the individuals depicted. Previous results suggest that autistic people may derive less benefit from exposure to this exemplar variation than non-autistic people. To date, however, it remains unclear why. One possibility is that autistic individuals possess atypical perceptual learning mechanisms. Alternatively, however, the learning mechanisms may be intact, but receive low-quality perceptual input from face encoding processes. To examine this second possibility, we investigated whether autistic people are less able to group ambient images of unfamiliar individuals based on their identity. Participants were asked to identify which of four ambient images depicted an oddball identity. Each trial assessed the grouping of different facial identities, thereby preventing face learning across trials. As such, the task assessed participants’ ability to group ambient images of unfamiliar people. In two experiments we found that matched non-autistic controls correctly identified the oddball identities more often than our autistic participants. These results imply that poor face learning from variation by autistic individuals may well be attributable to low-quality perceptual input, not aberrant learning mechanisms.

What happens to our representation of identity as familiar faces age? Evidence from priming and identity aftereffects

Matching identity in images of unfamiliar faces is error prone, but we can easily recognize highly variable images of familiar faces – even images taken decades apart. Recent theoretical development based on computational modelling can account for how we recognize extremely variable instances of the same identity. We provide complementary behavioural data by examining older adults’ representation of older celebrities who were also famous when young. In Experiment 1, participants completed a long‐lag repetition priming task in which primes and test stimuli were the same age or different ages. In Experiment 2, participants completed an identity after effects task in which the adapting stimulus was an older or young photograph of one celebrity and the test stimulus was a morph between the adapting identity and a different celebrity; the adapting stimulus was the same age as the test stimulus on some trials (e.g., both old) or a different age (e.g., adapter young, test stimulus old). The magnitude of priming and identity after effects were not influenced by whether the prime and adapting stimulus were the same age or different age as the test face. Collectively, our findings suggest that humans have one common mental representation for a familiar face (e.g., Paul McCartney) that incorporates visual changes across decades, rather than multiple age‐specific representations. These findings make novel predictions for state‐of‐the‐art algorithms (e.g., Deep Convolutional Neural Networks).

Learning faces from variability: Four- and five-year-olds differ from older children and adults

Children under 6 years of age have difficulty recognizing a familiar face across changes in appearance and telling the face apart from similar-looking people. Understanding the process by which newly encountered faces become familiar can provide insights into these difficulties. Exposure to the ways in which a person varies in appearance is one mechanism by which adults and older children (≥6 years) learn new faces. We provide the first investigation of whether this mechanism for face learning functions in younger children. Children aged 4 and 5 years were read two storybooks featuring an unfamiliar character. Participants viewed six images of the character in one story and one image of the character in the other story. After each story, children were asked to identify novel images of the character that were intermixed with images of a similar-looking distractor. Like older children, 4- and 5-year-olds were more sensitive to identity in the 6-image condition, but they also adapted a less conservative criterion. Young children identified more images of the character after viewing six images versus one image. However, many also incorrectly identified more images of the distractor after viewing six images versus one image, an effect not previously found for older children and adults. These results suggest that this mechanism for face learning is not fully refined before 6 years of age.

Naturalistic Face Learning in Infants and Adults

Everyday face recognition presents a difficult challenge because faces vary naturally in appearance as a result of changes in lighting, expression, viewing angle, and hairstyle. We know little about how humans develop the ability to learn faces despite natural facial variability. In the current study, we provide the first examination of attentional mechanisms underlying adults' and infants' learning of naturally varying faces. Adults (n = 48) and 6- to 12-month-old infants (n = 48) viewed videos of models reading a storybook; the facial appearance of these models was either high or low in variability. Participants then viewed the learned face paired with a novel face. Infants showed adultlike prioritization of face over nonface regions; both age groups fixated the face region more in the high- than low-variability condition. Overall, however, infants showed less ability to resist contextual distractions during learning, which potentially contributed to their lack of discrimination between the learned and novel faces. Mechanisms underlying face learning across natural variability are discussed.

Comparing the sensitivity of face matching assessments to detect face perception impairments

Numerous neurological, developmental, and psychiatric conditions demonstrate impaired face recognition, which can be socially debilitating. These impairments can be caused by either deficient face perception or face memory mechanisms. Though there are well-validated, sensitive measures of face memory impairments, it currently remains unclear which assessments best measure face perception impairments. A sensitive, validated face perception measure could help with diagnosing causes of face recognition deficits and be useful in characterizing individual differences in unimpaired populations. Here, we compared the computerized Benton Face Recognition Test (BFRT-c) and Cambridge Face Perception Test (CFPT) in their ability to differentiate developmental prosopagnosics (DPs, N = 30) and age-matched controls (N = 30). Participants completed the BFRT-c, CFPT, and two additional face perception assessments: the University of Southern California Face Perception Test (USCFPT) and a novel same/different face matching test (SDFMT). Participants were also evaluated on objective and subjective face recognition tasks including the Cambridge Face Memory Test, famous faces test, and Prosopagnosia Index-20. We performed a logistic regression with the perception tests predicting DP vs. control group membership and used multiple linear regressions to predict continuous objective and subjective face recognition memory. Our results show that the BFRT-c performed as well as, if not better than, the CFPT, and that both tests clearly outperformed the USCFPT and SDFMT. Further, exploratory analyses revealed that face lighting-change conditions better predicted DP group membership and face recognition abilities than viewpoint-change conditions. Together, these results support the combined use of the BFRT-c and CFPT to best assess face perception impairments.

The Episodic Prototypes Model (EPM): On the nature and genesis of facial representations

Faces undergo massive changes over time and life events. We need a mental representation which is flexible enough to cope with the existing visual varieties, but which is also stable enough to be the basis for valid recognition. Two main theoretical frameworks exist to describe facial representations: prototype models assuming one central item comprising all visual experiences of a face, and exemplar models assuming single representations of each visual experience of a face. We introduce a much more ecological valid model dealing with episodic prototypes (the Episodic Prototypes Model—EPM), where faces are represented by a low number of prototypes that refer to specific Episodes of Life (EoL, e.g., early adulthood, mature age) during which the facial appearance shows only moderate variation. Such an episodic view of mental representation allows for efficient storage, as the number of needed prototypes is relatively low, and it allows for the needed variation within a prototype that keeps the everyday and steadily ongoing changes across a certain period of time. Studies 1–3 provide evidence that facial representations are highly dependent on temporal aspects which is in accord with EoL, and that individual learning history generates the structure and content of respective prototypes. In Study 4, we used implicit measures (RT) in a face verification task to investigate the postulated power of the EPM. We could demonstrate that episodic prototypes clearly outperformed visual depictions of exhaustive prototypes, supporting the general idea of our approach.

Partitioning natural face image variability emphasises within-identity over between-identity representation for understanding accurate recognition

Accurately recognising faces enables social interactions. In recent years it has become clear that people's accuracy differs markedly depending on viewer's familiarity with a face and their individual skill, but the cognitive and neural bases of these accuracy differences are not understood. We examined cognitive representations underlying these accuracy differences by measuring similarity ratings to natural facial image variation. Natural variation was sampled from uncontrolled images on the internet to reflect the appearance of faces as they are encountered in daily life. Using image averaging, and inspired by the computation of Analysis of Variance, we partitioned this variation into differences between faces (between-identity variation) and differences between photos of the same face (within-identity variation). This allowed us to compare modulation of these two sources of variation attributable to: (i) a person's familiarity with a face and, (ii) their face recognition ability. Contrary to prevailing accounts of human face recognition and perceptual learning, we found that modulation of within-identity variation - rather than between-identity variation - was associated with high accuracy. First, familiarity modulated similarity ratings to within-identity variation more than to between-face variation. Second, viewers that are extremely accurate in face recognition - 'super-recognisers' - differed from typical perceivers mostly in their ratings of within-identity variation, compared to between-identity variation. In a final computational analysis, we found evidence that transformations of between- and within-identity variation make separable contributions to perceptual expertise in face recognition. We conclude that inter- and intra-individual accuracy differences primarily arise from differences in the representation of within-identity image variation.

Within-person variability can improve the identification of unfamiliar faces across changes in viewpoint

Unfamiliar face identification is concerningly error prone, especially across changes in viewing conditions. Within-person variability has been shown to improve matching performance for unfamiliar faces, but this has only been demonstrated using images of a front view. In this study, we test whether the advantage of within-person variability from front views extends to matching to target images of a face rotated in view. Participants completed either a simultaneous matching task (Experiment 1) or a sequential matching task (Experiment 2) in which they were tested on their ability to match the identity of a face shown in an array of either one or three ambient front-view images, with a target image shown in front, three-quarter, or profile view. While the effect was stronger in Experiment 2, we found a consistent pattern in match trials across both experiments in that there was a multiple image matching benefit for front, three-quarter, and profile-view targets. We found multiple image effects for match trials only, indicating that providing observers with multiple ambient images confers an advantage for recognising different images of the same identity but not for discriminating between images of different identities. Signal detection measures also indicate a multiple image advantage despite a more liberal response bias for multiple image trials. Our results show that within-person variability information for unfamiliar faces can be generalised across views and can provide insights into the initial processes involved in the representation of familiar faces.

Detecting a viewer's familiarity with a face: Evidence from event-related brain potentials and classifier analyses

Human observers recognize the faces of people they know efficiently and without apparent effort. Consequently, recognizing a familiar face is often assumed to be an automatic process beyond voluntary control. However, there are circumstances in which a person might seek to hide their recognition of a particular face. The present study therefore used event-related potentials (ERPs) and a classifier based on logistic regression to determine if it is possible to detect whether a viewer is familiar with a particular face, regardless of whether the participant is willing to acknowledge it or not. In three experiments, participants were presented with highly variable "ambient" images of personally familiar and unfamiliar faces, while performing an incidental butterfly detection task (Experiment 1), an explicit familiarity judgment task (Experiment 2), and a concealed familiarity task in which they were asked to deny familiarity with one truly known facial identity while acknowledging familiarity with a second known identity (Experiment 3). In all three experiments, we observed substantially more negative ERP amplitudes at occipito-temporal electrodes for familiar relative to unfamiliar faces starting approximately 200 ms after stimulus onset. Both the earlier N250 familiarity effect, reflecting visual recognition of a known face, and the later sustained familiarity effect, reflecting the integration of visual with additional identity-specific information, were similar across experiments and thus independent of task demands. These results were further supported by the classifier analysis. We conclude that ERP correlates of familiar face recognition are largely independent of voluntary control and discuss potential applications in forensic settings.

Visual search performance in 'CCTV' and mobile phone-like video footage

Finding an unfamiliar person in a crowd of others is an integral task for police officers, CCTV-operators, and security staff who may be looking for a suspect or missing person; however, research suggests that it is difficult and accuracy in such tasks is low. In two real-world visual-search experiments, we examined whether being provided with four images versus one image of an unfamiliar target person would help improve accuracy when searching for that person through video footage. In Experiment 1, videos were taken from above and at a distance to simulate CCTV, and images of the target showed their face and torso. In Experiment 2, videos were taken from approximately shoulder height, such as one would expect from body-camera or mobile phone recordings, and target images included only the face. Our findings suggest that having four images as exemplars leads to higher accuracy in the visual search tasks, but this only reached significance in Experiment 2. There also appears to be a conservative bias whereby participants are more likely to respond that the target is not in the video when presented with only one image as opposed to 4. These results point to there being an advantage for providing multiple images of targets for use in video visual-search.

Face Recognition by Humans and Machines: Three Fundamental Advances from Deep Learning

Deep learning models currently achieve human levels of performance on real-world face recognition tasks. We review scientific progress in understanding human face processing using computational approaches based on deep learning. This review is organized around three fundamental advances. First, deep networks trained for face identification generate a representation that retains structured information about the face (e.g., identity, demographics, appearance, social traits, expression) and the input image (e.g., viewpoint, illumination). This forces us to rethink the universe of possible solutions to the problem of inverse optics in vision. Second, deep learning models indicate that high-level visual representations of faces cannot be understood in terms of interpretable features. This has implications for understanding neural tuning and population coding in the high-level visual cortex. Third, learning in deep networks is a multistep process that forces theoretical consideration of diverse categories of learning that can overlap, accumulate over time, and interact. Diverse learning types are needed to model the development of human face processing skills, cross-race effects, and familiarity with individual faces.

Forgetting faces over a week: investigating self-reported face recognition ability and personality

Background

Although face recognition is now well studied, few researchers have considered the nature of forgetting over longer time periods. Here, I investigated how newly learned faces were recognised over the course of one week. In addition, I considered whether self-reported face recognition ability, as well as Big Five personality dimensions, were able to predict actual performance in a recognition task.

Methods

In this experiment (N = 570), faces were learned through short video interviews, and these identities were later presented in a recognition test (using previously unseen images) after no delay, six hours, twelve hours, one day, or seven days.

Results

The majority of forgetting took place within the first 24 hours, with no significant decrease after that timepoint. Further, self-reported face recognition abilities were moderately predictive of performance, while extraversion showed a small, negative association with performance. In both cases, these associations remained consistent across delay conditions.

Discussion

The current work begins to address important questions regarding face recognition using longitudinal, real-world time intervals, focussing on participant insight into their own abilities, and the process of forgetting more generally.

Unimodal and cross-modal identity judgements using an audio-visual sorting task: Evidence for independent processing of faces and voices

Unimodal and cross-modal information provided by faces and voices contribute to identity percepts. To examine how these sources of information interact, we devised a novel audio-visual sorting task in which participants were required to group video-only and audio-only clips into two identities. In a series of three experiments, we show that unimodal face and voice sorting were more accurate than cross-modal sorting: While face sorting was consistently most accurate followed by voice sorting, cross-modal sorting was at chancel level or below. In Experiment 1, we compared performance in our novel audio-visual sorting task to a traditional identity matching task, showing that unimodal and cross-modal identity perception were overall moderately more accurate than the traditional identity matching task. In Experiment 2, separating unimodal from cross-modal sorting led to small improvements in accuracy for unimodal sorting, but no change in cross-modal sorting performance. In Experiment 3, we explored the effect of minimal audio-visual training: Participants were shown a clip of the two identities in conversation prior to completing the sorting task. This led to small, nonsignificant improvements in accuracy for unimodal and cross-modal sorting. Our results indicate that unfamiliar face and voice perception operate relatively independently with no evidence of mutual benefit, suggesting that extracting reliable cross-modal identity information is challenging.

The sibling familiarity effect: Is within-person facial variability shared across siblings?

Humans are experts at familiar face recognition, but poor at unfamiliar face recognition. Familiarity is created when a face is encountered across varied conditions, but the way in which a person's appearance varies is identity-specific, so familiarity with one identity does not benefit recognition of other individuals. However, the faces of biological siblings share structural similarities, so we explored whether the benefits of familiarity are shared across siblings. Results show that familiarity with one half of a sibling pair improves kin detection (experiment 1), and that unfamiliar face matching is more accurate when targets are the siblings of familiar versus unfamiliar individuals (experiment 2). PCA applied to facial images of celebrities and their siblings demonstrates that faces are generally better reconstructed in the principal components of a same-sex sibling than those of an unrelated individual. When we encounter the unfamiliar sibling of someone we already know, our pre-existing representation of their familiar relation may usefully inform processing of the unfamiliar face. This can benefit both kin detection and identity processing, but the benefits are constrained by the degree to which facial variability is shared.

The effect of face masks and sunglasses on identity and expression recognition with super-recognizers and typical observers

Face masks present a new challenge to face identification (here matching) and emotion recognition in Western cultures. Here, we present the results of three experiments that test the effect of masks, and also the effect of sunglasses (an occlusion that individuals tend to have more experienced with) on (i) familiar face matching, (ii) unfamiliar face matching and (iii) emotion categorization. Occlusion reduced accuracy in all three tasks, with most errors in the mask condition; however, there was little difference in performance for faces in masks compared with faces in sunglasses. Super-recognizers, people who are highly skilled at matching unconcealed faces, were impaired by occlusion, but at the group level, performed with higher accuracy than controls on all tasks. Results inform psychology theory with implications for everyday interactions, security and policing in a mask-wearing society.

Multiple-image arrays in face matching tasks with and without memory

Previous research has shown that exposure to within-person variability facilitates face learning. A different body of work has examined potential benefits of providing multiple images in face matching tasks. Viewers are asked to judge whether a target face matches a single face image (as when checking photo-ID) or multiple face images of the same person. The evidence here is less clear, with some studies finding a small multiple-image benefit, and others finding no advantage. In four experiments, we address this discrepancy in the benefits of multiple images from learning and matching studies. We show that multiple-image arrays only facilitate face matching when arrays precede targets. Unlike simultaneous face matching tasks, sequential matching and learning tasks involve memory and require abstraction of a stable representation of the face from the array, for subsequent comparison with a target. Our results show that benefits from multiple-image arrays occur only when this abstraction is required, and not when array and target images are available at once. These studies reconcile apparent differences between face learning and face matching and provide a theoretical framework for the study of within-person variability in face perception.

Seeing through disguise: Getting to know you with a deep convolutional neural network

People use disguise to look unlike themselves (evasion) or to look like someone else (impersonation). Evasion disguise challenges human ability to see an identity across variable images; Impersonation challenges human ability to tell people apart. Personal familiarity with an individual face helps humans to see through disguise. Here we propose a model of familiarity based on high-level visual learning mechanisms that we tested using a deep convolutional neural network (DCNN) trained for face identification. DCNNs generate a face space in which identities and images co-exist in a unified computational framework, that is categorically structured around identity, rather than retinotopy. This allows for simultaneous manipulation of mechanisms that contrast identities and cluster images. In Experiment 1, we measured the DCNN's baseline accuracy (unfamiliar condition) for identification of faces in no disguise and disguise conditions. Disguise affected DCNN performance in much the same way it affects human performance for unfamiliar faces in disguise (cf. Noyes & Jenkins, 2019). In Experiment 2, we simulated familiarity for individual identities by averaging the DCNN-generated representations from multiple images of each identity. Averaging improved DCNN recognition of faces in evasion disguise, but reduced the ability of the DCNN to differentiate identities of similar appearance. In Experiment 3, we implemented a contrast learning technique to simultaneously teach the DCNN appearance variation and identity contrasts between different individuals. This facilitated identification with both evasion and impersonation disguise. Familiar face recognition requires an ability to group images of the same identity together and separate different identities. The deep network provides a high-level visual representation for face recognition that supports both of these mechanisms of face learning simultaneously.

Contrast Negation Impairs Sorting Unfamiliar Faces by Identity: A Comparison With Original (Contrast-Positive) and Stretched Images

Recognition of unfamiliar faces is difficult in part due to variations in expressions, angles, and image quality. Studies suggest shape and surface properties play varied roles in face learning, and identification of unfamiliar faces uses diagnostic pigmentation/surface reflectance relative to shape information. Here, participants sorted photo-cards of unfamiliar faces by identity, which were shown in their original, stretched, and contrast-negated forms, to examine the utility of diagnostic shape and surface properties in sorting unfamiliar faces by identity. In four experiments, we varied the presentation order of conditions (contrast-negated first or original first with stretched second across experiments) and whether the same or different photo-cards were seen across conditions. Stretching the images did not impair performance in any measures relative to other conditions. Contrast negation generally exacerbated poor sorting by identity compared with the other conditions. However, seeing the contrast-negated photo-cards last mitigated some of the effects of contrast negation. Together, results suggest an important role for surface properties such as pigmentation and reflectance for sorting by identity and add to literatures on informational content and appearance variability in discrimination of facial identity.

Understanding face identification through within-person variability in appearance: Introduction to a virtual special issue

In the effort to determine the cognitive processes underlying the identification of faces, the dissimilarities between images of different people have long been studied. In contrast, the inherent variability between different images of the same face has either been treated as a nuisance variable that should be eliminated from psychological experiments or it has not been considered at all. Over the past decade, research efforts have increased substantially to demonstrate that this within-person variation is meaningful and can give insight into various processes of face identification, such as identity matching, face learning, and familiar face recognition. In this virtual special issue of the Quarterly Journal of Experimental Psychology, we explain the importance of within-person variability for face identification and bring together recent relevant articles published in the journal.

May I Speak Freely? The Difficulty in Vocal Identity Processing Across Free and Scripted Speech

In the fields of face recognition and voice recognition, a growing literature now suggests that the ability to recognize an individual despite changes from one instance to the next is a considerable challenge. The present paper reports on one experiment in the voice domain designed to determine whether a change in the mere style of speech may result in a measurable difficulty when trying to discriminate between speakers. Participants completed a speaker discrimination task to pairs of speech clips, which represented either free speech or scripted speech segments. The results suggested that speaker discrimination was significantly better when the style of speech did not change compared to when it did change, and was significantly better from scripted than from free speech segments. These results support the emergent body of evidence suggesting that within-identity variability is a challenge, and the forensic implications of such a mild change in speech style are discussed.

Adults' facial impressions of children's niceness, but not shyness, show modest accuracy

Lay wisdom warns against "judging a book by its cover." However, facial first impressions influence people's behaviour towards others, so it is critical that we understand whether these impressions are at all accurate. Understanding impressions of children's faces is particularly important because these impressions can have social consequences during a crucial time of development. Here, we examined the accuracy of two traits that capture the most variance in impressions of children's faces, niceness and shyness. We collected face images and parental reports of actual niceness/shyness for 86 children (4-11 years old). Different images of the same person can lead to different impressions, and so we employed a novel approach by obtaining impressions from five images of each child. These images were ambient, representing the natural variability in faces. Adult strangers rated the faces for niceness (Study 1) or shyness (Study 2). Niceness impressions were modestly accurate for different images of the same child, regardless of whether these images were presented individually or simultaneously as a group. Shyness impressions were not accurate, for images presented either individually or as a group. Together, these results demonstrate modest accuracy in adults' impressions of niceness, but not shyness, from children's faces. Furthermore, our results reveal that this accuracy can be captured by images which contain natural face variability, and holds across different images of the same child's face. These results invite future research into the cues and causal mechanisms underlying this link between facial impressions of niceness and nice behaviour in children.

Body Language Influences on Facial Identification at Passport Control: An Exploration in Virtual Reality

Person identification at airports requires the matching of a passport photograph to its bearer. One aim of this process is to find identity impostors, who use valid identity documents of similar-looking people to avoid detection. In psychology, this process has been studied extensively with static pairs of face photographs that require identity match (same person shown) versus mismatch (two different people) decisions. However, this approach provides a limited proxy for studying how other factors, such as nonverbal behaviour, affect this task. The current study investigated the influence of body language on facial identity matching within a virtual reality airport environment, by manipulating activity levels of person avatars queueing at passport control. In a series of six experiments, detection of identity mismatches was unaffected when observers were not instructed to utilise body language. By contrast, under explicit instruction to look out for unusual body language, these cues enhanced detection of mismatches but also increased false classification of matches. This effect was driven by increased activity levels rather than body language that simply differed from the behaviour of the majority of passengers. The implications and limitations of these findings are discussed.

Eye spy a liar: assessing the utility of eye fixations and confidence judgments for detecting concealed recognition of faces, scenes and objects

BACKGROUND

In criminal investigations, uncooperative witnesses might deny knowing a perpetrator, the location of a murder scene or knowledge of a weapon. We sought to identify markers of recognition in eye fixations and confidence judgments whilst participants told the truth and lied about recognising faces (Experiment 1) and scenes and objects (Experiment 2) that varied in familiarity. To detect recognition we calculated effect size differences in markers of recognition between familiar and unfamiliar items that varied in familiarity (personally familiar, newly learned).

RESULTS

In Experiment 1, recognition of personally familiar faces was reliably detected across multiple fixation markers (e.g. fewer fixations, fewer interest areas viewed, fewer return fixations) during honest and concealed recognition. In Experiment 2, recognition of personally familiar non-face items (scenes and objects) was detected solely by fewer fixations during honest and concealed recognition; differences in other fixation measures were not consistent. In both experiments, fewer fixations exposed concealed recognition of newly learned faces, scenes and objects, but the same pattern was not observed during honest recognition. Confidence ratings were higher for recognition of personally familiar faces than for unfamiliar faces.

CONCLUSIONS

Robust memories of personally familiar faces were detected in patterns of fixations and confidence ratings, irrespective of task demands required to conceal recognition. Crucially, we demonstrate that newly learned faces should not be used as a proxy for real-world familiarity, and that conclusions should not be generalised across different types of familiarity or stimulus class.