Contributions of feature shapes and surface cues to the recognition and neural representation of facial identity

The Neural Dynamics of Face Ensemble and Central Face Processing

Extensive work has investigated the neural processing of single faces, including the role of shape and surface properties. However, much less is known about the neural basis of face ensemble perception (e.g., simultaneously viewing several faces in a crowd). Importantly, the contribution of shape and surface properties have not been elucidated in face ensemble processing. Furthermore, how single central faces are processed within the context of an ensemble remains unclear. Here, we probe the neural dynamics of ensemble representation using pattern analyses as applied to electrophysiology data in healthy adults (seven males, nine females). Our investigation relies on a unique set of stimuli, depicting different facial identities, which vary parametrically and independently along their shape and surface properties. These stimuli were organized into ensemble displays consisting of six surround faces arranged in a circle around one central face. Overall, our results indicate that both shape and surface properties play a significant role in face ensemble encoding, with the latter demonstrating a more pronounced contribution. Importantly, we find that the neural processing of the center face precedes that of the surround faces in an ensemble. Further, the temporal profile of center face decoding is similar to that of single faces, while those of single faces and face ensembles diverge extensively from each other. Thus, our work capitalizes on a new center-surround paradigm to elucidate the neural dynamics of ensemble processing and the information that underpins it. Critically, our results serve to bridge the study of single and ensemble face perception.

A narrow band of image dimensions is critical for face recognition

A key challenge in human and computer face recognition is to differentiate information that is diagnostic for identity from other sources of image variation. Here, we used a combined computational and behavioural approach to reveal critical image dimensions for face recognition. Behavioural data were collected using a sorting and matching task with unfamiliar faces and a recognition task with familiar faces. Principal components analysis was used to reveal the dimensions across which the shape and texture of faces in these tasks varied. We then asked which image dimensions were able to predict behavioural performance across these tasks. We found that the ability to predict behavioural responses in the unfamiliar face tasks increased when the early PCA dimensions (i.e. those accounting for most variance) of shape and texture were removed from the analysis. Image similarity also predicted the output of a computer model of face recognition, but again only when the early image dimensions were removed from the analysis. Finally, we found that recognition of familiar faces increased when the early image dimensions were removed, decreased when intermediate dimensions were removed, but then returned to baseline recognition when only later dimensions were removed. Together, these findings suggest that early image dimensions reflect ambient changes, such as changes in viewpoint or lighting, that do not contribute to face recognition. However, there is a narrow band of image dimensions for shape and texture that are critical for the recognition of identity in humans and computer models of face recognition.

Spikiness and animacy as potential organizing principles of human ventral visual cortex

Considerable research has been devoted to understanding the fundamental organizing principles of the ventral visual pathway. A recent study revealed a series of 3-4 topographical maps arranged along the macaque inferotemporal (IT) cortex. The maps articulated a two-dimensional space based on the spikiness and animacy of visual objects, with "inanimate-spiky" and "inanimate-stubby" regions of the maps constituting two previously unidentified cortical networks. The goal of our study was to determine whether a similar functional organization might exist in human IT. To address this question, we presented the same object stimuli and images from "classic" object categories (bodies, faces, houses) to humans while recording fMRI activity at 7 Tesla. Contrasts designed to reveal the spikiness-animacy object space evoked extensive significant activation across human IT. However, unlike the macaque, we did not observe a clear sequence of complete maps, and selectivity for the spikiness-animacy space was deeply and mutually entangled with category-selectivity. Instead, we observed multiple new stimulus preferences in category-selective regions, including functional sub-structure related to object spikiness in scene-selective cortex. Taken together, these findings highlight spikiness as a promising organizing principle of human IT and provide new insights into the role of category-selective regions in visual object processing.

Using image reconstruction to investigate face perception in amnesia

Damage to the medial temporal lobe (MTL), which is traditionally considered to subserve memory exclusively, has been reported to contribute to impaired face perception. However, it remains unknown how exactly such brain lesions may impact face representations and in particular facial shape and surface information, both of which are crucial for face perception. The present study employed a behavioral-based image reconstruction approach to reveal the pictorial representations of face perception in two amnesic patients: DA, who has an extensive bilateral MTL lesion that extends beyond the MTL in the right hemisphere, and BL, who has damage to the hippocampal dentate gyrus (DG). Both patients and their respective matched controls completed similarity judgments for pairs of faces, from which facial shape and surface features were subsequently derived and synthesized to create images of reconstructed facial appearance. Participants also completed a face oddity judgment task (FOJT) that has previously been shown to be sensitive to MTL cortical damage. While BL exhibited an impaired pattern of performance on the FOJT, DA demonstrated intact performance accuracy. Notably, the recovered pictorial content of faces was comparable between both patients and controls, although there was evidence for atypical face representations in BL particularly with regards to color. Our work provides novel insight into the face representations underlying face perception in two well-studied amnesic patients in the literature and demonstrates the applicability of the image reconstruction approach to individuals with brain damage.

The representation of shape and texture in category-selective regions of ventral-temporal cortex

Neuroimaging studies using univariate and multivariate approaches have shown that the fusiform face area (FFA) and parahippocampal place area (PPA) respond selectively to images of faces and places. The aim of this study was to determine the extent to which this selectivity to faces or places is based on the shape or texture properties of the images. Faces and houses were filtered to manipulate their texture properties, while preserving the shape properties (spatial envelope) of the images. In Experiment 1, multivariate pattern analysis (MVPA) showed that patterns of fMRI response to faces and houses in FFA and PPA were predicted by the shape properties, but not by the texture properties of the image. In Experiment 2, a univariate analysis (fMR‐adaptation) showed that responses in the FFA and PPA were sensitive to changes in both the shape and texture properties of the image. These findings can be explained by the spatial scale of the representation of images in the FFA and PPA. At a coarser scale (revealed by MVPA), the neural selectivity to faces and houses is sensitive to variation in the shape properties of the image. However, at a finer scale (revealed by fMR‐adaptation), the neural selectivity is sensitive to the texture properties of the image. By combining these neuroimaging paradigms, our results provide insights into the spatial scale of the neural representation of faces and places in the ventral‐temporal cortex.

Face processing in the temporal lobe

Face perception is a socially important but complex process with many stages and many facets. There is substantial evidence from many sources that it involves a large extent of the temporal lobe, from the ventral occipitotemporal cortex and superior temporal sulci to anterior temporal regions. While early human neuroimaging work suggested a core face network consisting of the occipital face area, fusiform face area, and posterior superior temporal sulcus, studies in both humans and monkeys show a system of face patches stretching from posterior to anterior in both the superior temporal sulcus and inferotemporal cortex. Sophisticated techniques such as fMRI adaptation have shown that these face-activated regions show responses that have many of the attributes of human face processing. Lesions of some of these regions in humans lead to variants of prosopagnosia, the inability to recognize the identity of a face. Lesion, imaging, and electrophysiologic data all suggest that there is a segregation between identity and expression processing, though some suggest this may be better characterized as a distinction between static and dynamic facial information.

The role of stimulus-based cues and conceptual information in processing facial expressions of emotion

Face shape and surface textures are two important cues that aid in the perception of facial expressions of emotion. Additionally, this perception is also influenced by high-level emotion concepts. Across two studies, we use representational similarity analysis to investigate the relative roles of shape, surface, and conceptual information in the perception, categorisation, and neural representation of facial expressions. In Study 1, 50 participants completed a perceptual task designed to measure the perceptual similarity of expression pairs, and a categorical task designed to measure the confusability between expression pairs when assigning emotion labels to a face. We used representational similarity analysis and constructed three models of the similarities between emotions using distinct information. Two models were based on stimulus-based cues (face shapes and surface textures) and one model was based on emotion concepts. Using multiple linear regression, we found that behaviour during both tasks was related with the similarity of emotion concepts. The model based on face shapes was more related with behaviour in the perceptual task than in the categorical, and the model based on surface textures was more related with behaviour in the categorical than the perceptual task. In Study 2, 30 participants viewed facial expressions while undergoing fMRI, allowing for the measurement of brain representational geometries of facial expressions of emotion in three core face-responsive regions (the Fusiform Face Area, Occipital Face Area, and Superior Temporal Sulcus), and a region involved in theory of mind (Medial Prefrontal Cortex). Across all four regions, the representational distances between facial expression pairs were related to the similarities of emotion concepts, but not to either of the stimulus-based cues. Together, these results highlight the important top-down influence of high-level emotion concepts both in behavioural tasks and in the neural representation of facial expressions.

Similar use of shape and texture cues for own- and other-race faces during face learning and recognition

Although the other-race effect (ORE; superior recognition of own- relative to other-race faces) is well established, the mechanisms underlying it are not well understood. We examined whether the ORE is attributable to differential use of shape and texture cues for own- vs. other-race faces. Shape cues are particularly important for detecting that an own-race face is unfamiliar, whereas texture cues are more important for recognizing familiar and newly learned own-race faces. We compared the influence of shape and texture cues on Caucasian participants' recognition of Caucasian and East Asian faces using two complementary approaches. In Experiment 1, participants studied veridical, shape-caricatured, or texture-caricatured faces and then were asked to recognize them in an old/new recognition task. In Experiment 2, all study faces were veridical and we independently removed the diagnosticity of shape (or texture) cues in the test phase by replacing original shape (or texture) with average shape (or texture). Despite an overall own-race advantage, participants' use of shape and texture cues was comparable for own- and other-race faces. These results suggest that the other-race effect is not attributable to qualitative differences in the use of shape and texture cues.

Analysing Texture in Portraits

This case study is an initial exploration as to whether the depiction of texture in a set of portraits, all portraying the same Sitter, is related to the familiar likeness assessments reported in a companion paper containing a principal component analysis (PCA) of the portraits' depiction of shape. Somewhat unexpectedly, a texture PCA failed to discriminate the high from low likeness portraits, despite experimentation with different pre-processing methods to reduce the portraits' high level of uninformative, image-level texture variability. There were some findings arising from these analyses, and while only indicative at this stage, include that linear histogram matching is effective in reducing variability in portrait brightness; that depicting, and perhaps exaggerating, shading relating to lighting direction may enhance portrait likeness; and, that whether minimised or exaggerated, lighting direction can be portrayed somewhat anomalously. Furthermore, and in agreement with findings from photographs, shape and texture were not found to be independent variables, and shape-free image registration, while very usefully enabling a comparison of closely corresponding pixel coordinate values, could itself be a confounding factor for undertaking a texture PCA with portraits produced under relatively ambient conditions.

Different and common brain signals of altered neurocognitive mechanisms for unfamiliar face processing in acquired and developmental prosopagnosia

Neuropsychological studies have shown that prosopagnosic individuals perceive face structure in an atypical way. This might preclude the formation of appropriate face representations and, consequently, hamper effective recognition. The present ERP study, in combination with Bayesian source reconstruction, investigates how information related to both external (E) and internal (I) features was processed by E.C. and I.P., suffering from acquired and developmental prosopagnosia, respectively. They carried out a face-feature matching task with new faces. E.C. showed poor performance and remarkable lack of early face-sensitive P1, N170 and P2 responses on right (damaged) posterior cortex. Although she presented the expected mismatch effect to target faces in the E-I sequence, it was of shorter duration than in Controls, and involved left parietal, right frontocentral and dorsofrontal regions, suggestive of reduced neural circuitry to process face configurations. In turn, I.P. performed efficiently but with a remarkable bias to give "match" responses. His face-sensitive potentials P1-N170 were comparable to those from Controls, however, he showed no subsequent P2 response and a mismatch effect only in the I-E sequence, reflecting activation confined to those regions that sustain typically the initial stages of face processing. Relevantly, neither of the prosopagnosics exhibited conspicuous P3 responses to features acting as primes, indicating that diagnostic information for constructing face representations could not be sufficiently attended nor deeply encoded. Our findings suggest a different locus for altered neurocognitive mechanisms in the face network in participants with different types of prosopagnosia, but common indicators of a deficient allocation of attentional resources for further recognition.

The Face of Image Reconstruction: Progress, Pitfalls, Prospects

Recent research has demonstrated that neural and behavioral data acquired in response to viewing face images can be used to reconstruct the images themselves. However, the theoretical implications, promises, and challenges of this direction of research remain unclear. We evaluate the potential of this research for elucidating the visual representations underlying face recognition. Specifically, we outline complementary and converging accounts of the visual content, the representational structure, and the neural dynamics of face processing. We illustrate how this research addresses fundamental questions in the study of normal and impaired face recognition, and how image reconstruction provides a powerful framework for uncovering face representations, for unifying multiple types of empirical data, and for facilitating both theoretical and methodological progress.

Contributions of shape and reflectance information to social judgments from faces

Face perception is based on both shape and reflectance information. However, we know little about the relative contribution of these kinds of information to social judgments of faces. In Experiment 1, we generated faces using validated computational models of attractiveness, competence, dominance, extroversion, and trustworthiness. Faces were manipulated orthogonally on five levels of shape and reflectance for each model. Both kinds of information had linear and additive effects on participants' social judgments. Shape information was more predictive of dominance, extroversion, and trustworthiness judgments, whereas reflectance information was more predictive of competence judgments. In Experiment 2, to test whether the amount of visual information alters the relative contribution of shape and reflectance information, we presented faces - varied on attractiveness, competence, and dominance - for five different durations (33-500 ms). For all judgments, the linear effect of both shape and reflectance increased as duration increased. Importantly, the relative contribution did not change across durations. These findings show that that the judged dimension is critical for which kind of information is weighted more heavily in judgments and that the relative contribution of shape and reflectance is stable across the amount of visual information available.

Selectivity for mid-level properties of faces and places in the fusiform face area and parahippocampal place area

Regions in the ventral visual pathway, such as the fusiform face area (FFA) and parahippocampal place area (PPA) are selective for images from specific object categories. Yet images from different object categories differ in their image properties. To investigate how these image properties are represented in the FFA and PPA, we compared neural responses to locally-SCRAMBLED images (in which mid-level, spatial properties are preserved) and globally-SCRAMBLED images (in which mid-level, spatial properties are not preserved). There was a greater response in the FFA and PPA to images from the preferred CATEGORY relative to their non-preferred category for the scrambled conditions. However, there was a greater selectivity for locally-scrambled compared to globally-scrambled images. Next, we compared the magnitude of fMR-adaptation to intact and scrambled images. fMR-adaptation was evident to locally-scrambled images from the preferred category. However, there was no adaptation to globally-scrambled images from the preferred category. These results show that the selectivity to faces and places in the FFA and PPA is dependent on mid-level properties of the image that are preserved by local-scrambling.

TMS over right OFA affects individuation of faces but not of exemplars of objects

In addition to its well-documented role in processing of faces, the occipital face area in the right hemisphere (rOFA) may also play a role in identifying specific individuals within a class of objects. Here we explored this issue by using fMRI-guided TMS. In a first experiment, participants had to judge whether two sequentially presented images of faces or objects represented exactly the same exemplar or two different exemplars of the same class, while receiving online TMS over either the rOFA, the right lateral occipital cortex (rLO) or the Vertex (control). We found that, relative to Vertex, stimulation of rOFA impaired individuation of faces only, with no effect on objects; in contrast, TMS over rLO reduced individuation of objects but not of faces. In a second control experiment participants judged whether a picture representing a fragment of a stimulus belonged or not to the subsequently presented image of a whole stimulus (part-whole matching task). Our results showed that rOFA stimulation selectively disrupted performance with faces, whereas performance with objects (but not with faces) was selectively affected by TMS over rLO. Overall, our findings suggest that rOFA does not contribute to discriminate between exemplars of non-face objects.

Patterns of neural response in face regions are predicted by low-level image properties

Models of face processing suggest that the neural response in different face regions is selective for higher-level attributes of the face, such as identity and expression. However, it remains unclear to what extent the response in these regions can also be explained by more basic organizing principles. Here, we used functional magnetic resonance imaging multivariate pattern analysis (fMRI-MVPA) to ask whether spatial patterns of response in the core face regions (occipital face area - OFA, fusiform face area - FFA, superior temporal sulcus - STS) can be predicted across different participants by lower level properties of the stimulus. First, we compared the neural response to face identity and viewpoint, by showing images of different identities from different viewpoints. The patterns of neural response in the core face regions were predicted by the viewpoint, but not the identity of the face. Next, we compared the neural response to viewpoint and expression, by showing images with different expressions from different viewpoints. Again, viewpoint, but not expression, predicted patterns of response in face regions. Finally, we show that the effect of viewpoint in both experiments could be explained by changes in low-level image properties. Our results suggest that a key determinant of the neural representation in these core face regions involves lower-level image properties rather than an explicit representation of higher-level attributes in the face. The advantage of a relatively image-based representation is that it can be used flexibly in the perception of faces.

Faces, people and the brain: The 45th Sir Frederic Bartlett Lecture

The fact that the face is a source of diverse social signals allows us to use face and person perception as a model system for asking important psychological questions about how our brains are organised. A key issue concerns whether we rely primarily on some form of generic representation of the common physical source of these social signals (the face) to interpret them, or instead create multiple representations by assigning different aspects of the task to different specialist components. Variants of the specialist components hypothesis have formed the dominant theoretical perspective on face perception for more than three decades, but despite this dominance of formally and informally expressed theories, the underlying principles and extent of any division of labour remain uncertain. Here, I discuss three important sources of constraint: first, the evolved structure of the brain; second, the need to optimise responses to different everyday tasks; and third, the statistical structure of faces in the perceiver's environment. I show how these constraints interact to determine the underlying functional organisation of face and person perception.

The Neural Dynamics of Facial Identity Processing: Insights from EEG-Based Pattern Analysis and Image Reconstruction

Uncovering the neural dynamics of facial identity processing along with its representational basis outlines a major endeavor in the study of visual processing. To this end, here, we record human electroencephalography (EEG) data associated with viewing face stimuli; then, we exploit spatiotemporal EEG information to determine the neural correlates of facial identity representations and to reconstruct the appearance of the corresponding stimuli. Our findings indicate that multiple temporal intervals support: facial identity classification, face space estimation, visual feature extraction and image reconstruction. In particular, we note that both classification and reconstruction accuracy peak in the proximity of the N170 component. Further, aggregate data from a larger interval (50–650 ms after stimulus onset) support robust reconstruction results, consistent with the availability of distinct visual information over time. Thus, theoretically, our findings shed light on the time course of face processing while, methodologically, they demonstrate the feasibility of EEG-based image reconstruction.