How face perception unfolds over time

From encoding to recognition: Exploring the shared neural signatures of visual memory

This study investigated the shared neural dynamics underlying encoding and recognition processes across diverse visual object stimulus types in short term experimental familiarization, using EEG-based representational similarity analysis and multivariate cross-classification. Building upon previous research, we extended our exploration to the encoding phase. We show early visual stimulus category effects around 150 ms post-stimulus onset and old/new effects around 400 to 600 ms. Notably, a divergence in neural responses for encoding, old, and new stimuli emerged around 300 ms, with items encountered during the study phase showing the highest differentiation from old items during the test phase. Cross-category classification demonstrated discernible memory-related effects as early as 150 ms. Anterior regions of interest, particularly in the right hemisphere, did not exhibit differentiation between experimental phases or between study and new items, hinting at similar processing for items first encountered, irrespective of experiment phase. While short-term experimental familiarity did not consistently adhere to the old >new pattern observed in long-term personal familiarity, statistically significant effects are observed specifically for experimentally familiarized faces, suggesting a potential unique phenomenon specific to facial stimuli. Further investigation is warranted to elucidate underlying mechanisms and determine the extent of face-specific effects. Lastly, our findings underscore the utility of multivariate cross-classification and cross-dataset classification as promising tools for probing abstraction and shared neural signatures of cognitive processing.

The state of modelling face processing in humans with deep learning

Deep learning models trained for facial recognition now surpass the highest performing human participants. Recent evidence suggests that they also model some qualitative aspects of face processing in humans. This review compares the current understanding of deep learning models with psychological models of the face processing system. Psychological models consist of two components that operate on the information encoded when people perceive a face, which we refer to here as 'face codes'. The first component, the core system, extracts face codes from retinal input that encode invariant and changeable properties. The second component, the extended system, links face codes to personal information about a person and their social context. Studies of face codes in existing deep learning models reveal some surprising results. For example, face codes in networks designed for identity recognition also encode expression information, which contrasts with psychological models that separate invariant and changeable properties. Deep learning can also be used to implement candidate models of the face processing system, for example to compare alternative cognitive architectures and codes that might support interchange between core and extended face processing systems. We conclude by summarizing seven key lessons from this research and outlining three open questions for future study.

How Linguistic and Nonlinguistic Vocalizations Shape the Perception of Emotional Faces-An Electroencephalography Study

Vocal emotions are crucial in guiding visual attention toward emotionally significant environmental events, such as recognizing emotional faces. This study employed continuous EEG recordings to examine the impact of linguistic and nonlinguistic vocalizations on facial emotion processing. Participants completed a facial emotion discrimination task while viewing fearful, happy, and neutral faces. The behavioral and ERP results indicated that fearful nonlinguistic vocalizations accelerated the recognition of fearful faces and elicited a larger P1 amplitude, whereas happy linguistic vocalizations accelerated the recognition of happy faces and similarly induced a greater P1 amplitude. In recognition of fearful faces, a greater N170 component was observed in the right hemisphere when the emotional category of the priming vocalization was consistent with the face stimulus. In contrast, this effect occurred in the left hemisphere while recognizing happy faces. Representational similarity analysis revealed that the temporoparietal regions automatically differentiate between linguistic and nonlinguistic vocalizations early in face processing. In conclusion, these findings enhance our understanding of the interplay between vocalization types and facial emotion recognition, highlighting the importance of cross-modal processing in emotional perception.

Distinguishing the roles of edge, color, and other surface information in basic and superordinate scene representation

The human brain possesses a remarkable ability to recognize scenes depicted in line drawings, despite that these drawings contain only edge information. It remains unclear how the brain uses this information alongside surface information in scene recognition. Here, we combined electroencephalogram (EEG) and multivariate pattern analysis (MVPA) methods to distinguish the roles of edge, color, and other surface information in scene representation at the basic category level and superordinate naturalness level over time. The time-resolved decoding results indicated that edge information in line drawings is both sufficient and more effective than in color photographs and grayscale images at the superordinate naturalness level. Meanwhile, color and other surface information are exclusively involved in neural representation at the basic category level. The time generalization analysis further revealed that edge information is crucial for representation at both levels of abstraction. These findings highlight the distinct roles of edge, color, and other surface information in dynamic neural scene processing, shedding light on how the human brain represents scene information at different levels of abstraction.

Core dimensions of human material perception

Visually categorizing and comparing materials is crucial for everyday behavior, but what organizational principles underlie our mental representation of materials? Here, we used a large-scale data-driven approach to uncover core latent dimensions of material representations from behavior. First, we created an image dataset of 200 systematically sampled materials and 600 photographs (STUFF dataset, https://osf.io/myutc/). Using these images, we next collected 1.87 million triplet similarity judgments and used a computational model to derive a set of sparse, positive dimensions underlying these judgments. The resulting multidimensional embedding space predicted independent material similarity judgments and the similarity matrix of all images close to the human intersubject consistency. We found that representations of individual images were captured by a combination of 36 material dimensions that were highly reproducible and interpretable, comprising perceptual (e.g., grainy, blue) as well as conceptual (e.g., mineral, viscous) dimensions. These results provide the foundation for a comprehensive understanding of how humans make sense of materials.

Revealing the neural representations underlying other-race face perception

The other-race effect (ORE) refers to poorer recognition for faces of other races than one's own. This study investigates the neural and representational basis of ORE in East Asian and White participants using behavioral measures, neural decoding, and image reconstruction based on electroencephalography (EEG) data. Our investigation identifies a reliable neural counterpart of ORE, with reduced decoding accuracy for other-race faces, and it relates this result to higher density of other-race face representations in face space. Then, we characterize the temporal dynamics and the prominence of ORE for individual variability at the neural level. Importantly, we use a data-driven image reconstruction approach to reveal visual biases underlying other-race face perception, including a tendency to perceive other-race faces as more typical, younger, and more expressive. These findings provide neural evidence for a classical account of ORE invoking face space compression for other-race faces. Further, they indicate that ORE involves not only reduced identity information but also broader, systematic distortions in visual representation with considerable cognitive and social implications.

Differential lateralization to faces in infants at risk of autism spectrum disorder with expressive language delay

Aim

Face-to-face communication between caregiver and infant is essential for the development of language and social skills in infancy. A previous study on brain response toward human faces showed that a lateralization right fusiform gyrus (FG) response when viewing faces was associated with better social skills. However, the relationship, between infant face processing and language development remains unclear. This study aimed to examine whether brain responses to faces vary based on the ability level of language expression.

Methods

Overall, 42 Japanese infants (aged 18-34 months, Mean of age (Mage) = 24.7 months, standard deviation (SD) = 4.57, 47% female) were assessed for expressive communication skills and classified into two groups: a delayed group (20 infants) and a control group (infants with typical expressive language development, 22 infants). Brain activity was recorded using a child-customized magnetoencephalography during presentation of a mother's face, a stranger's face, and a nonface (scrambled image). The lateralization index of the FG during face viewing was calculated using the following formula: (L - R)/(L + R).

Results

The results showed a significant difference in the lateralization index between the delayed and control groups. The control group showed rightward dominance of the FG activity when viewing the mother's face and others' faces, whereas the delayed group did not exhibit this lateralization. Based on behavioral observations, 75% of the delayed group met the criteria of autism spectrum disorder (ASD) risk, and infants with a high risk of ASD who had poor expressive language showed poor right hemispheric dominance compared to the control group in their brain responses to their mothers' faces.

Conclusion

This study suggests that lateralization of face processing in infancy may be a predictor of expressive language abilities.

Decoding the Spatiotemporal Dynamics of Neural Response Similarity in Auditory Processing: A Multivariate Analysis Based on OPM-MEG

The brain represents information through the encoding of neural populations, where the activity patterns of these neural groups constitute the content of this information. Understanding these activity patterns and their dynamic changes is of significant importance to cognitive neuroscience and related research areas. Current studies focus more on brain regions that show differential responses to stimuli, but they lack the ability to capture information about the representational or process-level dynamics within these regions. In this study, we recorded neural data from 10 healthy participants during auditory experiments using optically pumped magnetometer magnetoencephalography (OPM-MEG) and electroencephalography (EEG). We constructed representational similarity matrices (RSMs) to investigate the similarity of neural response patterns during auditory decoding. The results indicate that RSA can reveal the dynamic changes in pattern similarity during different stages of auditory processing through the neural activity patterns reflected by OPM-MEG. Comparisons with EEG results showed that both techniques captured the same processes during the early stages of auditory decoding. However, differences in sensitivity at later stages highlighted both common and distinct aspects of neural representation between the two modalities. Further analysis indicated that this process involved widespread neural network activation, including the Heschl's gyrus, superior temporal gyrus, middle temporal gyrus, inferior temporal gyrus, parahippocampal gyrus, and orbitofrontal gyrus. This study demonstrates that the combination of OPM-MEG and RSA is sufficiently sensitive to detect changes in pattern similarity during neural representation processes and to identify their anatomical origins, offering new insights and references for the future application of RSA and other multivariate pattern analysis methods in the MEG field.

Time Course of Orientation Ensemble Representation in the Human Brain

Natural scenes are filled with groups of similar items. Humans employ ensemble coding to extract the summary statistical information of the environment, thereby enhancing the efficiency of information processing, something particularly useful when observing natural scenes. However, the neural mechanisms underlying the representation of ensemble information in the brain remain elusive. In particular, whether ensemble representation results from the mere summation of individual item representations or it engages other specific processes remains unclear. In this study, we utilized a set of orientation ensembles wherein none of the individual item orientations were the same as the ensemble orientation. We recorded magnetoencephalography (MEG) signals from human participants (both sexes) when they performed an ensemble orientation discrimination task. Time-resolved multivariate pattern analysis (MVPA) and the inverted encoding model (IEM) were employed to unravel the neural mechanisms of the ensemble orientation representation and track its time course. First, we achieved successful decoding of the ensemble orientation, with a high correlation between the decoding and behavioral accuracies. Second, the IEM analysis demonstrated that the representation of the ensemble orientation differed from the sum of the representations of individual item orientations, suggesting that ensemble coding could further modulate orientation representation in the brain. Moreover, using source reconstruction, we showed that the representation of ensemble orientation manifested in early visual areas. Taken together, our findings reveal the emergence of the ensemble representation in the human visual cortex and advance the understanding of how the brain captures and represents ensemble information.

Human-like face pareidolia emerges in deep neural networks optimized for face and object recognition

The human visual system possesses a remarkable ability to detect and process faces across diverse contexts, including the phenomenon of face pareidolia--seeing faces in inanimate objects. Despite extensive research, it remains unclear why the visual system employs such broadly tuned face detection capabilities. We hypothesized that face pareidolia results from the visual system's optimization for recognizing both faces and objects. To test this hypothesis, we used task-optimized deep convolutional neural networks (CNNs) and evaluated their alignment with human behavioral signatures and neural responses, measured via magnetoencephalography (MEG), related to pareidolia processing. Specifically, we trained CNNs on tasks involving combinations of face identification, face detection, object categorization, and object detection. Using representational similarity analysis, we found that CNNs that included object categorization in their training tasks represented pareidolia faces, real faces, and matched objects more similarly to neural responses than those that did not. Although these CNNs showed similar overall alignment with neural data, a closer examination of their internal representations revealed that specific training tasks had distinct effects on how pareidolia faces were represented across layers. Finally, interpretability methods revealed that only a CNN trained for both face identification and object categorization relied on face-like features-such as 'eyes'-to classify pareidolia stimuli as faces, mirroring findings in human perception. Our results suggest that human-like face pareidolia may emerge from the visual system's optimization for face identification within the context of generalized object categorization.

The neural dynamics of familiar face recognition

Humans are highly efficient at recognising familiar faces. However, previous EEG/ERP research has given a partial and fragmented account of the neural basis of this remarkable ability. We argue that this is related to insufficient consideration of fundamental characteristics of familiar face recognition. These include image-independence (recognition across different pictures), levels of familiarity (familiar faces vary hugely in duration and intensity of our exposure to them), automaticity (we cannot voluntarily withhold from recognising a familiar face), and domain-selectivity (the degree to which face familiarity effects are selective). We review recent EEG/ERP work, combining uni- and multivariate methods, that has systematically targeted these shortcomings. We present a theoretical account of familiar face recognition, dividing it into early visual, domain-sensitive and domain-general phases, and integrating image-independence and levels of familiarity. Our account incorporates classic and more recent concepts, such as multi-dimensional face representation and course-to-fine processing. While several questions remain to be addressed, this new account represents a major step forward in our understanding of the neurophysiological basis of familiar face recognition.

Decoding the Temporal Structures and Interactions of Multiple Face Dimensions Using Optically Pumped Magnetometer Magnetoencephalography (OPM-MEG)

Humans possess a remarkable ability to rapidly access diverse information from others' faces with just a brief glance, which is crucial for intricate social interactions. While previous studies using event-related potentials/fields have explored various face dimensions during this process, the interplay between these dimensions remains unclear. Here, by applying multivariate decoding analysis to neural signals recorded with optically pumped magnetometer magnetoencephalography, we systematically investigated the temporal interactions between invariant and variable aspects of face stimuli, including race, gender, age, and expression. First, our analysis revealed unique temporal structures for each face dimension with high test-retest reliability. Notably, expression and race exhibited a dominant and stably maintained temporal structure according to temporal generalization analysis. Further exploration into the mutual interactions among face dimensions uncovered age effects on gender and race, as well as expression effects on race, during the early stage (∼200-300 ms postface presentation). Additionally, we observed a relatively late effect of race on gender representation, peaking ∼350 ms after the stimulus onset. Taken together, our findings provide novel insights into the neural dynamics underlying the multidimensional aspects of face perception and illuminate the promising future of utilizing OPM-MEG for exploring higher-level human cognition.

Expressions of emotions in minimal face perception stimuli

Face perception is considered to be a canonical example of configurational visual processing. However, not all facial information is equally important when reading facial expressions. The eyes and mouth seem to be crucial, but they seem to have different roles and significance. By varying the shape of the mouth, eyes, and other factors, we conducted two experiments: first, we examined eye movements depending on different facial configurations and different types of instructions (neutral and emotionally valenced); second, we used the same types of stimuli in a rating task. Our results indicate that the eyes provide a primary impact (when eye fixations are measured), which can be explained by the evolutionary need to establish gaze contact, but once facial expressions are observed, the mouth seems to be more significant.

What will we judge a book by its cover?-Content analysis of face perception in a Chinese sample

People can perceive various information from faces. Most of previous studies of face perception only focused on one of attributes, such as gender, expression, personality etc., the whole picture of face perception is far from clear. Therefore, the present study recruited Chinese participants to provide spontaneous descriptions of unfamiliar Chinese faces without any constraints of content. It turned out that descriptions employed a broad spectrum of descriptors, as well as a consistent pattern across different identities: descriptions that incorporated psychological characteristics were most prevalent, whereas mentions of physiological attributes generally occurred earlier in the description than other types of descriptive vocabulary. These results underscore the special role of free description analysis in revealing the panorama of face perception, where perceivers swiftly infer a plenty of character traits in an organized way, ultimately forming a comprehensive impression of others.

Touch to text: Spatiotemporal evolution of braille letter representations in blind readers

Visual deprivation does not silence the visual cortex, which is responsive to auditory, tactile, and other nonvisual tasks in blind persons. However, the underlying functional dynamics of the neural networks mediating such crossmodal responses remain unclear. Here, using braille reading as a model framework to investigate these networks, we presented sighted (N=13) and blind (N=12) readers with individual visual print and tactile braille alphabetic letters, respectively, during MEG recording. Using time-resolved multivariate pattern analysis and representational similarity analysis, we traced the alphabetic letter processing cascade in both groups of participants. We found that letter representations unfolded more slowly in blind than in sighted brains, with decoding peak latencies ~200 ms later in braille readers. Focusing on the blind group, we found that the format of neural letter representations transformed within the first 500 ms after stimulus onset from a low-level structure consistent with peripheral nerve afferent coding to high-level format reflecting pairwise letter embeddings in a text corpus. The spatiotemporal dynamics of the transformation suggest that the processing cascade proceeds from a starting point in somatosensory cortex to early visual cortex and then to inferotemporal cortex. Together our results give insight into the neural mechanisms underlying braille reading in blind persons and the dynamics of functional reorganization in sensory deprivation.

The temporal dynamics of familiar face recognition: Event-related brain potentials reveal the efficient activation of facial identity representations

While it is widely known that humans are typically highly accurate at recognizing familiar faces, it is less clear how efficiently recognition is achieved. In a series of three experiments, we used event-related brain potentials (ERP) in a repetition priming paradigm to examine the efficiency of familiar face recognition. Specifically, we varied the presentation time of the prime stimulus between 500 ms and 33 ms (Experiments 1 and 2), and additionally used backward masks (Experiment 3) to prevent the potential occurrence of visual aftereffects. Crucially, to test for the recognition of facial identity rather than a specific picture, we used different images of the same facial identities in repetition conditions. We observed clear ERP repetition priming effects between 300 and 500 ms after target onset at all prime durations, which suggests that the prime stimulus was sufficiently well processed to allow for facilitated recognition of the target in all conditions. This finding held true even in severely restricted viewing conditions including very brief prime durations and backward masks. We conclude that the facial recognition system is both highly effective and efficient, thus allowing for our impressive ability to recognise the faces that we know.

Specified functions of the first two fixations in face recognition: Sampling the general-to-specific facial information

Visual perception is enacted and constrained by the constantly moving eyes. Although it is well known that the first two fixations are crucial for face recognition, the function of each fixation remains unspecified. Here we demonstrate a central-to-divergent pattern of the two fixations and specify their functions: Fix I clustered along the nose bridge to cover the broad facial information; Fix II diverged to eyes, nostrils, and lips to get the local information. Fix II correlated more than Fix I with the differentiating information between faces and contributed more to recognition responses. While face categories can be significantly discriminated by Fix II's but not Fix I's patterns alone, the combined patterns of the two yield better discrimination. Our results suggest a functional division and collaboration of the two fixations in sampling the general-to-specific facial information and add to understanding visual perception as an active process undertaken by structural motor programs.

Reconstructing Spatiotemporal Trajectories of Visual Object Memories in the Human Brain

How the human brain reconstructs, step-by-step, the core elements of past experiences is still unclear. Here, we map the spatiotemporal trajectories along which visual object memories are reconstructed during associative recall. Specifically, we inquire whether retrieval reinstates feature representations in a copy-like but reversed direction with respect to the initial perceptual experience, or alternatively, this reconstruction involves format transformations and regions beyond initial perception. Participants from two cohorts studied new associations between verbs and randomly paired object images, and subsequently recalled the objects when presented with the corresponding verb cue. We first analyze multivariate fMRI patterns to map where in the brain high- and low-level object features can be decoded during perception and retrieval, showing that retrieval is dominated by conceptual features, represented in comparatively late visual and parietal areas. A separately acquired EEG dataset is then used to track the temporal evolution of the reactivated patterns using similarity-based EEG-fMRI fusion. This fusion suggests that memory reconstruction proceeds from anterior frontotemporal to posterior occipital and parietal regions, in line with a conceptual-to-perceptual gradient but only partly following the same trajectories as during perception. Specifically, a linear regression statistically confirms that the sequential activation of ventral visual stream regions is reversed between image perception and retrieval. The fusion analysis also suggests an information relay to frontoparietal areas late during retrieval. Together, the results shed light onto the temporal dynamics of memory recall and the transformations that the information undergoes between the initial experience and its later reconstruction from memory.

Establishing the relationship between subjective perception and neural responses: Insights from correlation analysis and representational similarity analysis

Exploring the relationship between sensory perception and brain responses holds important theoretical and clinical implications. However, commonly used methodologies like correlation analysis performed either intra- or inter- individually often yield inconsistent results across studies, limiting their generalizability. Representational similarity analysis (RSA), a method that assesses the perception-response relationship by calculating the correlation between behavioral and neural patterns, may offer a fresh perspective to reveal novel findings. Here, we delivered a series of graded sensory stimuli of four modalities (i.e., nociceptive somatosensory, non-nociceptive somatosensory, visual, and auditory) to/near the left or right hand of 107 healthy subjects and collected their single-trial perceptual ratings and electroencephalographic (EEG) responses. We examined the relationship between sensory perception and brain responses using within- and between-subject correlation analysis and RSA, and assessed their stability across different numbers of subjects and trials. We found that within-subject and between-subject correlations yielded distinct results: within-subject correlation revealed strong and reliable correlations between perceptual ratings and most brain responses, while between-subject correlation showed weak correlations that were vulnerable to the change of subject number. In addition to verifying the correlation results, RSA revealed some novel findings, i.e., correlations between behavioral and neural patterns were observed in some additional neural responses, such as "γ-ERS" in the visual modality. RSA results were sensitive to the trial number, but not to the subject number, suggesting that consistent results could be obtained for studies with relatively small sample sizes. In conclusion, our study provides a novel perspective on establishing the relationship between behavior and brain activity, emphasizing that RSA holds promise as a method for exploring this pattern relationship in future research.

Object recognition in primates: what can early visual areas contribute?

Introduction

If neuroscientists were asked which brain area is responsible for object recognition in primates, most would probably answer infero-temporal (IT) cortex. While IT is likely responsible for fine discriminations, and it is accordingly dominated by foveal visual inputs, there is more to object recognition than fine discrimination. Importantly, foveation of an object of interest usually requires recognizing, with reasonable confidence, its presence in the periphery. Arguably, IT plays a secondary role in such peripheral recognition, and other visual areas might instead be more critical.

Methods

To investigate how signals carried by early visual processing areas (such as LGN and V1) could be used for object recognition in the periphery, we focused here on the task of distinguishing faces from non-faces. We tested how sensitive various models were to nuisance parameters, such as changes in scale and orientation of the image, and the type of image background.

Results

We found that a model of V1 simple or complex cells could provide quite reliable information, resulting in performance better than 80% in realistic scenarios. An LGN model performed considerably worse.

Discussion

Because peripheral recognition is both crucial to enable fine recognition (by bringing an object of interest on the fovea), and probably sufficient to account for a considerable fraction of our daily recognition-guided behavior, we think that the current focus on area IT and foveal processing is too narrow. We propose that rather than a hierarchical system with IT-like properties as its primary aim, object recognition should be seen as a parallel process, with high-accuracy foveal modules operating in parallel with lower-accuracy and faster modules that can operate across the visual field.

Object recognition in primates: What can early visual areas contribute?

If neuroscientists were asked which brain area is responsible for object recognition in primates, most would probably answer infero-temporal (IT) cortex. While IT is likely responsible for fine discriminations, and it is accordingly dominated by foveal visual inputs, there is more to object recognition than fine discrimination. Importantly, foveation of an object of interest usually requires recognizing, with reasonable confidence, its presence in the periphery. Arguably, IT plays a secondary role in such peripheral recognition, and other visual areas might instead be more critical. To investigate how signals carried by early visual processing areas (such as LGN and V1) could be used for object recognition in the periphery, we focused here on the task of distinguishing faces from non-faces. We tested how sensitive various models were to nuisance parameters, such as changes in scale and orientation of the image, and the type of image background. We found that a model of V1 simple or complex cells could provide quite reliable information, resulting in performance better than 80% in realistic scenarios. An LGN model performed considerably worse. Because peripheral recognition is both crucial to enable fine recognition (by bringing an object of interest on the fovea), and probably sufficient to account for a considerable fraction of our daily recognition-guided behavior, we think that the current focus on area IT and foveal processing is too narrow. We propose that rather than a hierarchical system with IT-like properties as its primary aim, object recognition should be seen as a parallel process, with high-accuracy foveal modules operating in parallel with lower-accuracy and faster modules that can operate across the visual field.

The time course of person perception from voices in the brain

When listeners hear a voice, they rapidly form a complex first impression of who the person behind that voice might be. We characterize how these multivariate first impressions from voices emerge over time across different levels of abstraction using electroencephalography and representational similarity analysis. We find that for eight perceived physical (gender, age, and health), trait (attractiveness, dominance, and trustworthiness), and social characteristics (educatedness and professionalism), representations emerge early (~80 ms after stimulus onset), with voice acoustics contributing to those representations between ~100 ms and 400 ms. While impressions of person characteristics are highly correlated, we can find evidence for highly abstracted, independent representations of individual person characteristics. These abstracted representationse merge gradually over time. That is, representations of physical characteristics (age, gender) arise early (from ~120 ms), while representations of some trait and social characteristics emerge later (~360 ms onward). The findings align with recent theoretical models and shed light on the computations underpinning person perception from voices.

Movement trajectories as a window into the dynamics of emerging neural representations

The rapid transformation of sensory inputs into meaningful neural representations is critical to adaptive human behaviour. While non-invasive neuroimaging methods are the de-facto method for investigating neural representations, they remain expensive, not widely available, time-consuming, and restrictive. Here we show that movement trajectories can be used to measure emerging neural representations with fine temporal resolution. By combining online computer mouse-tracking and publicly available neuroimaging data via representational similarity analysis (RSA), we show that movement trajectories track the unfolding of stimulus- and category-wise neural representations along key dimensions of the human visual system. We demonstrate that time-resolved representational structures derived from movement trajectories overlap with those derived from M/EEG (albeit delayed) and those derived from fMRI in functionally-relevant brain areas. Our findings highlight the richness of movement trajectories and the power of the RSA framework to reveal and compare their information content, opening new avenues to better understand human perception.

Immigration documentation statuses evoke racialized faceism in mental representations

U.S. immigration discourse has spurred interest in characterizing who illegalized immigrants are or perceived to be. What are the associated visual representations of migrant illegality? Across two studies with undergraduate and online samples (N = 686), we used face-based reverse correlation and similarity sorting to capture and compare mental representations of illegalized immigrants, native-born U.S. citizens, and documented immigrants. Documentation statuses evoked racialized imagery. Immigrant representations were dark-skinned and perceived as non-white, while citizen representations were light-skinned, evaluated positively, and perceived as white. Legality further differentiated immigrant representations: documentation conjured trustworthy representations, illegality conjured threatening representations. Participants spontaneously sorted unlabeled faces by documentation status in a spatial arrangement task. Faces' spatial similarity correlated with their similarity in pixel luminance and "American" ratings, confirming racialized distinctions. Representations of illegalized immigrants were uniquely racialized as dark-skinned un-American threats, reflecting how U.S. imperialism and colorism set conditions of possibility for existing representations of migrant illegalization.

Neural correlates of retrospective memory confidence during face-name associative learning

The ability to accurately assess one's own memory performance during learning is essential for adaptive behavior, but the brain mechanisms underlying this metamemory function are not well understood. We investigated the neural correlates of memory accuracy and retrospective memory confidence in a face-name associative learning task using magnetoencephalography in healthy young adults (n = 32). We found that high retrospective confidence was associated with stronger occipital event-related fields during encoding and widespread event-related fields during retrieval compared to low confidence. On the other hand, memory accuracy was linked to medial temporal activities during both encoding and retrieval, but only in low-confidence trials. A decrease in oscillatory power at alpha/beta bands in the parietal regions during retrieval was associated with higher memory confidence. In addition, representational similarity analysis at the single-trial level revealed distributed but differentiable neural activities associated with memory accuracy and confidence during both encoding and retrieval. In summary, our study unveiled distinct neural activity patterns related to memory confidence and accuracy during associative learning and underscored the crucial role of parietal regions in metamemory.

Prediction error processing and sharpening of expected information across the face-processing hierarchy

The perception and neural processing of sensory information are strongly influenced by prior expectations. The integration of prior and sensory information can manifest through distinct underlying mechanisms: focusing on unexpected input, denoted as prediction error (PE) processing, or amplifying anticipated information via sharpened representation. In this study, we employed computational modeling using deep neural networks combined with representational similarity analyses of fMRI data to investigate these two processes during face perception. Participants were cued to see face images, some generated by morphing two faces, leading to ambiguity in face identity. We show that expected faces were identified faster and perception of ambiguous faces was shifted towards priors. Multivariate analyses uncovered evidence for PE processing across and beyond the face-processing hierarchy from the occipital face area (OFA), via the fusiform face area, to the anterior temporal lobe, and suggest sharpened representations in the OFA. Our findings support the proposition that the brain represents faces grounded in prior expectations.

Spatial representation of multidimensional information in emotional faces revealed by fMRI

Face perception is a complex process that involves highly specialized procedures and mechanisms. Investigating into face perception can help us better understand how the brain processes fine-grained, multidimensional information. This research aimed to delve deeply into how different dimensions of facial information are represented in specific brain regions or through inter-regional connections via an implicit face recognition task. To capture the representation of various facial information in the brain, we employed support vector machine decoding, functional connectivity, and model-based representational similarity analysis on fMRI data, resulting in the identification of three crucial findings. Firstly, despite the implicit nature of the task, emotions were still represented in the brain, contrasting with all other facial information. Secondly, the connection between the medial amygdala and the parahippocampal gyrus was found to be essential for the representation of facial emotion in implicit tasks. Thirdly, in implicit tasks, arousal representation occurred in the parahippocampal gyrus, while valence depended on the connection between the primary visual cortex and the parahippocampal gyrus. In conclusion, these findings dissociate the neural mechanisms of emotional valence and arousal, revealing the precise spatial patterns of multidimensional information processing in faces.

Common Sequential Organization of Face Processing in the Human Brain and Convolutional Neural Networks

Face processing includes two crucial processing levels - face detection and face recognition. However, it remains unclear how human brains organize the two processing levels sequentially. While some studies found that faces are recognized as fast as they are detected, others have reported that faces are detected first, followed by recognition. We discriminated the two processing levels on a fine time scale by combining human intracranial EEG (two females, three males, and three subjects without reported sex information) and representation similarity analysis. Our results demonstrate that the human brain exhibits a "detection-first, recognition-later" pattern during face processing. In addition, we used convolutional neural networks to test the hypothesis that the sequential organization of the two face processing levels in the brain reflects computational optimization. Our findings showed that the networks trained on face recognition also exhibited the "detection-first, recognition-later" pattern. Moreover, this sequential organization mechanism developed gradually during the training of the networks and was observed only for correctly predicted images. These findings collectively support the computational account as to why the brain organizes them in this way.

The Neural Dynamics of Face Ensemble and Central Face Processing

Extensive work has investigated the neural processing of single faces, including the role of shape and surface properties. However, much less is known about the neural basis of face ensemble perception (e.g., simultaneously viewing several faces in a crowd). Importantly, the contribution of shape and surface properties have not been elucidated in face ensemble processing. Furthermore, how single central faces are processed within the context of an ensemble remains unclear. Here, we probe the neural dynamics of ensemble representation using pattern analyses as applied to electrophysiology data in healthy adults (seven males, nine females). Our investigation relies on a unique set of stimuli, depicting different facial identities, which vary parametrically and independently along their shape and surface properties. These stimuli were organized into ensemble displays consisting of six surround faces arranged in a circle around one central face. Overall, our results indicate that both shape and surface properties play a significant role in face ensemble encoding, with the latter demonstrating a more pronounced contribution. Importantly, we find that the neural processing of the center face precedes that of the surround faces in an ensemble. Further, the temporal profile of center face decoding is similar to that of single faces, while those of single faces and face ensembles diverge extensively from each other. Thus, our work capitalizes on a new center-surround paradigm to elucidate the neural dynamics of ensemble processing and the information that underpins it. Critically, our results serve to bridge the study of single and ensemble face perception.

Neural mechanisms of face familiarity and learning in the human amygdala and hippocampus

Recognizing familiar faces and learning new faces play an important role in social cognition. However, the underlying neural computational mechanisms remain unclear. Here, we record from single neurons in the human amygdala and hippocampus and find a greater neuronal representational distance between pairs of familiar faces than unfamiliar faces, suggesting that neural representations for familiar faces are more distinct. Representational distance increases with exposures to the same identity, suggesting that neural face representations are sharpened with learning and familiarization. Furthermore, representational distance is positively correlated with visual dissimilarity between faces, and exposure to visually similar faces increases representational distance, thus sharpening neural representations. Finally, we construct a computational model that demonstrates an increase in the representational distance of artificial units with training. Together, our results suggest that the neuronal population geometry, quantified by the representational distance, encodes face familiarity, similarity, and learning, forming the basis of face recognition and memory.

Bridging the gap between EEG and DCNNs reveals a fatigue mechanism of facial repetition suppression

Facial repetition suppression, a well-studied phenomenon characterized by decreased neural responses to repeated faces in visual cortices, remains a subject of ongoing debate regarding its underlying neural mechanisms. Our research harnesses advanced multivariate analysis techniques and the prowess of deep convolutional neural networks (DCNNs) in face recognition to bridge the gap between human electroencephalogram (EEG) data and DCNNs, especially in the context of facial repetition suppression. Our innovative reverse engineering approach, manipulating the neuronal activity in DCNNs and conducted representational comparisons between brain activations derived from human EEG and manipulated DCNN activations, provided insights into the underlying facial repetition suppression. Significantly, our findings advocate the fatigue mechanism as the dominant force behind the facial repetition suppression effect. Broadly, this integrative framework, bridging the human brain and DCNNs, offers a promising tool for simulating brain activity and making inferences regarding the neural mechanisms underpinning complex human behaviors.

Your place or mine? The neural dynamics of personally familiar scene recognition suggests category independent familiarity encoding

Recognizing a stimulus as familiar is an important capacity in our everyday life. Recent investigation of visual processes has led to important insights into the nature of the neural representations of familiarity for human faces. Still, little is known about how familiarity affects the neural dynamics of non-face stimulus processing. Here we report the results of an EEG study, examining the representational dynamics of personally familiar scenes. Participants viewed highly variable images of their own apartments and unfamiliar ones, as well as personally familiar and unfamiliar faces. Multivariate pattern analyses were used to examine the time course of differential processing of familiar and unfamiliar stimuli. Time-resolved classification revealed that familiarity is decodable from the EEG data similarly for scenes and faces. The temporal dynamics showed delayed onsets and peaks for scenes as compared to faces. Familiarity information, starting at 200 ms, generalized across stimulus categories and led to a robust familiarity effect. In addition, familiarity enhanced category representations in early (250-300 ms) and later (>400 ms) processing stages. Our results extend previous face familiarity results to another stimulus category and suggest that familiarity as a construct can be understood as a general, stimulus-independent processing step during recognition.

The roles of edge-based and surface-based information in the dynamic neural representation of objects

We combined multivariate pattern analysis (MVPA) and electroencephalogram (EEG) to investigate the role of edge, color, and other surface information in the neural representation of visual objects. Participants completed a one-back task in which they were presented with color photographs, grayscale images, and line drawings of animals, tools, and fruits. Our results provide the first neural evidence that line drawings elicit similar neural activities as color photographs and grayscale images during the 175-305 ms window after the stimulus onset. Furthermore, we found that other surface information, rather than color information, facilitates decoding accuracy in the early stages of object representations and affects the speed of this. These results provide new insights into the role of edge-based and surface-based information in the dynamic process of neural representations of visual objects.

Stereotypes shape response competition when forming impressions

Dynamic models of impression formation posit that bottom-up factors (e.g., a target's facial features) and top-down factors (e.g., perceiver knowledge of stereotypes) continuously interact over time until a stable categorization or impression emerges. Most previous work on the dynamic resolution of judgments over time has focused on either categorization (e.g., "is this person male/female?") or specific trait impressions (e.g., "is this person trustworthy?"). In two mousetracking studies-exploratory (N = 226) and confirmatory (N = 300)-we test a domain-general effect of cultural stereotypes shaping the process underlying impressions of targets. We find that the trajectories of participants' mouse movements gravitate toward impressions congruent with their stereotype knowledge. For example, to the extent that a participant reports knowledge of a "Black men are less [trait]" stereotype, their mouse trajectory initially gravitates toward categorizing individual Black male faces as "less [trait]," regardless of their final judgment of the target.

Single neuron responses underlying face recognition in the human midfusiform face-selective cortex

Faces are critical for social interactions and their recognition constitutes one of the most important and challenging functions of the human brain. While neurons responding selectively to faces have been recorded for decades in the monkey brain, face-selective neural activations have been reported with neuroimaging primarily in the human midfusiform gyrus. Yet, the cellular mechanisms producing selective responses to faces in this hominoid neuroanatomical structure remain unknown. Here we report single neuron recordings performed in 5 human subjects (1 male, 4 females) implanted with intracerebral microelectrodes in the face-selective midfusiform gyrus, while they viewed pictures of familiar and unknown faces and places. We observed similar responses to faces and places at the single cell level, but a significantly higher number of neurons responding to faces, thus offering a mechanistic account for the face-selective activations observed in this region. Although individual neurons did not respond preferentially to familiar faces, a population level analysis could consistently determine whether or not the faces (but not the places) were familiar, only about 50 ms after the initial recognition of the stimuli as faces. These results provide insights into the neural mechanisms of face processing in the human brain.

Behavioral signatures of face perception emerge in deep neural networks optimized for face recognition

Human face recognition is highly accurate and exhibits a number of distinctive and well-documented behavioral "signatures" such as the use of a characteristic representational space, the disproportionate performance cost when stimuli are presented upside down, and the drop in accuracy for faces from races the participant is less familiar with. These and other phenomena have long been taken as evidence that face recognition is "special". But why does human face perception exhibit these properties in the first place? Here, we use deep convolutional neural networks (CNNs) to test the hypothesis that all of these signatures of human face perception result from optimization for the task of face recognition. Indeed, as predicted by this hypothesis, these phenomena are all found in CNNs trained on face recognition, but not in CNNs trained on object recognition, even when additionally trained to detect faces while matching the amount of face experience. To test whether these signatures are in principle specific to faces, we optimized a CNN on car discrimination and tested it on upright and inverted car images. As we found for face perception, the car-trained network showed a drop in performance for inverted vs. upright cars. Similarly, CNNs trained on inverted faces produced an inverted face inversion effect. These findings show that the behavioral signatures of human face perception reflect and are well explained as the result of optimization for the task of face recognition, and that the nature of the computations underlying this task may not be so special after all.

Developing familiarity during the first eight months of knowing a person: A longitudinal EEG study on face and identity learning

It is well-established that familiar and unfamiliar faces are processed differently, but surprisingly little is known about how familiarity builds up over time and how novel faces gradually become represented in the brain. Here, we used event-related brain potentials (ERPs) in a pre-registered, longitudinal study to examine the neural processes accompanying face and identity learning during the first eight months of knowing a person. Specifically, we examined how increasing real-life familiarity affects visual recognition (N250 Familiarity Effect) and the integration of person-related knowledge (Sustained Familiarity Effect, SFE). Sixteen first-year undergraduates were tested in three sessions, approximately one, five, and eight months after the start of the academic year, with highly variable "ambient" images of a new friend they had met at university and of an unfamiliar person. We observed clear ERP familiarity effects for the new friend after one month of familiarity. While there was an increase in the N250 effect over the course of the study, no change in the SFE was observed. These results suggest that visual face representations develop faster relative to the integration of identity-specific knowledge.

A model for person perception from familiar and unfamiliar voices

When hearing a voice, listeners can form a detailed impression of the person behind the voice. Existing models of voice processing focus primarily on one aspect of person perception - identity recognition from familiar voices - but do not account for the perception of other person characteristics (e.g., sex, age, personality traits). Here, we present a broader perspective, proposing that listeners have a common perceptual goal of perceiving who they are hearing, whether the voice is familiar or unfamiliar. We outline and discuss a model - the Person Perception from Voices (PPV) model - that achieves this goal via a common mechanism of recognising a familiar person, persona, or set of speaker characteristics. Our PPV model aims to provide a more comprehensive account of how listeners perceive the person they are listening to, using an approach that incorporates and builds on aspects of the hierarchical frameworks and prototype-based mechanisms proposed within existing models of voice identity recognition.

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues

Convolutional Neural Networks (CNN) are a class of machine learning models predominately used in computer vision tasks and can achieve human-like performance through learning from experience. Their striking similarities to the structural and functional principles of the primate visual system allow for comparisons between these artificial networks and their biological counterparts, enabling exploration of how visual functions and neural representations may emerge in the real brain from a limited set of computational principles. After considering the basic features of CNNs, we discuss the opportunities and challenges of endorsing CNNs as in silico models of the primate visual system. Specifically, we highlight several emerging notions about the anatomical and physiological properties of the visual system that still need to be systematically integrated into current CNN models. These tenets include the implementation of parallel processing pathways from the early stages of retinal input and the reconsideration of several assumptions concerning the serial progression of information flow. We suggest design choices and architectural constraints that could facilitate a closer alignment with biology provide causal evidence of the predictive link between the artificial and biological visual systems. Adopting this principled perspective could potentially lead to new research questions and applications of CNNs beyond modeling object recognition.

The effect of familiarity on behavioral oscillations in face perception

Studies on behavioral oscillations demonstrate that visual sensitivity fluctuates over time and visual processing varies periodically, mirroring neural oscillations at the same frequencies. Do these behavioral oscillations reflect fixed and relatively automatic sensory sampling, or top-down processes such as attention or predictive coding? To disentangle these theories, the current study used a dual-target rapid serial visual presentation paradigm, where participants indicated the gender of a face target embedded in streams of distractors presented at 30 Hz. On critical trials, two identical targets were presented with varied stimulus onset asynchrony from 200 to 833 ms. The target was either familiar or unfamiliar faces, divided into different blocks. We found a 4.6 Hz phase-coherent fluctuation in gender discrimination performance across both trial types, consistent with previous reports. In addition, however, we found an effect at the alpha frequency, with behavioral oscillations in the familiar blocks characterized by a faster high-alpha peak than for the unfamiliar face blocks. These results are consistent with the combination of both a relatively stable modulation in the theta band and faster modulation of the alpha oscillations. Therefore, the overall pattern of perceptual sampling in visual perception may depend, at least in part, on task demands. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on 16/08/2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/A98UF .

Neural representation of perceived race mediates the opposite relationship between subcomponents of self-construals and racial outgroup punishment

Outgroup aggression characterizes intergroup conflicts in human societies. Previous research on relationships between cultural traits and outgroup aggression behavior showed inconsistent results, leaving open questions regarding whether cultural traits predict individual differences in outgroup aggression and related neural underpinnings. We conducted 2 studies to address this issue by collecting self-construal scores, EEG signals in response to Asian and White faces with painful or neutral expressions, and decisions to apply electric shocks to other-race individuals in a context of interracial conflict. We found that interdependent self-construals were well explained by 2 subcomponents, including esteem for group (EG) and relational interdependence (RI), which are related to focus on group collectives and harmonious relationships, respectively. Moreover, EG was positively associated with the decisions to punish racial outgroup targets, whereas RI was negatively related to the decisions. These opposite relationships were mediated by neural representations of perceived race at 120-160 ms after face onset. Our findings highlight the multifaceted nature of interdependent self-construal and the key role of neural representations of race in mediating the relationships of different subcomponents of cultural traits with racial outgroup punishment decisions in a context of interracial conflict.

EEG alpha band functional connectivity reveals distinct cortical dynamics for overt and covert emotional face processing

Current knowledge regarding how the focus of our attention during face processing influences neural responses largely comes from neuroimaging studies reporting on regional brain activations. The present study was designed to add novel insights to this research by studying how attention can differentially impact the way cortical regions interact during emotional face processing. High-density electroencephalogram was recorded in a sample of fifty-two healthy participants during an emotional face processing task. The task required participants to either attend to the expressions (i.e., overt processing) or attend to a perceptual distractor, which rendered the expressions task-irrelevant (i.e., covert processing). Functional connectivity in the alpha band was estimated in source space and modeled using graph theory to quantify whole-brain integration and segregation. Results revealed that overt processing of facial expressions is linked to reduced cortical segregation and increased cortical integration, this latter specifically for negative expressions of fear and sadness. Furthermore, we observed increased communication efficiency during overt processing of negative expressions between the core and the extended face processing systems. Overall, these findings reveal that attention makes the interaction among the nodes involved in face processing more efficient, also uncovering a connectivity signature of the prioritized processing mechanism of negative expressions, that is an increased cross-communication within the nodes of the face processing network.

Intracranial Electroencephalography and Deep Neural Networks Reveal Shared Substrates for Representations of Face Identity and Expressions

According to a classical view of face perception (Bruce and Young, 1986; Haxby et al., 2000), face identity and facial expression recognition are performed by separate neural substrates (ventral and lateral temporal face-selective regions, respectively). However, recent studies challenge this view, showing that expression valence can also be decoded from ventral regions (Skerry and Saxe, 2014; Li et al., 2019), and identity from lateral regions (Anzellotti and Caramazza, 2017). These findings could be reconciled with the classical view if regions specialized for one task (either identity or expression) contain a small amount of information for the other task (that enables above-chance decoding). In this case, we would expect representations in lateral regions to be more similar to representations in deep convolutional neural networks (DCNNs) trained to recognize facial expression than to representations in DCNNs trained to recognize face identity (the converse should hold for ventral regions). We tested this hypothesis by analyzing neural responses to faces varying in identity and expression. Representational dissimilarity matrices (RDMs) computed from human intracranial recordings (n = 11 adults; 7 females) were compared with RDMs from DCNNs trained to label either identity or expression. We found that RDMs from DCNNs trained to recognize identity correlated with intracranial recordings more strongly in all regions tested-even in regions classically hypothesized to be specialized for expression. These results deviate from the classical view, suggesting that face-selective ventral and lateral regions contribute to the representation of both identity and expression.SIGNIFICANCE STATEMENT Previous work proposed that separate brain regions are specialized for the recognition of face identity and facial expression. However, identity and expression recognition mechanisms might share common brain regions instead. We tested these alternatives using deep neural networks and intracranial recordings from face-selective brain regions. Deep neural networks trained to recognize identity and networks trained to recognize expression learned representations that correlate with neural recordings. Identity-trained representations correlated with intracranial recordings more strongly in all regions tested, including regions hypothesized to be expression specialized in the classical hypothesis. These findings support the view that identity and expression recognition rely on common brain regions. This discovery may require reevaluation of the roles that the ventral and lateral neural pathways play in processing socially relevant stimuli.

Equivalent processing of facial expression and identity by macaque visual system and task-optimized neural network

Both the primate visual system and artificial deep neural network (DNN) models show an extraordinary ability to simultaneously classify facial expression and identity. However, the neural computations underlying the two systems are unclear. Here, we developed a multi-task DNN model that optimally classified both monkey facial expressions and identities. By comparing the fMRI neural representations of the macaque visual cortex with the best-performing DNN model, we found that both systems: (1) share initial stages for processing low-level face features which segregate into separate branches at later stages for processing facial expression and identity respectively, and (2) gain more specificity for the processing of either facial expression or identity as one progresses along each branch towards higher stages. Correspondence analysis between the DNN and monkey visual areas revealed that the amygdala and anterior fundus face patch (AF) matched well with later layers of the DNN's facial expression branch, while the anterior medial face patch (AM) matched well with later layers of the DNN's facial identity branch. Our results highlight the anatomical and functional similarities between macaque visual system and DNN model, suggesting a common mechanism between the two systems.

Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans

Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. Recently, neural decoding schemes, such as machine learning in functional magnetic resonance imaging (fMRI), has been employed to elucidate the underlying neural mechanisms. However, it remains unclear as to how spatially distributed brain regions temporally represent visual object categories and sub-categories. One promising strategy to address this issue is neural decoding with concurrently obtained neural response data of high spatial and temporal resolution. In this study, we explored the spatial and temporal organization of visual object representations using concurrent fMRI and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). We hypothesized that neural decoding by multimodal neural data with DNN would show high classification performance in visual object categorization (faces or non-face objects) and sub-categorization within faces and objects. Visualization of the fMRI DNN was more sensitive than that in the univariate approach and revealed that visual categorization occurred in brain-wide regions. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization compared with fMRI DNN or EEG DNN alone. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner, and provide strong evidence of the ability of multimodal deep learning to uncover spatiotemporal neural machinery in sensory processing.

Dimensions underlying human understanding of the reachable world

Near-scale environments, like work desks, restaurant place settings or lab benches, are the interface of our hand-based interactions with the world. How are our conceptual representations of these environments organized? What properties distinguish among reachspaces, and why? We obtained 1.25 million similarity judgments on 990 reachspace images, and generated a 30-dimensional embedding which accurately predicts these judgments. Examination of the embedding dimensions revealed key properties underlying these judgments, such as reachspace layout, affordance, and visual appearance. Clustering performed over the embedding revealed four distinct interpretable classes of reachspaces, distinguishing among spaces related to food, electronics, analog activities, and storage or display. Finally, we found that reachspace similarity ratings were better predicted by the function of the spaces than their locations, suggesting that reachspaces are largely conceptualized in terms of the actions they support. Altogether, these results reveal the behaviorally-relevant principles that structure our internal representations of reach-relevant environments.

Towards a pluralistic neurobiological understanding of consciousness

Theories of consciousness are often based on the assumption that a single, unified neurobiological account will explain different types of conscious awareness. However, recent findings show that, even within a single modality such as conscious visual perception, the anatomical location, timing, and information flow of neural activity related to conscious awareness vary depending on both external and internal factors. This suggests that the search for generic neural correlates of consciousness may not be fruitful. I argue that consciousness science requires a more pluralistic approach and propose a new framework: joint determinant theory (JDT). This theory may be capable of accommodating different brain circuit mechanisms for conscious contents as varied as percepts, wills, memories, emotions, and thoughts, as well as their integrated experience.

Decomposing Neural Representational Patterns of Discriminatory and Hedonic Information during Somatosensory Stimulation

The ability to interrogate specific representations in the brain, determining how, and where, difference sources of information are instantiated can provide invaluable insight into neural functioning. Pattern component modeling (PCM) is a recent analytic technique for human neuroimaging that allows the decomposition of representational patterns in brain into contributing subcomponents. In the current study, we present a novel PCM variant that tracks the contribution of prespecified representational patterns to brain representation across areas, thus allowing hypothesis-guided employment of the technique. We apply this technique to investigate the contributions of hedonic and nonhedonic information to the neural representation of tactile experience. We applied aversive pressure (AP) and appetitive brush (AB) to stimulate distinct peripheral nerve pathways for tactile information (C-/CT-fibers, respectively) while patients underwent functional magnetic resonance imaging (fMRI) scanning. We performed representational similarity analyses (RSAs) with pattern component modeling to dissociate how discriminatory versus hedonic tactile information contributes to population code representations in the human brain. Results demonstrated that information about appetitive and aversive tactile sensation is represented separately from nonhedonic tactile information across cortical structures. This also demonstrates the potential of new hypothesis-guided PCM variants to help delineate how information is instantiated in the brain.

How do we describe other people from voices and faces?

When seeing someone's face or hearing their voice, perceivers routinely infer information about a person's age, sex and social traits. While many experiments have explored how individual person characteristics are perceived in isolation, less is known about which person characteristics are described spontaneously from voices and faces and how descriptions may differ across modalities. In Experiment 1, participants provided free descriptions for voices and faces. These free descriptions followed similar patterns for voices and faces - and for individual identities: Participants spontaneously referred to a wide range of descriptors. Psychological descriptors, such as character traits, were used most frequently; physical characteristics, such as age and sex, were notable as they were mentioned earlier than other types of descriptors. After finding primarily similarities between modalities when analysing person descriptions across identities, Experiment 2 asked whether free descriptions encode how individual identities differ. For this purpose, the measures derived from the free descriptions were linked to voice/face discrimination judgements that are known to describe differences in perceptual properties between identity pairs. Significant relationships emerged within and across modalities, showing that free descriptions indeed encode differences between identities - information that is shared with discrimination judgements. This suggests that the two tasks tap into similar, high-level person representations. These findings show that free description data can offer valuable insights into person perception and underline that person perception is a multivariate process during which perceivers rapidly and spontaneously infer many different person characteristics to form a holistic impression of a person.