1
|
Ozdemir B, Ambrus GG. From encoding to recognition: Exploring the shared neural signatures of visual memory. Brain Res 2025; 1857:149616. [PMID: 40187518 DOI: 10.1016/j.brainres.2025.149616] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2025] [Revised: 03/24/2025] [Accepted: 04/03/2025] [Indexed: 04/07/2025]
Abstract
This study investigated the shared neural dynamics underlying encoding and recognition processes across diverse visual object stimulus types in short term experimental familiarization, using EEG-based representational similarity analysis and multivariate cross-classification. Building upon previous research, we extended our exploration to the encoding phase. We show early visual stimulus category effects around 150 ms post-stimulus onset and old/new effects around 400 to 600 ms. Notably, a divergence in neural responses for encoding, old, and new stimuli emerged around 300 ms, with items encountered during the study phase showing the highest differentiation from old items during the test phase. Cross-category classification demonstrated discernible memory-related effects as early as 150 ms. Anterior regions of interest, particularly in the right hemisphere, did not exhibit differentiation between experimental phases or between study and new items, hinting at similar processing for items first encountered, irrespective of experiment phase. While short-term experimental familiarity did not consistently adhere to the old >new pattern observed in long-term personal familiarity, statistically significant effects are observed specifically for experimentally familiarized faces, suggesting a potential unique phenomenon specific to facial stimuli. Further investigation is warranted to elucidate underlying mechanisms and determine the extent of face-specific effects. Lastly, our findings underscore the utility of multivariate cross-classification and cross-dataset classification as promising tools for probing abstraction and shared neural signatures of cognitive processing.
Collapse
Affiliation(s)
- Berfin Ozdemir
- Department of Psychology, Bournemouth University, Poole House, Talbot Campus, Fern Barrow, Poole, Dorset BH12 5BB, United Kingdom
| | - Géza Gergely Ambrus
- Department of Psychology, Bournemouth University, Poole House, Talbot Campus, Fern Barrow, Poole, Dorset BH12 5BB, United Kingdom.
| |
Collapse
|
2
|
Phillips PJ, White D. The state of modelling face processing in humans with deep learning. Br J Psychol 2025. [PMID: 40364689 DOI: 10.1111/bjop.12794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2024] [Accepted: 04/20/2025] [Indexed: 05/15/2025]
Abstract
Deep learning models trained for facial recognition now surpass the highest performing human participants. Recent evidence suggests that they also model some qualitative aspects of face processing in humans. This review compares the current understanding of deep learning models with psychological models of the face processing system. Psychological models consist of two components that operate on the information encoded when people perceive a face, which we refer to here as 'face codes'. The first component, the core system, extracts face codes from retinal input that encode invariant and changeable properties. The second component, the extended system, links face codes to personal information about a person and their social context. Studies of face codes in existing deep learning models reveal some surprising results. For example, face codes in networks designed for identity recognition also encode expression information, which contrasts with psychological models that separate invariant and changeable properties. Deep learning can also be used to implement candidate models of the face processing system, for example to compare alternative cognitive architectures and codes that might support interchange between core and extended face processing systems. We conclude by summarizing seven key lessons from this research and outlining three open questions for future study.
Collapse
Affiliation(s)
| | - David White
- School of Psychology, UNSW Sydney, Sydney, New South Wales, Australia
| |
Collapse
|
3
|
Liang J, Zhang M, Yang L, Li Y, Li Y, Wang L, Li H, Chen J, Luo W. How Linguistic and Nonlinguistic Vocalizations Shape the Perception of Emotional Faces-An Electroencephalography Study. J Cogn Neurosci 2025; 37:970-987. [PMID: 39620941 DOI: 10.1162/jocn_a_02284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/08/2025]
Abstract
Vocal emotions are crucial in guiding visual attention toward emotionally significant environmental events, such as recognizing emotional faces. This study employed continuous EEG recordings to examine the impact of linguistic and nonlinguistic vocalizations on facial emotion processing. Participants completed a facial emotion discrimination task while viewing fearful, happy, and neutral faces. The behavioral and ERP results indicated that fearful nonlinguistic vocalizations accelerated the recognition of fearful faces and elicited a larger P1 amplitude, whereas happy linguistic vocalizations accelerated the recognition of happy faces and similarly induced a greater P1 amplitude. In recognition of fearful faces, a greater N170 component was observed in the right hemisphere when the emotional category of the priming vocalization was consistent with the face stimulus. In contrast, this effect occurred in the left hemisphere while recognizing happy faces. Representational similarity analysis revealed that the temporoparietal regions automatically differentiate between linguistic and nonlinguistic vocalizations early in face processing. In conclusion, these findings enhance our understanding of the interplay between vocalization types and facial emotion recognition, highlighting the importance of cross-modal processing in emotional perception.
Collapse
Affiliation(s)
- Junyu Liang
- South China Normal University
- Liaoning Normal University
- Key Laboratory of Brain and Cognitive Neuroscience
| | - Mingming Zhang
- Liaoning Normal University
- Key Laboratory of Brain and Cognitive Neuroscience
| | - Lan Yang
- South China Normal University
- Liaoning Normal University
- Key Laboratory of Brain and Cognitive Neuroscience
| | - Yiwen Li
- Liaoning Normal University
- Key Laboratory of Brain and Cognitive Neuroscience
- Beijing Normal University
| | - Yuchen Li
- Liaoning Normal University
- Key Laboratory of Brain and Cognitive Neuroscience
| | - Li Wang
- South China Normal University
| | | | | | - Wenbo Luo
- Liaoning Normal University
- Key Laboratory of Brain and Cognitive Neuroscience
| |
Collapse
|
4
|
Yao L, Fu Q, Liu CH, Wang J, Yi Z. Distinguishing the roles of edge, color, and other surface information in basic and superordinate scene representation. Neuroimage 2025; 310:121100. [PMID: 40021071 DOI: 10.1016/j.neuroimage.2025.121100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2024] [Revised: 02/13/2025] [Accepted: 02/26/2025] [Indexed: 03/03/2025] Open
Abstract
The human brain possesses a remarkable ability to recognize scenes depicted in line drawings, despite that these drawings contain only edge information. It remains unclear how the brain uses this information alongside surface information in scene recognition. Here, we combined electroencephalogram (EEG) and multivariate pattern analysis (MVPA) methods to distinguish the roles of edge, color, and other surface information in scene representation at the basic category level and superordinate naturalness level over time. The time-resolved decoding results indicated that edge information in line drawings is both sufficient and more effective than in color photographs and grayscale images at the superordinate naturalness level. Meanwhile, color and other surface information are exclusively involved in neural representation at the basic category level. The time generalization analysis further revealed that edge information is crucial for representation at both levels of abstraction. These findings highlight the distinct roles of edge, color, and other surface information in dynamic neural scene processing, shedding light on how the human brain represents scene information at different levels of abstraction.
Collapse
Affiliation(s)
- Liansheng Yao
- State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Qiufang Fu
- State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China.
| | - Chang Hong Liu
- Department of Psychology, Bournemouth University, Fern Barrow, Poole, UK
| | - Jianyong Wang
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, China
| | - Zhang Yi
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu, China
| |
Collapse
|
5
|
Schmidt F, Hebart MN, Schmid AC, Fleming RW. Core dimensions of human material perception. Proc Natl Acad Sci U S A 2025; 122:e2417202122. [PMID: 40042912 PMCID: PMC11912425 DOI: 10.1073/pnas.2417202122] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2024] [Accepted: 01/24/2025] [Indexed: 03/19/2025] Open
Abstract
Visually categorizing and comparing materials is crucial for everyday behavior, but what organizational principles underlie our mental representation of materials? Here, we used a large-scale data-driven approach to uncover core latent dimensions of material representations from behavior. First, we created an image dataset of 200 systematically sampled materials and 600 photographs (STUFF dataset, https://osf.io/myutc/). Using these images, we next collected 1.87 million triplet similarity judgments and used a computational model to derive a set of sparse, positive dimensions underlying these judgments. The resulting multidimensional embedding space predicted independent material similarity judgments and the similarity matrix of all images close to the human intersubject consistency. We found that representations of individual images were captured by a combination of 36 material dimensions that were highly reproducible and interpretable, comprising perceptual (e.g., grainy, blue) as well as conceptual (e.g., mineral, viscous) dimensions. These results provide the foundation for a comprehensive understanding of how humans make sense of materials.
Collapse
Affiliation(s)
- Filipp Schmidt
- Experimental Psychology, Justus Liebig University, Giessen35394, Germany
- Center for Mind, Brain and Behavior, Universities of Marburg, Giessen, and Darmstadt, Marburg35032, Germany
| | - Martin N. Hebart
- Center for Mind, Brain and Behavior, Universities of Marburg, Giessen, and Darmstadt, Marburg35032, Germany
- Department of Medicine, Justus Liebig University, Giessen35390, Germany
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig04103, Germany
| | - Alexandra C. Schmid
- Experimental Psychology, Justus Liebig University, Giessen35394, Germany
- Laboratory of Brain and Cognition, National Institute of Mental Health, Bethesda, MD20814
| | - Roland W. Fleming
- Experimental Psychology, Justus Liebig University, Giessen35394, Germany
- Center for Mind, Brain and Behavior, Universities of Marburg, Giessen, and Darmstadt, Marburg35032, Germany
| |
Collapse
|
6
|
Shoura M, Liang YZ, Sama MA, De A, Nestor A. Revealing the neural representations underlying other-race face perception. Front Hum Neurosci 2025; 19:1543840. [PMID: 40110535 PMCID: PMC11920127 DOI: 10.3389/fnhum.2025.1543840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2024] [Accepted: 02/17/2025] [Indexed: 03/22/2025] Open
Abstract
The other-race effect (ORE) refers to poorer recognition for faces of other races than one's own. This study investigates the neural and representational basis of ORE in East Asian and White participants using behavioral measures, neural decoding, and image reconstruction based on electroencephalography (EEG) data. Our investigation identifies a reliable neural counterpart of ORE, with reduced decoding accuracy for other-race faces, and it relates this result to higher density of other-race face representations in face space. Then, we characterize the temporal dynamics and the prominence of ORE for individual variability at the neural level. Importantly, we use a data-driven image reconstruction approach to reveal visual biases underlying other-race face perception, including a tendency to perceive other-race faces as more typical, younger, and more expressive. These findings provide neural evidence for a classical account of ORE invoking face space compression for other-race faces. Further, they indicate that ORE involves not only reduced identity information but also broader, systematic distortions in visual representation with considerable cognitive and social implications.
Collapse
Affiliation(s)
- Moaz Shoura
- Department of Psychology at Scarborough, University of Toronto, Toronto, ON, Canada
| | - Yong Z Liang
- Department of Psychology at Scarborough, University of Toronto, Toronto, ON, Canada
| | - Marco A Sama
- Department of Psychology at Scarborough, University of Toronto, Toronto, ON, Canada
| | - Arijit De
- Department of Psychology at Scarborough, University of Toronto, Toronto, ON, Canada
| | - Adrian Nestor
- Department of Psychology at Scarborough, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
7
|
Araya J, Ikeda T, Hasegawa C, Iwasaki S, Yaoi K, Yoshimura Y. Differential lateralization to faces in infants at risk of autism spectrum disorder with expressive language delay. PCN REPORTS : PSYCHIATRY AND CLINICAL NEUROSCIENCES 2025; 4:e70054. [PMID: 39935969 PMCID: PMC11811886 DOI: 10.1002/pcn5.70054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/15/2024] [Revised: 12/15/2024] [Accepted: 01/02/2025] [Indexed: 02/13/2025]
Abstract
Aim Face-to-face communication between caregiver and infant is essential for the development of language and social skills in infancy. A previous study on brain response toward human faces showed that a lateralization right fusiform gyrus (FG) response when viewing faces was associated with better social skills. However, the relationship, between infant face processing and language development remains unclear. This study aimed to examine whether brain responses to faces vary based on the ability level of language expression. Methods Overall, 42 Japanese infants (aged 18-34 months, Mean of age (Mage) = 24.7 months, standard deviation (SD) = 4.57, 47% female) were assessed for expressive communication skills and classified into two groups: a delayed group (20 infants) and a control group (infants with typical expressive language development, 22 infants). Brain activity was recorded using a child-customized magnetoencephalography during presentation of a mother's face, a stranger's face, and a nonface (scrambled image). The lateralization index of the FG during face viewing was calculated using the following formula: (L - R)/(L + R). Results The results showed a significant difference in the lateralization index between the delayed and control groups. The control group showed rightward dominance of the FG activity when viewing the mother's face and others' faces, whereas the delayed group did not exhibit this lateralization. Based on behavioral observations, 75% of the delayed group met the criteria of autism spectrum disorder (ASD) risk, and infants with a high risk of ASD who had poor expressive language showed poor right hemispheric dominance compared to the control group in their brain responses to their mothers' faces. Conclusion This study suggests that lateralization of face processing in infancy may be a predictor of expressive language abilities.
Collapse
Affiliation(s)
- Junko Araya
- United Graduate School of Child DevelopmentOsaka UniversitySuitaJapan
- Research Center for Child Mental DevelopmentKanazawa UniversityKanazawaJapan
| | - Takashi Ikeda
- Research Center for Child Mental DevelopmentKanazawa UniversityKanazawaJapan
- United Graduate School of Child DevelopmentKanazawa UniversityKanazawaJapan
| | - Chiaki Hasegawa
- Research Center for Child Mental DevelopmentKanazawa UniversityKanazawaJapan
- United Graduate School of Child DevelopmentKanazawa UniversityKanazawaJapan
| | - Sumie Iwasaki
- Institute of Human and Social SciencesKanazawa UniversityKanazawaJapan
- Japan Society for the Promotion of ScienceTokyoJapan
| | - Ken Yaoi
- Department of Psychology, Faculty of Liberal ArtsTeikyo UniversityTokyoJapan
| | - Yuko Yoshimura
- Research Center for Child Mental DevelopmentKanazawa UniversityKanazawaJapan
- United Graduate School of Child DevelopmentKanazawa UniversityKanazawaJapan
- Institute of Human and Social SciencesKanazawa UniversityKanazawaJapan
| |
Collapse
|
8
|
Liu C, Ma Y, Liang X, Xiang M, Wu H, Ning X. Decoding the Spatiotemporal Dynamics of Neural Response Similarity in Auditory Processing: A Multivariate Analysis Based on OPM-MEG. Hum Brain Mapp 2025; 46:e70175. [PMID: 40016919 PMCID: PMC11868016 DOI: 10.1002/hbm.70175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2024] [Revised: 01/26/2025] [Accepted: 02/12/2025] [Indexed: 03/01/2025] Open
Abstract
The brain represents information through the encoding of neural populations, where the activity patterns of these neural groups constitute the content of this information. Understanding these activity patterns and their dynamic changes is of significant importance to cognitive neuroscience and related research areas. Current studies focus more on brain regions that show differential responses to stimuli, but they lack the ability to capture information about the representational or process-level dynamics within these regions. In this study, we recorded neural data from 10 healthy participants during auditory experiments using optically pumped magnetometer magnetoencephalography (OPM-MEG) and electroencephalography (EEG). We constructed representational similarity matrices (RSMs) to investigate the similarity of neural response patterns during auditory decoding. The results indicate that RSA can reveal the dynamic changes in pattern similarity during different stages of auditory processing through the neural activity patterns reflected by OPM-MEG. Comparisons with EEG results showed that both techniques captured the same processes during the early stages of auditory decoding. However, differences in sensitivity at later stages highlighted both common and distinct aspects of neural representation between the two modalities. Further analysis indicated that this process involved widespread neural network activation, including the Heschl's gyrus, superior temporal gyrus, middle temporal gyrus, inferior temporal gyrus, parahippocampal gyrus, and orbitofrontal gyrus. This study demonstrates that the combination of OPM-MEG and RSA is sufficiently sensitive to detect changes in pattern similarity during neural representation processes and to identify their anatomical origins, offering new insights and references for the future application of RSA and other multivariate pattern analysis methods in the MEG field.
Collapse
Affiliation(s)
- Changzeng Liu
- Key Laboratory of Ultra‐Weak Magnetic Field Measurement Technology, Ministry of Education, School of Instrumentation and Optoelectronic EngineeringBeihang UniversityBeijingChina
- Hangzhou Institute of National Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
| | - Yuyu Ma
- Key Laboratory of Ultra‐Weak Magnetic Field Measurement Technology, Ministry of Education, School of Instrumentation and Optoelectronic EngineeringBeihang UniversityBeijingChina
- Hangzhou Institute of National Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
| | - Xiaoyu Liang
- Key Laboratory of Ultra‐Weak Magnetic Field Measurement Technology, Ministry of Education, School of Instrumentation and Optoelectronic EngineeringBeihang UniversityBeijingChina
- Hangzhou Institute of National Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
| | - Min Xiang
- Key Laboratory of Ultra‐Weak Magnetic Field Measurement Technology, Ministry of Education, School of Instrumentation and Optoelectronic EngineeringBeihang UniversityBeijingChina
- Hangzhou Institute of National Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
- Hefei National LaboratoryHefeiAnhuiChina
- Key Laboratory of Traditional Chinese Medicine SyndromeNational Institute of Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
| | - Huanqi Wu
- Key Laboratory of Ultra‐Weak Magnetic Field Measurement Technology, Ministry of Education, School of Instrumentation and Optoelectronic EngineeringBeihang UniversityBeijingChina
- Hangzhou Institute of National Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
| | - Xiaoling Ning
- Key Laboratory of Ultra‐Weak Magnetic Field Measurement Technology, Ministry of Education, School of Instrumentation and Optoelectronic EngineeringBeihang UniversityBeijingChina
- Hangzhou Institute of National Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
- Hefei National LaboratoryHefeiAnhuiChina
- Key Laboratory of Traditional Chinese Medicine SyndromeNational Institute of Extremely‐Weak Magnetic Field InfrastructureHangzhouZhejiangChina
- Shandong Key Laboratory for Magnetic Field‐Free Medicine and Functional Imaging, Institute of Magnetic Field‐Free Medicine and Functional ImagingShandong UniversityJinanShandongChina
| |
Collapse
|
9
|
Gong X, He T, Wang Q, Lu J, Fang F. Time Course of Orientation Ensemble Representation in the Human Brain. J Neurosci 2025; 45:e1688232024. [PMID: 39746825 PMCID: PMC11823330 DOI: 10.1523/jneurosci.1688-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Revised: 12/02/2024] [Accepted: 12/10/2024] [Indexed: 01/04/2025] Open
Abstract
Natural scenes are filled with groups of similar items. Humans employ ensemble coding to extract the summary statistical information of the environment, thereby enhancing the efficiency of information processing, something particularly useful when observing natural scenes. However, the neural mechanisms underlying the representation of ensemble information in the brain remain elusive. In particular, whether ensemble representation results from the mere summation of individual item representations or it engages other specific processes remains unclear. In this study, we utilized a set of orientation ensembles wherein none of the individual item orientations were the same as the ensemble orientation. We recorded magnetoencephalography (MEG) signals from human participants (both sexes) when they performed an ensemble orientation discrimination task. Time-resolved multivariate pattern analysis (MVPA) and the inverted encoding model (IEM) were employed to unravel the neural mechanisms of the ensemble orientation representation and track its time course. First, we achieved successful decoding of the ensemble orientation, with a high correlation between the decoding and behavioral accuracies. Second, the IEM analysis demonstrated that the representation of the ensemble orientation differed from the sum of the representations of individual item orientations, suggesting that ensemble coding could further modulate orientation representation in the brain. Moreover, using source reconstruction, we showed that the representation of ensemble orientation manifested in early visual areas. Taken together, our findings reveal the emergence of the ensemble representation in the human visual cortex and advance the understanding of how the brain captures and represents ensemble information.
Collapse
Affiliation(s)
- Xizi Gong
- School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing 100871, People's Republic of China
- IDG/McGovern Institute for Brain Research, Peking University, Beijing 100871, People's Republic of China
- Peking-Tsinghua Center for Life Sciences, Peking University, Beijing 100871, People's Republic of China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing 100871, People's Republic of China
| | - Tao He
- Center for the Cognitive Science of Language, Beijing Language and Culture University, Beijing 100083, People's Republic of China
| | - Qian Wang
- School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing 100871, People's Republic of China
- IDG/McGovern Institute for Brain Research, Peking University, Beijing 100871, People's Republic of China
- Peking-Tsinghua Center for Life Sciences, Peking University, Beijing 100871, People's Republic of China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing 100871, People's Republic of China
| | - Junshi Lu
- School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing 100871, People's Republic of China
- IDG/McGovern Institute for Brain Research, Peking University, Beijing 100871, People's Republic of China
| | - Fang Fang
- School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing 100871, People's Republic of China
- IDG/McGovern Institute for Brain Research, Peking University, Beijing 100871, People's Republic of China
- Peking-Tsinghua Center for Life Sciences, Peking University, Beijing 100871, People's Republic of China
- Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing 100871, People's Republic of China
| |
Collapse
|
10
|
Gupta P, Dobs K. Human-like face pareidolia emerges in deep neural networks optimized for face and object recognition. PLoS Comput Biol 2025; 21:e1012751. [PMID: 39869654 PMCID: PMC11790231 DOI: 10.1371/journal.pcbi.1012751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Revised: 02/03/2025] [Accepted: 12/24/2024] [Indexed: 01/29/2025] Open
Abstract
The human visual system possesses a remarkable ability to detect and process faces across diverse contexts, including the phenomenon of face pareidolia--seeing faces in inanimate objects. Despite extensive research, it remains unclear why the visual system employs such broadly tuned face detection capabilities. We hypothesized that face pareidolia results from the visual system's optimization for recognizing both faces and objects. To test this hypothesis, we used task-optimized deep convolutional neural networks (CNNs) and evaluated their alignment with human behavioral signatures and neural responses, measured via magnetoencephalography (MEG), related to pareidolia processing. Specifically, we trained CNNs on tasks involving combinations of face identification, face detection, object categorization, and object detection. Using representational similarity analysis, we found that CNNs that included object categorization in their training tasks represented pareidolia faces, real faces, and matched objects more similarly to neural responses than those that did not. Although these CNNs showed similar overall alignment with neural data, a closer examination of their internal representations revealed that specific training tasks had distinct effects on how pareidolia faces were represented across layers. Finally, interpretability methods revealed that only a CNN trained for both face identification and object categorization relied on face-like features-such as 'eyes'-to classify pareidolia stimuli as faces, mirroring findings in human perception. Our results suggest that human-like face pareidolia may emerge from the visual system's optimization for face identification within the context of generalized object categorization.
Collapse
Affiliation(s)
- Pranjul Gupta
- Department of Experimental Psychology, Justus Liebig University Giessen, Giessen, Germany
| | - Katharina Dobs
- Department of Experimental Psychology, Justus Liebig University Giessen, Giessen, Germany
- Center for Mind, Brain, and Behavior, Universities of Marburg, Giessen and Darmstadt, Marburg, Germany
| |
Collapse
|
11
|
Wiese H, Schweinberger SR, Kovács G. The neural dynamics of familiar face recognition. Neurosci Biobehav Rev 2024; 167:105943. [PMID: 39557351 DOI: 10.1016/j.neubiorev.2024.105943] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2024] [Revised: 09/17/2024] [Accepted: 11/11/2024] [Indexed: 11/20/2024]
Abstract
Humans are highly efficient at recognising familiar faces. However, previous EEG/ERP research has given a partial and fragmented account of the neural basis of this remarkable ability. We argue that this is related to insufficient consideration of fundamental characteristics of familiar face recognition. These include image-independence (recognition across different pictures), levels of familiarity (familiar faces vary hugely in duration and intensity of our exposure to them), automaticity (we cannot voluntarily withhold from recognising a familiar face), and domain-selectivity (the degree to which face familiarity effects are selective). We review recent EEG/ERP work, combining uni- and multivariate methods, that has systematically targeted these shortcomings. We present a theoretical account of familiar face recognition, dividing it into early visual, domain-sensitive and domain-general phases, and integrating image-independence and levels of familiarity. Our account incorporates classic and more recent concepts, such as multi-dimensional face representation and course-to-fine processing. While several questions remain to be addressed, this new account represents a major step forward in our understanding of the neurophysiological basis of familiar face recognition.
Collapse
|
12
|
Xu W, Lyu B, Ru X, Li D, Gu W, Ma X, Zheng F, Li T, Liao P, Cheng H, Yang R, Song J, Jin Z, Li C, He K, Gao JH. Decoding the Temporal Structures and Interactions of Multiple Face Dimensions Using Optically Pumped Magnetometer Magnetoencephalography (OPM-MEG). J Neurosci 2024; 44:e2237232024. [PMID: 39358044 PMCID: PMC11580774 DOI: 10.1523/jneurosci.2237-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Revised: 09/18/2024] [Accepted: 09/25/2024] [Indexed: 10/04/2024] Open
Abstract
Humans possess a remarkable ability to rapidly access diverse information from others' faces with just a brief glance, which is crucial for intricate social interactions. While previous studies using event-related potentials/fields have explored various face dimensions during this process, the interplay between these dimensions remains unclear. Here, by applying multivariate decoding analysis to neural signals recorded with optically pumped magnetometer magnetoencephalography, we systematically investigated the temporal interactions between invariant and variable aspects of face stimuli, including race, gender, age, and expression. First, our analysis revealed unique temporal structures for each face dimension with high test-retest reliability. Notably, expression and race exhibited a dominant and stably maintained temporal structure according to temporal generalization analysis. Further exploration into the mutual interactions among face dimensions uncovered age effects on gender and race, as well as expression effects on race, during the early stage (∼200-300 ms postface presentation). Additionally, we observed a relatively late effect of race on gender representation, peaking ∼350 ms after the stimulus onset. Taken together, our findings provide novel insights into the neural dynamics underlying the multidimensional aspects of face perception and illuminate the promising future of utilizing OPM-MEG for exploring higher-level human cognition.
Collapse
Affiliation(s)
- Wei Xu
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
| | | | - Xingyu Ru
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Dongxu Li
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Wenyu Gu
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
| | - Xiao Ma
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Fufu Zheng
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Tingyue Li
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Pan Liao
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
| | - Hao Cheng
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
| | - Rui Yang
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Jingqi Song
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | - Zeyu Jin
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
| | | | - Kaiyan He
- Changping Laboratory, Beijing 102206, China
| | - Jia-Hong Gao
- Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China
- Changping Laboratory, Beijing 102206, China
- Beijing City Key Lab for Medical Physics and Engineering, Institution of Heavy Ion Physics, School of Physics, Peking University, Beijing 100871, China
- McGovern Institute for Brain Research, Peking University, Beijing 100871, China
- National Biomedical Imaging Center, Peking University, Beijing 100871, China
| |
Collapse
|
13
|
Ransom M. The perceptual learning of socially constructed kinds: how culture biases and shapes perception. PHILOSOPHICAL STUDIES 2024; 181:3113-3133. [DOI: 10.1007/s11098-024-02211-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 08/18/2024] [Indexed: 01/05/2025]
|
14
|
Šķilters J, Zariņa L, Ceple I, Monstvilaite A, Umbraško S, Bartušēvica S, Pinna B. Expressions of emotions in minimal face perception stimuli. Iperception 2024; 15:20416695241291648. [PMID: 40291772 PMCID: PMC12032488 DOI: 10.1177/20416695241291648] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Accepted: 09/30/2024] [Indexed: 04/30/2025] Open
Abstract
Face perception is considered to be a canonical example of configurational visual processing. However, not all facial information is equally important when reading facial expressions. The eyes and mouth seem to be crucial, but they seem to have different roles and significance. By varying the shape of the mouth, eyes, and other factors, we conducted two experiments: first, we examined eye movements depending on different facial configurations and different types of instructions (neutral and emotionally valenced); second, we used the same types of stimuli in a rating task. Our results indicate that the eyes provide a primary impact (when eye fixations are measured), which can be explained by the evolutionary need to establish gaze contact, but once facial expressions are observed, the mouth seems to be more significant.
Collapse
Affiliation(s)
| | | | | | - Alina Monstvilaite
- Laboratory for Perceptual and Cognitive Systems
- Faculty of Science and Technology, University of Latvia, Riga, Latvia
| | - Solvita Umbraško
- Laboratory for Perceptual and Cognitive Systems
- Faculty of Education Sciences and Psychology, University of Latvia, Riga, Latvia
| | - Santa Bartušēvica
- Laboratory for Perceptual and Cognitive Systems
- Faculty of Science and Technology, University of Latvia, Riga, Latvia
| | | |
Collapse
|
15
|
Liu Y, Li D, Wang W, Jiang Z. What will we judge a book by its cover?-Content analysis of face perception in a Chinese sample. Acta Psychol (Amst) 2024; 251:104631. [PMID: 39622149 DOI: 10.1016/j.actpsy.2024.104631] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2024] [Revised: 11/28/2024] [Accepted: 11/28/2024] [Indexed: 12/16/2024] Open
Abstract
People can perceive various information from faces. Most of previous studies of face perception only focused on one of attributes, such as gender, expression, personality etc., the whole picture of face perception is far from clear. Therefore, the present study recruited Chinese participants to provide spontaneous descriptions of unfamiliar Chinese faces without any constraints of content. It turned out that descriptions employed a broad spectrum of descriptors, as well as a consistent pattern across different identities: descriptions that incorporated psychological characteristics were most prevalent, whereas mentions of physiological attributes generally occurred earlier in the description than other types of descriptive vocabulary. These results underscore the special role of free description analysis in revealing the panorama of face perception, where perceivers swiftly infer a plenty of character traits in an organized way, ultimately forming a comprehensive impression of others.
Collapse
Affiliation(s)
- Yangtao Liu
- School of Psychology, Liaoning Normal University, Dalian, China
| | - Dong Li
- School of Psychology, Liaoning Normal University, Dalian, China
| | - Wenbo Wang
- School of Psychology, Liaoning Normal University, Dalian, China
| | - Zhongqing Jiang
- School of Psychology, Liaoning Normal University, Dalian, China.
| |
Collapse
|
16
|
Teng S, Cichy R, Pantazis D, Oliva A. Touch to text: Spatiotemporal evolution of braille letter representations in blind readers. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.10.30.620429. [PMID: 39553970 PMCID: PMC11565808 DOI: 10.1101/2024.10.30.620429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 11/19/2024]
Abstract
Visual deprivation does not silence the visual cortex, which is responsive to auditory, tactile, and other nonvisual tasks in blind persons. However, the underlying functional dynamics of the neural networks mediating such crossmodal responses remain unclear. Here, using braille reading as a model framework to investigate these networks, we presented sighted (N=13) and blind (N=12) readers with individual visual print and tactile braille alphabetic letters, respectively, during MEG recording. Using time-resolved multivariate pattern analysis and representational similarity analysis, we traced the alphabetic letter processing cascade in both groups of participants. We found that letter representations unfolded more slowly in blind than in sighted brains, with decoding peak latencies ~200 ms later in braille readers. Focusing on the blind group, we found that the format of neural letter representations transformed within the first 500 ms after stimulus onset from a low-level structure consistent with peripheral nerve afferent coding to high-level format reflecting pairwise letter embeddings in a text corpus. The spatiotemporal dynamics of the transformation suggest that the processing cascade proceeds from a starting point in somatosensory cortex to early visual cortex and then to inferotemporal cortex. Together our results give insight into the neural mechanisms underlying braille reading in blind persons and the dynamics of functional reorganization in sensory deprivation.
Collapse
Affiliation(s)
- Santani Teng
- The Smith-Kettlewell Eye Research Institute
- Computer Science and Artificial Intelligence Laboratory, MIT
| | - Radoslaw Cichy
- Department of Education and Psychology, Freie Universität Berlin
| | | | - Aude Oliva
- Computer Science and Artificial Intelligence Laboratory, MIT
| |
Collapse
|
17
|
Wiese H, Popova T, Lidborg LH, Burton AM. The temporal dynamics of familiar face recognition: Event-related brain potentials reveal the efficient activation of facial identity representations. Int J Psychophysiol 2024; 204:112423. [PMID: 39168164 DOI: 10.1016/j.ijpsycho.2024.112423] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2024] [Revised: 08/14/2024] [Accepted: 08/17/2024] [Indexed: 08/23/2024]
Abstract
While it is widely known that humans are typically highly accurate at recognizing familiar faces, it is less clear how efficiently recognition is achieved. In a series of three experiments, we used event-related brain potentials (ERP) in a repetition priming paradigm to examine the efficiency of familiar face recognition. Specifically, we varied the presentation time of the prime stimulus between 500 ms and 33 ms (Experiments 1 and 2), and additionally used backward masks (Experiment 3) to prevent the potential occurrence of visual aftereffects. Crucially, to test for the recognition of facial identity rather than a specific picture, we used different images of the same facial identities in repetition conditions. We observed clear ERP repetition priming effects between 300 and 500 ms after target onset at all prime durations, which suggests that the prime stimulus was sufficiently well processed to allow for facilitated recognition of the target in all conditions. This finding held true even in severely restricted viewing conditions including very brief prime durations and backward masks. We conclude that the facial recognition system is both highly effective and efficient, thus allowing for our impressive ability to recognise the faces that we know.
Collapse
Affiliation(s)
| | | | | | - A Mike Burton
- University of York, United Kingdom; Bond University, Australia
| |
Collapse
|
18
|
Liu M, Zhan J, Wang L. Specified functions of the first two fixations in face recognition: Sampling the general-to-specific facial information. iScience 2024; 27:110686. [PMID: 39246447 PMCID: PMC11378928 DOI: 10.1016/j.isci.2024.110686] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 06/14/2024] [Accepted: 08/05/2024] [Indexed: 09/10/2024] Open
Abstract
Visual perception is enacted and constrained by the constantly moving eyes. Although it is well known that the first two fixations are crucial for face recognition, the function of each fixation remains unspecified. Here we demonstrate a central-to-divergent pattern of the two fixations and specify their functions: Fix I clustered along the nose bridge to cover the broad facial information; Fix II diverged to eyes, nostrils, and lips to get the local information. Fix II correlated more than Fix I with the differentiating information between faces and contributed more to recognition responses. While face categories can be significantly discriminated by Fix II's but not Fix I's patterns alone, the combined patterns of the two yield better discrimination. Our results suggest a functional division and collaboration of the two fixations in sampling the general-to-specific facial information and add to understanding visual perception as an active process undertaken by structural motor programs.
Collapse
Affiliation(s)
- Meng Liu
- Institute of Psychology and Behavioral Science, Antai College of Economics and Management, Shanghai Jiao Tong University, Shanghai 200030, China
- School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China
- Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai 200030, China
| | - Jiayu Zhan
- School of Psychological and Cognitive Sciences, Peking University, Beijing 100871, China
- Institute for Artificial Intelligence, Peking University, Beijing 100871, China
- State Key Laboratory of General Artificial Intelligence (BIGAI), Beijing 100871, China
| | - Lihui Wang
- Institute of Psychology and Behavioral Science, Antai College of Economics and Management, Shanghai Jiao Tong University, Shanghai 200030, China
- School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China
- Shanghai Mental Health Center, Shanghai Jiao Tong University School of Medicine, Shanghai 200030, China
| |
Collapse
|
19
|
Lifanov-Carr J, Griffiths BJ, Linde-Domingo J, Ferreira CS, Wilson M, Mayhew SD, Charest I, Wimber M. Reconstructing Spatiotemporal Trajectories of Visual Object Memories in the Human Brain. eNeuro 2024; 11:ENEURO.0091-24.2024. [PMID: 39242212 PMCID: PMC11439564 DOI: 10.1523/eneuro.0091-24.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 07/03/2024] [Accepted: 08/09/2024] [Indexed: 09/09/2024] Open
Abstract
How the human brain reconstructs, step-by-step, the core elements of past experiences is still unclear. Here, we map the spatiotemporal trajectories along which visual object memories are reconstructed during associative recall. Specifically, we inquire whether retrieval reinstates feature representations in a copy-like but reversed direction with respect to the initial perceptual experience, or alternatively, this reconstruction involves format transformations and regions beyond initial perception. Participants from two cohorts studied new associations between verbs and randomly paired object images, and subsequently recalled the objects when presented with the corresponding verb cue. We first analyze multivariate fMRI patterns to map where in the brain high- and low-level object features can be decoded during perception and retrieval, showing that retrieval is dominated by conceptual features, represented in comparatively late visual and parietal areas. A separately acquired EEG dataset is then used to track the temporal evolution of the reactivated patterns using similarity-based EEG-fMRI fusion. This fusion suggests that memory reconstruction proceeds from anterior frontotemporal to posterior occipital and parietal regions, in line with a conceptual-to-perceptual gradient but only partly following the same trajectories as during perception. Specifically, a linear regression statistically confirms that the sequential activation of ventral visual stream regions is reversed between image perception and retrieval. The fusion analysis also suggests an information relay to frontoparietal areas late during retrieval. Together, the results shed light onto the temporal dynamics of memory recall and the transformations that the information undergoes between the initial experience and its later reconstruction from memory.
Collapse
Affiliation(s)
- Julia Lifanov-Carr
- School of Psychology and Centre for Human Brain Health (CHBH), University of Birmingham, Birmingham B15 2TT, United Kingdom
| | - Benjamin J Griffiths
- School of Psychology and Centre for Human Brain Health (CHBH), University of Birmingham, Birmingham B15 2TT, United Kingdom
| | - Juan Linde-Domingo
- School of Psychology and Centre for Human Brain Health (CHBH), University of Birmingham, Birmingham B15 2TT, United Kingdom
- Department of Experimental Psychology, Mind, Brain and Behavior Research Center (CIMCYC), University of Granada, 18011 Granada, Spain
- Center for Adaptive Rationality, Max Planck Institute for Human Development, 14195 Berlin, Germany
| | - Catarina S Ferreira
- School of Psychology and Centre for Human Brain Health (CHBH), University of Birmingham, Birmingham B15 2TT, United Kingdom
| | - Martin Wilson
- School of Psychology and Centre for Human Brain Health (CHBH), University of Birmingham, Birmingham B15 2TT, United Kingdom
| | - Stephen D Mayhew
- Institute of Health and Neurodevelopment (IHN), School of Psychology, Aston University, Birmingham B4 7ET, United Kingdom
| | - Ian Charest
- Département de Psychologie, Université de Montréal, Montréal, Quebec H2V 2S9, Canada
| | - Maria Wimber
- School of Psychology and Centre for Human Brain Health (CHBH), University of Birmingham, Birmingham B15 2TT, United Kingdom
- School of Psychology & Neuroscience and Centre for Cognitive Neuroimaging (CCNi), University of Glasgow, Glasgow G12 8QB, United Kingdom
| |
Collapse
|
20
|
Liu X, Wei S, Zhao X, Bi Y, Hu L. Establishing the relationship between subjective perception and neural responses: Insights from correlation analysis and representational similarity analysis. Neuroimage 2024; 295:120650. [PMID: 38768740 DOI: 10.1016/j.neuroimage.2024.120650] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2024] [Revised: 05/12/2024] [Accepted: 05/17/2024] [Indexed: 05/22/2024] Open
Abstract
Exploring the relationship between sensory perception and brain responses holds important theoretical and clinical implications. However, commonly used methodologies like correlation analysis performed either intra- or inter- individually often yield inconsistent results across studies, limiting their generalizability. Representational similarity analysis (RSA), a method that assesses the perception-response relationship by calculating the correlation between behavioral and neural patterns, may offer a fresh perspective to reveal novel findings. Here, we delivered a series of graded sensory stimuli of four modalities (i.e., nociceptive somatosensory, non-nociceptive somatosensory, visual, and auditory) to/near the left or right hand of 107 healthy subjects and collected their single-trial perceptual ratings and electroencephalographic (EEG) responses. We examined the relationship between sensory perception and brain responses using within- and between-subject correlation analysis and RSA, and assessed their stability across different numbers of subjects and trials. We found that within-subject and between-subject correlations yielded distinct results: within-subject correlation revealed strong and reliable correlations between perceptual ratings and most brain responses, while between-subject correlation showed weak correlations that were vulnerable to the change of subject number. In addition to verifying the correlation results, RSA revealed some novel findings, i.e., correlations between behavioral and neural patterns were observed in some additional neural responses, such as "γ-ERS" in the visual modality. RSA results were sensitive to the trial number, but not to the subject number, suggesting that consistent results could be obtained for studies with relatively small sample sizes. In conclusion, our study provides a novel perspective on establishing the relationship between behavior and brain activity, emphasizing that RSA holds promise as a method for exploring this pattern relationship in future research.
Collapse
Affiliation(s)
- Xu Liu
- CAS Key Laboratory of Mental Health, Institute of Psychology, Beijing, 100101, China; Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian, 116029, China
| | - Shiyu Wei
- CAS Key Laboratory of Mental Health, Institute of Psychology, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Xiangyue Zhao
- CAS Key Laboratory of Mental Health, Institute of Psychology, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China
| | - Yanzhi Bi
- CAS Key Laboratory of Mental Health, Institute of Psychology, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Li Hu
- CAS Key Laboratory of Mental Health, Institute of Psychology, Beijing, 100101, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, 100049, China
| |
Collapse
|
21
|
Quaia C, Krauzlis RJ. Object recognition in primates: what can early visual areas contribute? Front Behav Neurosci 2024; 18:1425496. [PMID: 39070778 PMCID: PMC11272660 DOI: 10.3389/fnbeh.2024.1425496] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2024] [Accepted: 07/01/2024] [Indexed: 07/30/2024] Open
Abstract
Introduction If neuroscientists were asked which brain area is responsible for object recognition in primates, most would probably answer infero-temporal (IT) cortex. While IT is likely responsible for fine discriminations, and it is accordingly dominated by foveal visual inputs, there is more to object recognition than fine discrimination. Importantly, foveation of an object of interest usually requires recognizing, with reasonable confidence, its presence in the periphery. Arguably, IT plays a secondary role in such peripheral recognition, and other visual areas might instead be more critical. Methods To investigate how signals carried by early visual processing areas (such as LGN and V1) could be used for object recognition in the periphery, we focused here on the task of distinguishing faces from non-faces. We tested how sensitive various models were to nuisance parameters, such as changes in scale and orientation of the image, and the type of image background. Results We found that a model of V1 simple or complex cells could provide quite reliable information, resulting in performance better than 80% in realistic scenarios. An LGN model performed considerably worse. Discussion Because peripheral recognition is both crucial to enable fine recognition (by bringing an object of interest on the fovea), and probably sufficient to account for a considerable fraction of our daily recognition-guided behavior, we think that the current focus on area IT and foveal processing is too narrow. We propose that rather than a hierarchical system with IT-like properties as its primary aim, object recognition should be seen as a parallel process, with high-accuracy foveal modules operating in parallel with lower-accuracy and faster modules that can operate across the visual field.
Collapse
Affiliation(s)
- Christian Quaia
- Laboratory of Sensorimotor Research, National Eye Institute, NIH, Bethesda, MD, United States
| | | |
Collapse
|
22
|
Quaia C, Krauzlis RJ. Object recognition in primates: What can early visual areas contribute? ARXIV 2024:arXiv:2407.04816v1. [PMID: 39398202 PMCID: PMC11468158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Subscribe] [Scholar Register] [Indexed: 10/15/2024]
Abstract
If neuroscientists were asked which brain area is responsible for object recognition in primates, most would probably answer infero-temporal (IT) cortex. While IT is likely responsible for fine discriminations, and it is accordingly dominated by foveal visual inputs, there is more to object recognition than fine discrimination. Importantly, foveation of an object of interest usually requires recognizing, with reasonable confidence, its presence in the periphery. Arguably, IT plays a secondary role in such peripheral recognition, and other visual areas might instead be more critical. To investigate how signals carried by early visual processing areas (such as LGN and V1) could be used for object recognition in the periphery, we focused here on the task of distinguishing faces from non-faces. We tested how sensitive various models were to nuisance parameters, such as changes in scale and orientation of the image, and the type of image background. We found that a model of V1 simple or complex cells could provide quite reliable information, resulting in performance better than 80% in realistic scenarios. An LGN model performed considerably worse. Because peripheral recognition is both crucial to enable fine recognition (by bringing an object of interest on the fovea), and probably sufficient to account for a considerable fraction of our daily recognition-guided behavior, we think that the current focus on area IT and foveal processing is too narrow. We propose that rather than a hierarchical system with IT-like properties as its primary aim, object recognition should be seen as a parallel process, with high-accuracy foveal modules operating in parallel with lower-accuracy and faster modules that can operate across the visual field.
Collapse
Affiliation(s)
- Christian Quaia
- Laboratory of Sensorimotor Research, National Eye Institute, NIH, Bethesda, MD, USA
| | - Richard J Krauzlis
- Laboratory of Sensorimotor Research, National Eye Institute, NIH, Bethesda, MD, USA
| |
Collapse
|
23
|
Lavan N, Rinke P, Scharinger M. The time course of person perception from voices in the brain. Proc Natl Acad Sci U S A 2024; 121:e2318361121. [PMID: 38889147 PMCID: PMC11214051 DOI: 10.1073/pnas.2318361121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 04/26/2024] [Indexed: 06/20/2024] Open
Abstract
When listeners hear a voice, they rapidly form a complex first impression of who the person behind that voice might be. We characterize how these multivariate first impressions from voices emerge over time across different levels of abstraction using electroencephalography and representational similarity analysis. We find that for eight perceived physical (gender, age, and health), trait (attractiveness, dominance, and trustworthiness), and social characteristics (educatedness and professionalism), representations emerge early (~80 ms after stimulus onset), with voice acoustics contributing to those representations between ~100 ms and 400 ms. While impressions of person characteristics are highly correlated, we can find evidence for highly abstracted, independent representations of individual person characteristics. These abstracted representationse merge gradually over time. That is, representations of physical characteristics (age, gender) arise early (from ~120 ms), while representations of some trait and social characteristics emerge later (~360 ms onward). The findings align with recent theoretical models and shed light on the computations underpinning person perception from voices.
Collapse
Affiliation(s)
- Nadine Lavan
- Department of Biological and Experimental Psychology, School of Biological and Behavioural Sciences, Queen Mary University of London, LondonE1 4NS, United Kingdom
| | - Paula Rinke
- Research Group Phonetics, Institute of German Linguistics, Philipps-University Marburg, Marburg35037, Germany
| | - Mathias Scharinger
- Research Group Phonetics, Institute of German Linguistics, Philipps-University Marburg, Marburg35037, Germany
- Research Center “Deutscher Sprachatlas”, Philipps-University Marburg, Marburg35037, Germany
- Center for Mind, Brain & Behavior, Universities of Marburg & Gießen, Marburg35032, Germany
| |
Collapse
|
24
|
Koenig-Robert R, Quek GL, Grootswagers T, Varlet M. Movement trajectories as a window into the dynamics of emerging neural representations. Sci Rep 2024; 14:11499. [PMID: 38769313 PMCID: PMC11106280 DOI: 10.1038/s41598-024-62135-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Accepted: 05/14/2024] [Indexed: 05/22/2024] Open
Abstract
The rapid transformation of sensory inputs into meaningful neural representations is critical to adaptive human behaviour. While non-invasive neuroimaging methods are the de-facto method for investigating neural representations, they remain expensive, not widely available, time-consuming, and restrictive. Here we show that movement trajectories can be used to measure emerging neural representations with fine temporal resolution. By combining online computer mouse-tracking and publicly available neuroimaging data via representational similarity analysis (RSA), we show that movement trajectories track the unfolding of stimulus- and category-wise neural representations along key dimensions of the human visual system. We demonstrate that time-resolved representational structures derived from movement trajectories overlap with those derived from M/EEG (albeit delayed) and those derived from fMRI in functionally-relevant brain areas. Our findings highlight the richness of movement trajectories and the power of the RSA framework to reveal and compare their information content, opening new avenues to better understand human perception.
Collapse
Affiliation(s)
- Roger Koenig-Robert
- The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, 2751, Australia
- School of Psychology, University of New South Wales, Sydney, NSW, Australia
| | - Genevieve L Quek
- The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, 2751, Australia
| | - Tijl Grootswagers
- The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, 2751, Australia
- School of Computer, Data and Mathematical Sciences, Western Sydney University, Penrith, NSW, 2751, Australia
| | - Manuel Varlet
- The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, 2751, Australia.
- School of Psychology, Western Sydney University, Sydney, NSW, 2751, Australia.
| |
Collapse
|
25
|
Martinez JE, Oh D, Todorov A. Immigration documentation statuses evoke racialized faceism in mental representations. Sci Rep 2024; 14:10673. [PMID: 38724676 PMCID: PMC11082198 DOI: 10.1038/s41598-024-61203-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2023] [Accepted: 05/02/2024] [Indexed: 05/12/2024] Open
Abstract
U.S. immigration discourse has spurred interest in characterizing who illegalized immigrants are or perceived to be. What are the associated visual representations of migrant illegality? Across two studies with undergraduate and online samples (N = 686), we used face-based reverse correlation and similarity sorting to capture and compare mental representations of illegalized immigrants, native-born U.S. citizens, and documented immigrants. Documentation statuses evoked racialized imagery. Immigrant representations were dark-skinned and perceived as non-white, while citizen representations were light-skinned, evaluated positively, and perceived as white. Legality further differentiated immigrant representations: documentation conjured trustworthy representations, illegality conjured threatening representations. Participants spontaneously sorted unlabeled faces by documentation status in a spatial arrangement task. Faces' spatial similarity correlated with their similarity in pixel luminance and "American" ratings, confirming racialized distinctions. Representations of illegalized immigrants were uniquely racialized as dark-skinned un-American threats, reflecting how U.S. imperialism and colorism set conditions of possibility for existing representations of migrant illegalization.
Collapse
Affiliation(s)
- Joel E Martinez
- Data Science Initiative, Harvard University, Cambridge, MA, USA.
- Department of Psychology, Harvard University, Cambridge, MA, USA.
| | - DongWon Oh
- Department of Psychology, National University of Singapore, Singapore, Singapore
| | - Alexander Todorov
- Booth School of Business, The University of Chicago, Chicago, IL, USA
| |
Collapse
|
26
|
Xu W, Li X, Parviainen T, Nokia M. Neural correlates of retrospective memory confidence during face-name associative learning. Cereb Cortex 2024; 34:bhae194. [PMID: 38801420 PMCID: PMC11411154 DOI: 10.1093/cercor/bhae194] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Revised: 04/18/2024] [Accepted: 04/23/2024] [Indexed: 05/29/2024] Open
Abstract
The ability to accurately assess one's own memory performance during learning is essential for adaptive behavior, but the brain mechanisms underlying this metamemory function are not well understood. We investigated the neural correlates of memory accuracy and retrospective memory confidence in a face-name associative learning task using magnetoencephalography in healthy young adults (n = 32). We found that high retrospective confidence was associated with stronger occipital event-related fields during encoding and widespread event-related fields during retrieval compared to low confidence. On the other hand, memory accuracy was linked to medial temporal activities during both encoding and retrieval, but only in low-confidence trials. A decrease in oscillatory power at alpha/beta bands in the parietal regions during retrieval was associated with higher memory confidence. In addition, representational similarity analysis at the single-trial level revealed distributed but differentiable neural activities associated with memory accuracy and confidence during both encoding and retrieval. In summary, our study unveiled distinct neural activity patterns related to memory confidence and accuracy during associative learning and underscored the crucial role of parietal regions in metamemory.
Collapse
Affiliation(s)
- Weiyong Xu
- Department of Psychology, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
- Jyväskylä Centre for Interdisciplinary Brain Research, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
| | - Xueqiao Li
- Department of Psychology, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
- Jyväskylä Centre for Interdisciplinary Brain Research, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
| | - Tiina Parviainen
- Department of Psychology, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
- Jyväskylä Centre for Interdisciplinary Brain Research, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
| | - Miriam Nokia
- Department of Psychology, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
- Jyväskylä Centre for Interdisciplinary Brain Research, University of Jyväskylä, Mattilanniemi 6, 40014, Jyväskylä, Finland
| |
Collapse
|
27
|
Garlichs A, Blank H. Prediction error processing and sharpening of expected information across the face-processing hierarchy. Nat Commun 2024; 15:3407. [PMID: 38649694 PMCID: PMC11035707 DOI: 10.1038/s41467-024-47749-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Accepted: 04/10/2024] [Indexed: 04/25/2024] Open
Abstract
The perception and neural processing of sensory information are strongly influenced by prior expectations. The integration of prior and sensory information can manifest through distinct underlying mechanisms: focusing on unexpected input, denoted as prediction error (PE) processing, or amplifying anticipated information via sharpened representation. In this study, we employed computational modeling using deep neural networks combined with representational similarity analyses of fMRI data to investigate these two processes during face perception. Participants were cued to see face images, some generated by morphing two faces, leading to ambiguity in face identity. We show that expected faces were identified faster and perception of ambiguous faces was shifted towards priors. Multivariate analyses uncovered evidence for PE processing across and beyond the face-processing hierarchy from the occipital face area (OFA), via the fusiform face area, to the anterior temporal lobe, and suggest sharpened representations in the OFA. Our findings support the proposition that the brain represents faces grounded in prior expectations.
Collapse
Affiliation(s)
- Annika Garlichs
- Department of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, 20246, Hamburg, Germany.
| | - Helen Blank
- Department of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, 20246, Hamburg, Germany.
| |
Collapse
|
28
|
Li Y, Li S, Hu W, Yang L, Luo W. Spatial representation of multidimensional information in emotional faces revealed by fMRI. Neuroimage 2024; 290:120578. [PMID: 38499051 DOI: 10.1016/j.neuroimage.2024.120578] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2023] [Revised: 03/13/2024] [Accepted: 03/15/2024] [Indexed: 03/20/2024] Open
Abstract
Face perception is a complex process that involves highly specialized procedures and mechanisms. Investigating into face perception can help us better understand how the brain processes fine-grained, multidimensional information. This research aimed to delve deeply into how different dimensions of facial information are represented in specific brain regions or through inter-regional connections via an implicit face recognition task. To capture the representation of various facial information in the brain, we employed support vector machine decoding, functional connectivity, and model-based representational similarity analysis on fMRI data, resulting in the identification of three crucial findings. Firstly, despite the implicit nature of the task, emotions were still represented in the brain, contrasting with all other facial information. Secondly, the connection between the medial amygdala and the parahippocampal gyrus was found to be essential for the representation of facial emotion in implicit tasks. Thirdly, in implicit tasks, arousal representation occurred in the parahippocampal gyrus, while valence depended on the connection between the primary visual cortex and the parahippocampal gyrus. In conclusion, these findings dissociate the neural mechanisms of emotional valence and arousal, revealing the precise spatial patterns of multidimensional information processing in faces.
Collapse
Affiliation(s)
- Yiwen Li
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, PR China; Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian 116029, PR China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, Dalian 116029, PR China
| | - Shuaixia Li
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian 116029, PR China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, Dalian 116029, PR China
| | - Weiyu Hu
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, PR China
| | - Lan Yang
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian 116029, PR China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, Dalian 116029, PR China
| | - Wenbo Luo
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, Dalian 116029, PR China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, Dalian 116029, PR China.
| |
Collapse
|
29
|
Li W, Li J, Chu C, Cao D, Shi W, Zhang Y, Jiang T. Common Sequential Organization of Face Processing in the Human Brain and Convolutional Neural Networks. Neuroscience 2024; 541:1-13. [PMID: 38266906 DOI: 10.1016/j.neuroscience.2024.01.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 01/11/2024] [Accepted: 01/16/2024] [Indexed: 01/26/2024]
Abstract
Face processing includes two crucial processing levels - face detection and face recognition. However, it remains unclear how human brains organize the two processing levels sequentially. While some studies found that faces are recognized as fast as they are detected, others have reported that faces are detected first, followed by recognition. We discriminated the two processing levels on a fine time scale by combining human intracranial EEG (two females, three males, and three subjects without reported sex information) and representation similarity analysis. Our results demonstrate that the human brain exhibits a "detection-first, recognition-later" pattern during face processing. In addition, we used convolutional neural networks to test the hypothesis that the sequential organization of the two face processing levels in the brain reflects computational optimization. Our findings showed that the networks trained on face recognition also exhibited the "detection-first, recognition-later" pattern. Moreover, this sequential organization mechanism developed gradually during the training of the networks and was observed only for correctly predicted images. These findings collectively support the computational account as to why the brain organizes them in this way.
Collapse
Affiliation(s)
- Wenlu Li
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Jin Li
- School of Psychology, Capital Normal University, Beijing 100048, China
| | - Congying Chu
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Dan Cao
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Weiyang Shi
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
| | - Yu Zhang
- Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 311100, China
| | - Tianzi Jiang
- Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China; School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China; Research Center for Augmented Intelligence, Zhejiang Lab, Hangzhou 311100, China; Xiaoxiang Institute for Brain Health and Yongzhou Central Hospital, Yongzhou 425000, Hunan Province, China.
| |
Collapse
|
30
|
Sama MA, Nestor A, Cant JS. The Neural Dynamics of Face Ensemble and Central Face Processing. J Neurosci 2024; 44:e1027232023. [PMID: 38148151 PMCID: PMC10869155 DOI: 10.1523/jneurosci.1027-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Revised: 11/21/2023] [Accepted: 12/11/2023] [Indexed: 12/28/2023] Open
Abstract
Extensive work has investigated the neural processing of single faces, including the role of shape and surface properties. However, much less is known about the neural basis of face ensemble perception (e.g., simultaneously viewing several faces in a crowd). Importantly, the contribution of shape and surface properties have not been elucidated in face ensemble processing. Furthermore, how single central faces are processed within the context of an ensemble remains unclear. Here, we probe the neural dynamics of ensemble representation using pattern analyses as applied to electrophysiology data in healthy adults (seven males, nine females). Our investigation relies on a unique set of stimuli, depicting different facial identities, which vary parametrically and independently along their shape and surface properties. These stimuli were organized into ensemble displays consisting of six surround faces arranged in a circle around one central face. Overall, our results indicate that both shape and surface properties play a significant role in face ensemble encoding, with the latter demonstrating a more pronounced contribution. Importantly, we find that the neural processing of the center face precedes that of the surround faces in an ensemble. Further, the temporal profile of center face decoding is similar to that of single faces, while those of single faces and face ensembles diverge extensively from each other. Thus, our work capitalizes on a new center-surround paradigm to elucidate the neural dynamics of ensemble processing and the information that underpins it. Critically, our results serve to bridge the study of single and ensemble face perception.
Collapse
Affiliation(s)
- Marco Agazio Sama
- Department of Psychology, University of Toronto Scarborough, Toronto, Ontario M1C 1A4, Canada
| | - Adrian Nestor
- Department of Psychology, University of Toronto Scarborough, Toronto, Ontario M1C 1A4, Canada
| | - Jonathan Samuel Cant
- Department of Psychology, University of Toronto Scarborough, Toronto, Ontario M1C 1A4, Canada
| |
Collapse
|
31
|
Cao R, Wang J, Brunner P, Willie JT, Li X, Rutishauser U, Brandmeir NJ, Wang S. Neural mechanisms of face familiarity and learning in the human amygdala and hippocampus. Cell Rep 2024; 43:113520. [PMID: 38151023 PMCID: PMC10834150 DOI: 10.1016/j.celrep.2023.113520] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 09/12/2023] [Accepted: 11/14/2023] [Indexed: 12/29/2023] Open
Abstract
Recognizing familiar faces and learning new faces play an important role in social cognition. However, the underlying neural computational mechanisms remain unclear. Here, we record from single neurons in the human amygdala and hippocampus and find a greater neuronal representational distance between pairs of familiar faces than unfamiliar faces, suggesting that neural representations for familiar faces are more distinct. Representational distance increases with exposures to the same identity, suggesting that neural face representations are sharpened with learning and familiarization. Furthermore, representational distance is positively correlated with visual dissimilarity between faces, and exposure to visually similar faces increases representational distance, thus sharpening neural representations. Finally, we construct a computational model that demonstrates an increase in the representational distance of artificial units with training. Together, our results suggest that the neuronal population geometry, quantified by the representational distance, encodes face familiarity, similarity, and learning, forming the basis of face recognition and memory.
Collapse
Affiliation(s)
- Runnan Cao
- Department of Radiology, Washington University in St. Louis, St. Louis, MO 63110, USA; Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA.
| | - Jinge Wang
- Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA
| | - Peter Brunner
- Department of Neurosurgery, Washington University in St. Louis, St. Louis, MO 63110, USA
| | - Jon T Willie
- Department of Neurosurgery, Washington University in St. Louis, St. Louis, MO 63110, USA
| | - Xin Li
- Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA
| | - Ueli Rutishauser
- Departments of Neurosurgery and Neurology, Cedars-Sinai Medical Center, Los Angeles, CA 90048, USA
| | | | - Shuo Wang
- Department of Radiology, Washington University in St. Louis, St. Louis, MO 63110, USA; Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA; Department of Neurosurgery, Washington University in St. Louis, St. Louis, MO 63110, USA.
| |
Collapse
|
32
|
Lu Z, Ku Y. Bridging the gap between EEG and DCNNs reveals a fatigue mechanism of facial repetition suppression. iScience 2023; 26:108501. [PMID: 38089588 PMCID: PMC10711494 DOI: 10.1016/j.isci.2023.108501] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Revised: 10/26/2023] [Accepted: 11/17/2023] [Indexed: 08/05/2024] Open
Abstract
Facial repetition suppression, a well-studied phenomenon characterized by decreased neural responses to repeated faces in visual cortices, remains a subject of ongoing debate regarding its underlying neural mechanisms. Our research harnesses advanced multivariate analysis techniques and the prowess of deep convolutional neural networks (DCNNs) in face recognition to bridge the gap between human electroencephalogram (EEG) data and DCNNs, especially in the context of facial repetition suppression. Our innovative reverse engineering approach, manipulating the neuronal activity in DCNNs and conducted representational comparisons between brain activations derived from human EEG and manipulated DCNN activations, provided insights into the underlying facial repetition suppression. Significantly, our findings advocate the fatigue mechanism as the dominant force behind the facial repetition suppression effect. Broadly, this integrative framework, bridging the human brain and DCNNs, offers a promising tool for simulating brain activity and making inferences regarding the neural mechanisms underpinning complex human behaviors.
Collapse
Affiliation(s)
- Zitong Lu
- Department of Psychology, The Ohio State University, Columbus, OH, USA
| | - Yixuan Ku
- Guangdong Provincial Key Laboratory of Brain Function and Disease, Center for Brain and Mental Well-being, Department of Psychology, Sun Yat-sen University, Guangzhou, China
- Peng Cheng Laboratory, Shenzhen, China
| |
Collapse
|
33
|
Klink H, Kaiser D, Stecher R, Ambrus GG, Kovács G. Your place or mine? The neural dynamics of personally familiar scene recognition suggests category independent familiarity encoding. Cereb Cortex 2023; 33:11634-11645. [PMID: 37885126 DOI: 10.1093/cercor/bhad397] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 09/29/2023] [Accepted: 09/30/2023] [Indexed: 10/28/2023] Open
Abstract
Recognizing a stimulus as familiar is an important capacity in our everyday life. Recent investigation of visual processes has led to important insights into the nature of the neural representations of familiarity for human faces. Still, little is known about how familiarity affects the neural dynamics of non-face stimulus processing. Here we report the results of an EEG study, examining the representational dynamics of personally familiar scenes. Participants viewed highly variable images of their own apartments and unfamiliar ones, as well as personally familiar and unfamiliar faces. Multivariate pattern analyses were used to examine the time course of differential processing of familiar and unfamiliar stimuli. Time-resolved classification revealed that familiarity is decodable from the EEG data similarly for scenes and faces. The temporal dynamics showed delayed onsets and peaks for scenes as compared to faces. Familiarity information, starting at 200 ms, generalized across stimulus categories and led to a robust familiarity effect. In addition, familiarity enhanced category representations in early (250-300 ms) and later (>400 ms) processing stages. Our results extend previous face familiarity results to another stimulus category and suggest that familiarity as a construct can be understood as a general, stimulus-independent processing step during recognition.
Collapse
Affiliation(s)
- Hannah Klink
- Department of Neurology, Universitätsklinikum, Kastanienstraße1 Jena, D-07747 Jena, Thüringen, Germany
- Department of Biological Psychology and Cognitive Neurosciences, Institute of Psychology, Friedrich Schiller University Jena, Leutragraben 1, D-07743 Jena, Thüringen, Germany
| | - Daniel Kaiser
- Mathematical Institute, Department of Mathematics and Computer Science, Physics, Geography, Justus-Liebig-University Gießen, Arndtstraße 2, D-35392 Gießen, Hessen, Germany
- Center for Mind, Brain and Behavior (CMBB), Justus-Liebig-University Gießen and Philipps-University Marburg, Hans-Meerwein-Straße 6 Mehrzweckgeb, 03C022, Marburg, D-35032, Hessen, Germany
| | - Rico Stecher
- Mathematical Institute, Department of Mathematics and Computer Science, Physics, Geography, Justus-Liebig-University Gießen, Arndtstraße 2, D-35392 Gießen, Hessen, Germany
| | - Géza G Ambrus
- Department of Psychology, Bournemouth University, Poole House P319, Talbot Campus, Fern Barrow, Poole, Dorset BH12 5BB, United Kingdom
| | - Gyula Kovács
- Department of Biological Psychology and Cognitive Neurosciences, Institute of Psychology, Friedrich Schiller University Jena, Leutragraben 1, D-07743 Jena, Thüringen, Germany
| |
Collapse
|
34
|
Yao L, Fu Q, Liu CH. The roles of edge-based and surface-based information in the dynamic neural representation of objects. Neuroimage 2023; 283:120425. [PMID: 37890562 DOI: 10.1016/j.neuroimage.2023.120425] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Revised: 10/22/2023] [Accepted: 10/24/2023] [Indexed: 10/29/2023] Open
Abstract
We combined multivariate pattern analysis (MVPA) and electroencephalogram (EEG) to investigate the role of edge, color, and other surface information in the neural representation of visual objects. Participants completed a one-back task in which they were presented with color photographs, grayscale images, and line drawings of animals, tools, and fruits. Our results provide the first neural evidence that line drawings elicit similar neural activities as color photographs and grayscale images during the 175-305 ms window after the stimulus onset. Furthermore, we found that other surface information, rather than color information, facilitates decoding accuracy in the early stages of object representations and affects the speed of this. These results provide new insights into the role of edge-based and surface-based information in the dynamic process of neural representations of visual objects.
Collapse
Affiliation(s)
- Liansheng Yao
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Qiufang Fu
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, China; Department of Psychology, University of Chinese Academy of Sciences, Beijing, China.
| | - Chang Hong Liu
- Department of Psychology, Bournemouth University, Fern Barrow, Poole, UK
| |
Collapse
|
35
|
Hester N, Xie SY, Bertin JA, Hehman E. Stereotypes shape response competition when forming impressions. GROUP PROCESSES & INTERGROUP RELATIONS 2023; 26:1706-1725. [PMID: 38021317 PMCID: PMC10665134 DOI: 10.1177/13684302221129429] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2021] [Accepted: 08/29/2022] [Indexed: 12/01/2023]
Abstract
Dynamic models of impression formation posit that bottom-up factors (e.g., a target's facial features) and top-down factors (e.g., perceiver knowledge of stereotypes) continuously interact over time until a stable categorization or impression emerges. Most previous work on the dynamic resolution of judgments over time has focused on either categorization (e.g., "is this person male/female?") or specific trait impressions (e.g., "is this person trustworthy?"). In two mousetracking studies-exploratory (N = 226) and confirmatory (N = 300)-we test a domain-general effect of cultural stereotypes shaping the process underlying impressions of targets. We find that the trajectories of participants' mouse movements gravitate toward impressions congruent with their stereotype knowledge. For example, to the extent that a participant reports knowledge of a "Black men are less [trait]" stereotype, their mouse trajectory initially gravitates toward categorizing individual Black male faces as "less [trait]," regardless of their final judgment of the target.
Collapse
Affiliation(s)
- Neil Hester
- Department of Psychology, McGill University, Canada
| | - Sally Y. Xie
- Department of Psychology, McGill University, Canada
| | | | - Eric Hehman
- Department of Psychology, McGill University, Canada
| |
Collapse
|
36
|
Quian Quiroga R, Boscaglia M, Jonas J, Rey HG, Yan X, Maillard L, Colnat-Coulbois S, Koessler L, Rossion B. Single neuron responses underlying face recognition in the human midfusiform face-selective cortex. Nat Commun 2023; 14:5661. [PMID: 37704636 PMCID: PMC10499913 DOI: 10.1038/s41467-023-41323-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Accepted: 08/28/2023] [Indexed: 09/15/2023] Open
Abstract
Faces are critical for social interactions and their recognition constitutes one of the most important and challenging functions of the human brain. While neurons responding selectively to faces have been recorded for decades in the monkey brain, face-selective neural activations have been reported with neuroimaging primarily in the human midfusiform gyrus. Yet, the cellular mechanisms producing selective responses to faces in this hominoid neuroanatomical structure remain unknown. Here we report single neuron recordings performed in 5 human subjects (1 male, 4 females) implanted with intracerebral microelectrodes in the face-selective midfusiform gyrus, while they viewed pictures of familiar and unknown faces and places. We observed similar responses to faces and places at the single cell level, but a significantly higher number of neurons responding to faces, thus offering a mechanistic account for the face-selective activations observed in this region. Although individual neurons did not respond preferentially to familiar faces, a population level analysis could consistently determine whether or not the faces (but not the places) were familiar, only about 50 ms after the initial recognition of the stimuli as faces. These results provide insights into the neural mechanisms of face processing in the human brain.
Collapse
Affiliation(s)
- Rodrigo Quian Quiroga
- Hospital del Mar Research Institute (IMIM), Barcelona, Spain.
- Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain.
- Centre for Systems Neuroscience, University of Leicester, Leicester, UK.
- Ruijin hospital, Shanghai Jiao Tong university school of medicine, Shanghai, China.
| | - Marta Boscaglia
- Centre for Systems Neuroscience, University of Leicester, Leicester, UK
| | - Jacques Jonas
- Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France
- Université de Lorraine, CHRU-Nancy, Service de Neurologie, F-54000, Nancy, France
| | - Hernan G Rey
- Centre for Systems Neuroscience, University of Leicester, Leicester, UK
| | - Xiaoqian Yan
- Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France
| | - Louis Maillard
- Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France
- Université de Lorraine, CHRU-Nancy, Service de Neurologie, F-54000, Nancy, France
| | - Sophie Colnat-Coulbois
- Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France
- Université de Lorraine, CHRU-Nancy, Service de Neurochirurgie, F-54000, Nancy, France
| | - Laurent Koessler
- Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France
- Université de Lorraine, CHRU-Nancy, Service de Neurologie, F-54000, Nancy, France
| | - Bruno Rossion
- Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France.
- Université de Lorraine, CHRU-Nancy, Service de Neurologie, F-54000, Nancy, France.
| |
Collapse
|
37
|
Dobs K, Yuan J, Martinez J, Kanwisher N. Behavioral signatures of face perception emerge in deep neural networks optimized for face recognition. Proc Natl Acad Sci U S A 2023; 120:e2220642120. [PMID: 37523537 PMCID: PMC10410721 DOI: 10.1073/pnas.2220642120] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 06/08/2023] [Indexed: 08/02/2023] Open
Abstract
Human face recognition is highly accurate and exhibits a number of distinctive and well-documented behavioral "signatures" such as the use of a characteristic representational space, the disproportionate performance cost when stimuli are presented upside down, and the drop in accuracy for faces from races the participant is less familiar with. These and other phenomena have long been taken as evidence that face recognition is "special". But why does human face perception exhibit these properties in the first place? Here, we use deep convolutional neural networks (CNNs) to test the hypothesis that all of these signatures of human face perception result from optimization for the task of face recognition. Indeed, as predicted by this hypothesis, these phenomena are all found in CNNs trained on face recognition, but not in CNNs trained on object recognition, even when additionally trained to detect faces while matching the amount of face experience. To test whether these signatures are in principle specific to faces, we optimized a CNN on car discrimination and tested it on upright and inverted car images. As we found for face perception, the car-trained network showed a drop in performance for inverted vs. upright cars. Similarly, CNNs trained on inverted faces produced an inverted face inversion effect. These findings show that the behavioral signatures of human face perception reflect and are well explained as the result of optimization for the task of face recognition, and that the nature of the computations underlying this task may not be so special after all.
Collapse
Affiliation(s)
- Katharina Dobs
- Department of Psychology, Justus Liebig University Giessen, Giessen35394, Germany
- Center for Mind, Brain and Behavior (CMBB), University of Marburg and Justus Liebig University Giessen, Marburg35302, Germany
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA02139
| | - Joanne Yuan
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
| | - Julio Martinez
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
- Department of Psychology, Stanford University, Stanford, CA94305
| | - Nancy Kanwisher
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA02139
| |
Collapse
|
38
|
Popova T, Wiese H. Developing familiarity during the first eight months of knowing a person: A longitudinal EEG study on face and identity learning. Cortex 2023; 165:26-37. [PMID: 37245406 DOI: 10.1016/j.cortex.2023.04.008] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Revised: 02/03/2023] [Accepted: 04/23/2023] [Indexed: 05/30/2023]
Abstract
It is well-established that familiar and unfamiliar faces are processed differently, but surprisingly little is known about how familiarity builds up over time and how novel faces gradually become represented in the brain. Here, we used event-related brain potentials (ERPs) in a pre-registered, longitudinal study to examine the neural processes accompanying face and identity learning during the first eight months of knowing a person. Specifically, we examined how increasing real-life familiarity affects visual recognition (N250 Familiarity Effect) and the integration of person-related knowledge (Sustained Familiarity Effect, SFE). Sixteen first-year undergraduates were tested in three sessions, approximately one, five, and eight months after the start of the academic year, with highly variable "ambient" images of a new friend they had met at university and of an unfamiliar person. We observed clear ERP familiarity effects for the new friend after one month of familiarity. While there was an increase in the N250 effect over the course of the study, no change in the SFE was observed. These results suggest that visual face representations develop faster relative to the integration of identity-specific knowledge.
Collapse
|
39
|
Lavan N, McGettigan C. A model for person perception from familiar and unfamiliar voices. COMMUNICATIONS PSYCHOLOGY 2023; 1:1. [PMID: 38665246 PMCID: PMC11041786 DOI: 10.1038/s44271-023-00001-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Accepted: 04/28/2023] [Indexed: 04/28/2024]
Abstract
When hearing a voice, listeners can form a detailed impression of the person behind the voice. Existing models of voice processing focus primarily on one aspect of person perception - identity recognition from familiar voices - but do not account for the perception of other person characteristics (e.g., sex, age, personality traits). Here, we present a broader perspective, proposing that listeners have a common perceptual goal of perceiving who they are hearing, whether the voice is familiar or unfamiliar. We outline and discuss a model - the Person Perception from Voices (PPV) model - that achieves this goal via a common mechanism of recognising a familiar person, persona, or set of speaker characteristics. Our PPV model aims to provide a more comprehensive account of how listeners perceive the person they are listening to, using an approach that incorporates and builds on aspects of the hierarchical frameworks and prototype-based mechanisms proposed within existing models of voice identity recognition.
Collapse
Affiliation(s)
- Nadine Lavan
- Department of Experimental and Biological Psychology, Queen Mary University of London, London, UK
| | - Carolyn McGettigan
- Department of Speech, Hearing, and Phonetic Sciences, University College London, London, UK
| |
Collapse
|
40
|
Celeghin A, Borriero A, Orsenigo D, Diano M, Méndez Guerrero CA, Perotti A, Petri G, Tamietto M. Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues. Front Comput Neurosci 2023; 17:1153572. [PMID: 37485400 PMCID: PMC10359983 DOI: 10.3389/fncom.2023.1153572] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2023] [Accepted: 06/19/2023] [Indexed: 07/25/2023] Open
Abstract
Convolutional Neural Networks (CNN) are a class of machine learning models predominately used in computer vision tasks and can achieve human-like performance through learning from experience. Their striking similarities to the structural and functional principles of the primate visual system allow for comparisons between these artificial networks and their biological counterparts, enabling exploration of how visual functions and neural representations may emerge in the real brain from a limited set of computational principles. After considering the basic features of CNNs, we discuss the opportunities and challenges of endorsing CNNs as in silico models of the primate visual system. Specifically, we highlight several emerging notions about the anatomical and physiological properties of the visual system that still need to be systematically integrated into current CNN models. These tenets include the implementation of parallel processing pathways from the early stages of retinal input and the reconsideration of several assumptions concerning the serial progression of information flow. We suggest design choices and architectural constraints that could facilitate a closer alignment with biology provide causal evidence of the predictive link between the artificial and biological visual systems. Adopting this principled perspective could potentially lead to new research questions and applications of CNNs beyond modeling object recognition.
Collapse
Affiliation(s)
| | | | - Davide Orsenigo
- Department of Psychology, University of Torino, Turin, Italy
| | - Matteo Diano
- Department of Psychology, University of Torino, Turin, Italy
| | | | | | | | - Marco Tamietto
- Department of Psychology, University of Torino, Turin, Italy
- Department of Medical and Clinical Psychology, and CoRPS–Center of Research on Psychology in Somatic Diseases–Tilburg University, Tilburg, Netherlands
| |
Collapse
|
41
|
Liu X, Melcher D. The effect of familiarity on behavioral oscillations in face perception. Sci Rep 2023; 13:10145. [PMID: 37349366 PMCID: PMC10287701 DOI: 10.1038/s41598-023-34812-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 05/08/2023] [Indexed: 06/24/2023] Open
Abstract
Studies on behavioral oscillations demonstrate that visual sensitivity fluctuates over time and visual processing varies periodically, mirroring neural oscillations at the same frequencies. Do these behavioral oscillations reflect fixed and relatively automatic sensory sampling, or top-down processes such as attention or predictive coding? To disentangle these theories, the current study used a dual-target rapid serial visual presentation paradigm, where participants indicated the gender of a face target embedded in streams of distractors presented at 30 Hz. On critical trials, two identical targets were presented with varied stimulus onset asynchrony from 200 to 833 ms. The target was either familiar or unfamiliar faces, divided into different blocks. We found a 4.6 Hz phase-coherent fluctuation in gender discrimination performance across both trial types, consistent with previous reports. In addition, however, we found an effect at the alpha frequency, with behavioral oscillations in the familiar blocks characterized by a faster high-alpha peak than for the unfamiliar face blocks. These results are consistent with the combination of both a relatively stable modulation in the theta band and faster modulation of the alpha oscillations. Therefore, the overall pattern of perceptual sampling in visual perception may depend, at least in part, on task demands. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on 16/08/2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/A98UF .
Collapse
Affiliation(s)
- Xiaoyi Liu
- New York University Abu Dhabi, Abu Dhabi, UAE
| | | |
Collapse
|
42
|
Zhou Y, Li W, Gao T, Pan X, Han S. Neural representation of perceived race mediates the opposite relationship between subcomponents of self-construals and racial outgroup punishment. Cereb Cortex 2023; 33:8759-8772. [PMID: 37143178 PMCID: PMC10786092 DOI: 10.1093/cercor/bhad157] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2022] [Revised: 04/18/2023] [Accepted: 04/19/2023] [Indexed: 05/06/2023] Open
Abstract
Outgroup aggression characterizes intergroup conflicts in human societies. Previous research on relationships between cultural traits and outgroup aggression behavior showed inconsistent results, leaving open questions regarding whether cultural traits predict individual differences in outgroup aggression and related neural underpinnings. We conducted 2 studies to address this issue by collecting self-construal scores, EEG signals in response to Asian and White faces with painful or neutral expressions, and decisions to apply electric shocks to other-race individuals in a context of interracial conflict. We found that interdependent self-construals were well explained by 2 subcomponents, including esteem for group (EG) and relational interdependence (RI), which are related to focus on group collectives and harmonious relationships, respectively. Moreover, EG was positively associated with the decisions to punish racial outgroup targets, whereas RI was negatively related to the decisions. These opposite relationships were mediated by neural representations of perceived race at 120-160 ms after face onset. Our findings highlight the multifaceted nature of interdependent self-construal and the key role of neural representations of race in mediating the relationships of different subcomponents of cultural traits with racial outgroup punishment decisions in a context of interracial conflict.
Collapse
Affiliation(s)
- Yuqing Zhou
- CAS Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences, 16 Lincui Road, Beijing 100101, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Wenxin Li
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, PKU-IDG/McGovern Institute for Brain Research, Peking University, 52 Haidian Road, Beijing 100080, China
| | - Tianyu Gao
- Department of Psychology, Faculty of Arts and Sciences, Beijing Normal University, Zhuhai 519087, China
| | - Xinyue Pan
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, PKU-IDG/McGovern Institute for Brain Research, Peking University, 52 Haidian Road, Beijing 100080, China
| | - Shihui Han
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, PKU-IDG/McGovern Institute for Brain Research, Peking University, 52 Haidian Road, Beijing 100080, China
| |
Collapse
|
43
|
Maffei A, Coccaro A, Jaspers-Fayer F, Goertzen J, Sessa P, Liotti M. EEG alpha band functional connectivity reveals distinct cortical dynamics for overt and covert emotional face processing. Sci Rep 2023; 13:9951. [PMID: 37337009 DOI: 10.1038/s41598-023-36860-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2022] [Accepted: 06/13/2023] [Indexed: 06/21/2023] Open
Abstract
Current knowledge regarding how the focus of our attention during face processing influences neural responses largely comes from neuroimaging studies reporting on regional brain activations. The present study was designed to add novel insights to this research by studying how attention can differentially impact the way cortical regions interact during emotional face processing. High-density electroencephalogram was recorded in a sample of fifty-two healthy participants during an emotional face processing task. The task required participants to either attend to the expressions (i.e., overt processing) or attend to a perceptual distractor, which rendered the expressions task-irrelevant (i.e., covert processing). Functional connectivity in the alpha band was estimated in source space and modeled using graph theory to quantify whole-brain integration and segregation. Results revealed that overt processing of facial expressions is linked to reduced cortical segregation and increased cortical integration, this latter specifically for negative expressions of fear and sadness. Furthermore, we observed increased communication efficiency during overt processing of negative expressions between the core and the extended face processing systems. Overall, these findings reveal that attention makes the interaction among the nodes involved in face processing more efficient, also uncovering a connectivity signature of the prioritized processing mechanism of negative expressions, that is an increased cross-communication within the nodes of the face processing network.
Collapse
Affiliation(s)
- Antonio Maffei
- Department of Developmental Psychology and Socialisation, University of Padova, Padua, Italy
- Padova Neuroscience Center (PNC), University of Padova, Padua, Italy
| | - Ambra Coccaro
- Department of Developmental Psychology and Socialisation, University of Padova, Padua, Italy
- Padova Neuroscience Center (PNC), University of Padova, Padua, Italy
| | | | | | - Paola Sessa
- Department of Developmental Psychology and Socialisation, University of Padova, Padua, Italy
- Padova Neuroscience Center (PNC), University of Padova, Padua, Italy
| | - Mario Liotti
- Department of Developmental Psychology and Socialisation, University of Padova, Padua, Italy.
- Padova Neuroscience Center (PNC), University of Padova, Padua, Italy.
- Department of Psychology, Simon Fraser University, Burnaby, Canada.
| |
Collapse
|
44
|
Schwartz E, Alreja A, Richardson RM, Ghuman A, Anzellotti S. Intracranial Electroencephalography and Deep Neural Networks Reveal Shared Substrates for Representations of Face Identity and Expressions. J Neurosci 2023; 43:4291-4303. [PMID: 37142430 PMCID: PMC10255163 DOI: 10.1523/jneurosci.1277-22.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2022] [Revised: 03/25/2023] [Accepted: 04/17/2023] [Indexed: 05/06/2023] Open
Abstract
According to a classical view of face perception (Bruce and Young, 1986; Haxby et al., 2000), face identity and facial expression recognition are performed by separate neural substrates (ventral and lateral temporal face-selective regions, respectively). However, recent studies challenge this view, showing that expression valence can also be decoded from ventral regions (Skerry and Saxe, 2014; Li et al., 2019), and identity from lateral regions (Anzellotti and Caramazza, 2017). These findings could be reconciled with the classical view if regions specialized for one task (either identity or expression) contain a small amount of information for the other task (that enables above-chance decoding). In this case, we would expect representations in lateral regions to be more similar to representations in deep convolutional neural networks (DCNNs) trained to recognize facial expression than to representations in DCNNs trained to recognize face identity (the converse should hold for ventral regions). We tested this hypothesis by analyzing neural responses to faces varying in identity and expression. Representational dissimilarity matrices (RDMs) computed from human intracranial recordings (n = 11 adults; 7 females) were compared with RDMs from DCNNs trained to label either identity or expression. We found that RDMs from DCNNs trained to recognize identity correlated with intracranial recordings more strongly in all regions tested-even in regions classically hypothesized to be specialized for expression. These results deviate from the classical view, suggesting that face-selective ventral and lateral regions contribute to the representation of both identity and expression.SIGNIFICANCE STATEMENT Previous work proposed that separate brain regions are specialized for the recognition of face identity and facial expression. However, identity and expression recognition mechanisms might share common brain regions instead. We tested these alternatives using deep neural networks and intracranial recordings from face-selective brain regions. Deep neural networks trained to recognize identity and networks trained to recognize expression learned representations that correlate with neural recordings. Identity-trained representations correlated with intracranial recordings more strongly in all regions tested, including regions hypothesized to be expression specialized in the classical hypothesis. These findings support the view that identity and expression recognition rely on common brain regions. This discovery may require reevaluation of the roles that the ventral and lateral neural pathways play in processing socially relevant stimuli.
Collapse
Affiliation(s)
- Emily Schwartz
- Department of Psychology and Neuroscience, Boston College, Chestnut Hill, Massachusetts 02467
| | - Arish Alreja
- Center for the Neural Basis of Cognition, Carnegie Mellon University/University of Pittsburgh, Pittsburgh, Pennsylvania 15213
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213
- Machine Learning Department, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213
- Department of Neurological Surgery, University of Pittsburgh Medical Center Presbyterian, Pittsburgh, Pennsylvania 15213
| | - R Mark Richardson
- Department of Neurosurgery, Massachusetts General Hospital, Boston, Massachusetts 02114
- Harvard Medical School, Boston, Massachusetts 02115
| | - Avniel Ghuman
- Center for the Neural Basis of Cognition, Carnegie Mellon University/University of Pittsburgh, Pittsburgh, Pennsylvania 15213
- Department of Neurological Surgery, University of Pittsburgh Medical Center Presbyterian, Pittsburgh, Pennsylvania 15213
- Center for Neuroscience, University of Pittsburgh, Pittsburgh, Pennsylvania 15260
| | - Stefano Anzellotti
- Department of Psychology and Neuroscience, Boston College, Chestnut Hill, Massachusetts 02467
| |
Collapse
|
45
|
Zhang H, Ding X, Liu N, Nolan R, Ungerleider LG, Japee S. Equivalent processing of facial expression and identity by macaque visual system and task-optimized neural network. Neuroimage 2023; 273:120067. [PMID: 36997134 PMCID: PMC10165955 DOI: 10.1016/j.neuroimage.2023.120067] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2022] [Revised: 02/20/2023] [Accepted: 03/27/2023] [Indexed: 03/30/2023] Open
Abstract
Both the primate visual system and artificial deep neural network (DNN) models show an extraordinary ability to simultaneously classify facial expression and identity. However, the neural computations underlying the two systems are unclear. Here, we developed a multi-task DNN model that optimally classified both monkey facial expressions and identities. By comparing the fMRI neural representations of the macaque visual cortex with the best-performing DNN model, we found that both systems: (1) share initial stages for processing low-level face features which segregate into separate branches at later stages for processing facial expression and identity respectively, and (2) gain more specificity for the processing of either facial expression or identity as one progresses along each branch towards higher stages. Correspondence analysis between the DNN and monkey visual areas revealed that the amygdala and anterior fundus face patch (AF) matched well with later layers of the DNN's facial expression branch, while the anterior medial face patch (AM) matched well with later layers of the DNN's facial identity branch. Our results highlight the anatomical and functional similarities between macaque visual system and DNN model, suggesting a common mechanism between the two systems.
Collapse
Affiliation(s)
- Hui Zhang
- School of Engineering Medicine, Beihang University; Key Laboratory of Biomechanics and Mechanobiology (Beihang University), Ministry of Education, Key Laboratory of Big Data-Based Precision Medicine, Ministry of Industry and Information Technology of the People's Republic of China, Beijing 100191, China; Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, Maryland 20892, USA.
| | - Xuetong Ding
- School of Engineering Medicine, Beihang University; Key Laboratory of Biomechanics and Mechanobiology (Beihang University), Ministry of Education, Key Laboratory of Big Data-Based Precision Medicine, Ministry of Industry and Information Technology of the People's Republic of China, Beijing 100191, China
| | - Ning Liu
- State Key Laboratory of Brain and Cognitive Science, Institute of Biophysics, Chinese Academy of Sciences, Beijing 100101, China..óSchool of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China; Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei 230088, China; Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, Maryland 20892, USA
| | - Rachel Nolan
- Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, Maryland 20892, USA
| | | | - Shruti Japee
- Laboratory of Brain and Cognition, NIMH, NIH, Bethesda, Maryland 20892, USA
| |
Collapse
|
46
|
Watanabe N, Miyoshi K, Jimura K, Shimane D, Keerativittayayut R, Nakahara K, Takeda M. Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans. Neuroimage 2023; 275:120164. [PMID: 37169115 DOI: 10.1016/j.neuroimage.2023.120164] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2022] [Revised: 05/02/2023] [Accepted: 05/09/2023] [Indexed: 05/13/2023] Open
Abstract
Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. Recently, neural decoding schemes, such as machine learning in functional magnetic resonance imaging (fMRI), has been employed to elucidate the underlying neural mechanisms. However, it remains unclear as to how spatially distributed brain regions temporally represent visual object categories and sub-categories. One promising strategy to address this issue is neural decoding with concurrently obtained neural response data of high spatial and temporal resolution. In this study, we explored the spatial and temporal organization of visual object representations using concurrent fMRI and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). We hypothesized that neural decoding by multimodal neural data with DNN would show high classification performance in visual object categorization (faces or non-face objects) and sub-categorization within faces and objects. Visualization of the fMRI DNN was more sensitive than that in the univariate approach and revealed that visual categorization occurred in brain-wide regions. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization compared with fMRI DNN or EEG DNN alone. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner, and provide strong evidence of the ability of multimodal deep learning to uncover spatiotemporal neural machinery in sensory processing.
Collapse
Affiliation(s)
- Noriya Watanabe
- Research Center for Brain Communication, Kochi University of Technology, Kami, Kochi, 782-8502, Japan
| | - Kosuke Miyoshi
- Narrative Nights, Inc., Yokohama, Kanagawa, 236-0011, Japan
| | - Koji Jimura
- Research Center for Brain Communication, Kochi University of Technology, Kami, Kochi, 782-8502, Japan; Department of Informatics, Gunma University, Maebashi, Gunma, 371-8510, Japan
| | - Daisuke Shimane
- Research Center for Brain Communication, Kochi University of Technology, Kami, Kochi, 782-8502, Japan
| | - Ruedeerat Keerativittayayut
- Research Center for Brain Communication, Kochi University of Technology, Kami, Kochi, 782-8502, Japan; Chulabhorn Royal Academy, Bangkok, 10210, Thailand
| | - Kiyoshi Nakahara
- Research Center for Brain Communication, Kochi University of Technology, Kami, Kochi, 782-8502, Japan
| | - Masaki Takeda
- Research Center for Brain Communication, Kochi University of Technology, Kami, Kochi, 782-8502, Japan.
| |
Collapse
|
47
|
Josephs EL, Hebart MN, Konkle T. Dimensions underlying human understanding of the reachable world. Cognition 2023; 234:105368. [PMID: 36641868 PMCID: PMC11562958 DOI: 10.1016/j.cognition.2023.105368] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 12/20/2022] [Accepted: 01/09/2023] [Indexed: 01/15/2023]
Abstract
Near-scale environments, like work desks, restaurant place settings or lab benches, are the interface of our hand-based interactions with the world. How are our conceptual representations of these environments organized? What properties distinguish among reachspaces, and why? We obtained 1.25 million similarity judgments on 990 reachspace images, and generated a 30-dimensional embedding which accurately predicts these judgments. Examination of the embedding dimensions revealed key properties underlying these judgments, such as reachspace layout, affordance, and visual appearance. Clustering performed over the embedding revealed four distinct interpretable classes of reachspaces, distinguishing among spaces related to food, electronics, analog activities, and storage or display. Finally, we found that reachspace similarity ratings were better predicted by the function of the spaces than their locations, suggesting that reachspaces are largely conceptualized in terms of the actions they support. Altogether, these results reveal the behaviorally-relevant principles that structure our internal representations of reach-relevant environments.
Collapse
Affiliation(s)
- Emilie L Josephs
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, USA; Psychology Department, Harvard University, Cambridge, USA.
| | - Martin N Hebart
- Vision and Computational Cognition Group, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
| | - Talia Konkle
- Psychology Department, Harvard University, Cambridge, USA.
| |
Collapse
|
48
|
He BJ. Towards a pluralistic neurobiological understanding of consciousness. Trends Cogn Sci 2023; 27:420-432. [PMID: 36842851 PMCID: PMC10101889 DOI: 10.1016/j.tics.2023.02.001] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2022] [Revised: 02/02/2023] [Accepted: 02/03/2023] [Indexed: 02/27/2023]
Abstract
Theories of consciousness are often based on the assumption that a single, unified neurobiological account will explain different types of conscious awareness. However, recent findings show that, even within a single modality such as conscious visual perception, the anatomical location, timing, and information flow of neural activity related to conscious awareness vary depending on both external and internal factors. This suggests that the search for generic neural correlates of consciousness may not be fruitful. I argue that consciousness science requires a more pluralistic approach and propose a new framework: joint determinant theory (JDT). This theory may be capable of accommodating different brain circuit mechanisms for conscious contents as varied as percepts, wills, memories, emotions, and thoughts, as well as their integrated experience.
Collapse
Affiliation(s)
- Biyu J He
- Neuroscience Institute, New York University Grossman School of Medicine, New York, NY 10016, USA; Departments of Neurology, Neuroscience and Physiology, Radiology, New York University Grossman School of Medicine, New York, NY 10016.
| |
Collapse
|
49
|
Decomposing Neural Representational Patterns of Discriminatory and Hedonic Information during Somatosensory Stimulation. eNeuro 2023; 10:ENEURO.0274-22.2022. [PMID: 36549914 PMCID: PMC9829099 DOI: 10.1523/eneuro.0274-22.2022] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2022] [Revised: 12/06/2022] [Accepted: 12/07/2022] [Indexed: 12/24/2022] Open
Abstract
The ability to interrogate specific representations in the brain, determining how, and where, difference sources of information are instantiated can provide invaluable insight into neural functioning. Pattern component modeling (PCM) is a recent analytic technique for human neuroimaging that allows the decomposition of representational patterns in brain into contributing subcomponents. In the current study, we present a novel PCM variant that tracks the contribution of prespecified representational patterns to brain representation across areas, thus allowing hypothesis-guided employment of the technique. We apply this technique to investigate the contributions of hedonic and nonhedonic information to the neural representation of tactile experience. We applied aversive pressure (AP) and appetitive brush (AB) to stimulate distinct peripheral nerve pathways for tactile information (C-/CT-fibers, respectively) while patients underwent functional magnetic resonance imaging (fMRI) scanning. We performed representational similarity analyses (RSAs) with pattern component modeling to dissociate how discriminatory versus hedonic tactile information contributes to population code representations in the human brain. Results demonstrated that information about appetitive and aversive tactile sensation is represented separately from nonhedonic tactile information across cortical structures. This also demonstrates the potential of new hypothesis-guided PCM variants to help delineate how information is instantiated in the brain.
Collapse
|
50
|
Lavan N. How do we describe other people from voices and faces? Cognition 2023; 230:105253. [PMID: 36215763 DOI: 10.1016/j.cognition.2022.105253] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2021] [Revised: 07/29/2022] [Accepted: 08/06/2022] [Indexed: 11/07/2022]
Abstract
When seeing someone's face or hearing their voice, perceivers routinely infer information about a person's age, sex and social traits. While many experiments have explored how individual person characteristics are perceived in isolation, less is known about which person characteristics are described spontaneously from voices and faces and how descriptions may differ across modalities. In Experiment 1, participants provided free descriptions for voices and faces. These free descriptions followed similar patterns for voices and faces - and for individual identities: Participants spontaneously referred to a wide range of descriptors. Psychological descriptors, such as character traits, were used most frequently; physical characteristics, such as age and sex, were notable as they were mentioned earlier than other types of descriptors. After finding primarily similarities between modalities when analysing person descriptions across identities, Experiment 2 asked whether free descriptions encode how individual identities differ. For this purpose, the measures derived from the free descriptions were linked to voice/face discrimination judgements that are known to describe differences in perceptual properties between identity pairs. Significant relationships emerged within and across modalities, showing that free descriptions indeed encode differences between identities - information that is shared with discrimination judgements. This suggests that the two tasks tap into similar, high-level person representations. These findings show that free description data can offer valuable insights into person perception and underline that person perception is a multivariate process during which perceivers rapidly and spontaneously infer many different person characteristics to form a holistic impression of a person.
Collapse
Affiliation(s)
- Nadine Lavan
- Department of Biological and Experimental Psychology, School of Biological and Behavioural Sciences, Queen Mary University of London, Mile End Road, London E1 4NS, United Kingdom.
| |
Collapse
|