A unified coding strategy for processing faces and voices

Face and voice identity matching accuracy is not improved by multimodal identity information

Identity verification from both faces and voices can be error-prone. Previous research has shown that faces and voices signal concordant information and cross-modal unfamiliar face-to-voice matching is possible, albeit often with low accuracy. In the current study, we ask whether performance on a face or voice identity matching task can be improved by using multimodal stimuli which add a second modality (voice or face). We find that overall accuracy is higher for face matching than for voice matching. However, contrary to predictions, presenting one unimodal and one multimodal stimulus within a matching task did not improve face or voice matching compared to presenting two unimodal stimuli. Additionally, we find that presenting two multimodal stimuli does not improve accuracy compared to presenting two unimodal face stimuli. Thus, multimodal information does not improve accuracy. However, intriguingly, we find that cross-modal face-voice matching accuracy predicts voice matching accuracy but not face matching accuracy. This suggests cross-modal information can nonetheless play a role in identity matching, and face and voice information combine to inform matching decisions. We discuss our findings in light of current models of person perception, and consider the implications for identity verification in security and forensic settings.

Decoding the Spatiotemporal Dynamics of Neural Response Similarity in Auditory Processing: A Multivariate Analysis Based on OPM-MEG

The brain represents information through the encoding of neural populations, where the activity patterns of these neural groups constitute the content of this information. Understanding these activity patterns and their dynamic changes is of significant importance to cognitive neuroscience and related research areas. Current studies focus more on brain regions that show differential responses to stimuli, but they lack the ability to capture information about the representational or process-level dynamics within these regions. In this study, we recorded neural data from 10 healthy participants during auditory experiments using optically pumped magnetometer magnetoencephalography (OPM-MEG) and electroencephalography (EEG). We constructed representational similarity matrices (RSMs) to investigate the similarity of neural response patterns during auditory decoding. The results indicate that RSA can reveal the dynamic changes in pattern similarity during different stages of auditory processing through the neural activity patterns reflected by OPM-MEG. Comparisons with EEG results showed that both techniques captured the same processes during the early stages of auditory decoding. However, differences in sensitivity at later stages highlighted both common and distinct aspects of neural representation between the two modalities. Further analysis indicated that this process involved widespread neural network activation, including the Heschl's gyrus, superior temporal gyrus, middle temporal gyrus, inferior temporal gyrus, parahippocampal gyrus, and orbitofrontal gyrus. This study demonstrates that the combination of OPM-MEG and RSA is sufficiently sensitive to detect changes in pattern similarity during neural representation processes and to identify their anatomical origins, offering new insights and references for the future application of RSA and other multivariate pattern analysis methods in the MEG field.

Introducing the Sisu Voice Matching Test (SVMT): A novel tool for assessing voice discrimination in Chinese

Existing standardized tests for voice discrimination are based mainly on Indo-European languages, particularly English. However, voice identity perception is influenced by language familiarity, with listeners generally performing better in their native language than in a foreign one. To provide a more accurate and comprehensive assessment of voice discrimination, it is crucial to develop tests tailored to the native language of the test takers. In response, we developed the Sisu Voice Matching Test (SVMT), a pioneering tool designed specifically for Mandarin Chinese speakers. The SVMT was designed to model real-world communication since it includes both pseudo-word and pseudo-sentence stimuli and covers both the ability to categorize identical voices as the same and the ability to categorize distinct voices as different. Built on a neurally validated voice-space model and item response theory, the SVMT ensures high reliability, validity, appropriate difficulty, and strong discriminative power, while maintaining a concise test duration of approximately 10 min. Therefore, by taking into account the effects of language nativeness, the SVMT complements existing voice tests based on other languages' phonologies to provide a more accurate assessment of voice discrimination ability for Mandarin Chinese speakers. Future research can use the SVMT to deepen our understanding of the mechanisms underlying human voice identity perception, especially in special populations, and to examining the relationship between voice identity recognition and other cognitive processes.

Setting the tone: crossmodal emotional face-voice combinations in continuous flash suppression

Emotional stimuli are preferentially processed in the visual system, in particular, fearful faces. Evidence comes from unimodal studies with emotional faces, although real-life emotional encounters typically involve input from multiple sensory channels, such as a face paired with a voice. Therefore, in this study, we investigated how emotional voices influence preferential processing of co-occurring emotional faces. To investigate early visual processing, we used the breaking continuous flash suppression paradigm (b-CFS): We presented fearful, happy, or neutral faces to one eye, which were initially inaccessible to conscious awareness due to the predominant perception of a dynamic mask presented to the other eye. Faces were presented either unimodally or paired with non-linguistic vocalizations (fearful, happy, neutral). Thirty-six healthy participants were asked to respond as soon as the faces reached conscious awareness. We replicated earlier findings that fearful faces broke suppression faster overall, supporting a threat bias. Moreover, all faces broke suppression faster when paired with voices. Interestingly, faces paired with neutral and happy voices broke suppression the fastest, followed by faces with fearful voices. Thus, in addition to supporting a threat bias in unimodally presented fearful faces, we found evidence for crossmodal facilitation.

Toward characterization of perceptual specialization for faces in Multiracial contexts

This conceptual analysis focuses on opportunities to advance research and current hypotheses of perceptual development by examining what is presently known and unknown about perceptual specialization in a Multiracial context during the first year of life. The impact of being raised in a Multiracial family or community is discussed to further characterize the development of perceptual expertise for faces and languages. Historical and present-day challenges faced by researchers in defining what race is, identifying Multiracial individuals or contexts, and how to study perceptual and cognitive processes in this population are discussed. We propose to leverage current data from developmental Multilingual populations as a guide for future research questions and hypotheses characterizing perceptual specialization based on face race for Multiracial/Multiethnic individuals and contexts. Variability of input and the pattern of specialization are two factors identified from the developmental Multilingual literature that are likely useful for studying Multiracial contexts and development. Several methodological considerations are proposed in hopes of facilitating research questions and practices that are reflective of and informed by the diversity of experiences and social complexities within Multiracial populations.

Diverse Face Images (DFI): Validated for racial representation and eye gaze

Face processing is a central component of human communication and social engagement. The present investigation introduces a set of racially and ethnically inclusive faces created for researchers interested in perceptual and socio-cognitive processes linked to human faces. The Diverse Face Images (DFI) stimulus set includes high-quality still images of female faces that are racially and ethnically representative, include multiple images of direct and indirect gaze for each model and control for low-level perceptual variance between images. The DFI stimuli will support researchers interested in studying face processing throughout the lifespan as well as other questions that require a diversity of faces or gazes. This report includes a detailed description of stimuli development and norming data for each model. Adults completed a questionnaire rating each image in the DFI stimuli set on three major qualities relevant to face processing: (1) strength of race/ethnicity group associations, (2) strength of eye gaze orientation, and (3) strength of emotion expression. These validation data highlight the presence of rater variability within and between individual model images as well as within and between race and ethnicity groups.

Neural Representations of Emotions in Visual, Auditory, and Modality-Independent Regions Reflect Idiosyncratic Conceptual Knowledge

Growing evidence suggests that conceptual knowledge influences emotion perception, yet the neural mechanisms underlying this effect are not fully understood. Recent studies have shown that brain representations of facial emotion categories in visual-perceptual areas are predicted by conceptual knowledge, but it remains to be seen if auditory regions are similarly affected. Moreover, it is not fully clear whether these conceptual influences operate at a modality-independent level. To address these questions, we conducted a functional magnetic resonance imaging study presenting participants with both facial and vocal emotional stimuli. This dual-modality approach allowed us to investigate effects on both modality-specific and modality-independent brain regions. Using univariate and representational similarity analyses, we found that brain representations in both visual (middle and lateral occipital cortices) and auditory (superior temporal gyrus) regions were predicted by conceptual understanding of emotions for faces and voices, respectively. Additionally, we discovered that conceptual knowledge also influenced supra-modal representations in the superior temporal sulcus. Dynamic causal modeling revealed a brain network showing both bottom-up and top-down flows, suggesting a complex interplay of modality-specific and modality-independent regions in emotional processing. These findings collectively indicate that the neural representations of emotions in both sensory-perceptual and modality-independent regions are likely shaped by each individual's conceptual knowledge.

Idiosyncratic and shared contributions shape impressions from voices and faces

Voices elicit rich first impressions of what the person we are hearing might be like. Research stresses that these impressions from voices are shared across different listeners, such that people on average agree which voices sound trustworthy or old and which do not. However, can impressions from voices also be shaped by the 'ear of the beholder'? We investigated whether - and how - listeners' idiosyncratic, personal preferences contribute to first impressions from voices. In two studies (993 participants, 156 voices), we find evidence for substantial idiosyncratic contributions to voice impressions using a variance portioning approach. Overall, idiosyncratic contributions were as important as shared contributions to impressions from voices for inferred person characteristics (e.g., trustworthiness, friendliness). Shared contributions were only more influential for impressions of more directly apparent person characteristics (e.g., gender, age). Both idiosyncratic and shared contributions were reduced when stimuli were limited in their (perceived) variability, suggesting that natural variation in voices is key to understanding this impression formation. When comparing voice impressions to face impressions, we found that idiosyncratic and shared contributions to impressions similarly across modality when stimulus properties are closely matched - although voice impressions were overall less consistent than face impressions. We thus reconceptualise impressions from voices as being formed not only based on shared but also idiosyncratic contributions. We use this new framing to suggest future directions of research, including understanding idiosyncratic mechanisms, development, and malleability of voice impression formation.

Functional alterations of lateral temporal cortex for processing voice prosody in adults with autism spectrum disorder

The human auditory system includes discrete cortical patches and selective regions for processing voice information, including emotional prosody. Although behavioral evidence indicates individuals with autism spectrum disorder (ASD) have difficulties in recognizing emotional prosody, it remains understudied whether and how localized voice patches (VPs) and other voice-sensitive regions are functionally altered in processing prosody. This fMRI study investigated neural responses to prosodic voices in 25 adult males with ASD and 33 controls using voices of anger, sadness, and happiness with varying degrees of emotion. We used a functional region-of-interest analysis with an independent voice localizer to identify multiple VPs from combined ASD and control data. We observed a general response reduction to prosodic voices in specific VPs of left posterior temporal VP (TVP) and right middle TVP. Reduced cortical responses in right middle TVP were consistently correlated with the severity of autistic symptoms for all examined emotional prosodies. Moreover, representation similarity analysis revealed the reduced effect of emotional intensity in multivoxel activation patterns in left anterior superior temporal cortex only for sad prosody. These results indicate reduced response magnitudes to voice prosodies in specific TVPs and altered emotion intensity-dependent multivoxel activation patterns in adult ASDs, potentially underlying their socio-communicative difficulties.

The neuropsychological evaluation of face identity recognition

Facial identity recognition (FIR) is arguably the ultimate form of recognition for the adult human brain. Even if the term prosopagnosia is reserved for exceptionally rare brain-damaged cases with a category-specific abrupt loss of FIR at adulthood, subjective and objective impairments or difficulties of FIR are common in the neuropsychological population. Here we provide a critical overview of the evaluation of FIR both for clinicians and researchers in neuropsychology. FIR impairments occur following many causes that should be identified objectively by both general and specific, behavioral and neural examinations. We refute the commonly used dissociation between perceptual and memory deficits/tests for FIR, since even a task involving the discrimination of unfamiliar face images presented side-by-side relies on cortical memories of faces in the right-lateralized ventral occipito-temporal cortex. Another frequently encountered confusion is between specific deficits of the FIR function and a more general impairment of semantic memory (of people), the latter being most often encountered following anterior temporal lobe damage. Many computerized tests aimed at evaluating FIR have appeared over the last two decades, as reviewed here. However, despite undeniable strengths, they often suffer from ecological limitations, difficulties of instruction, as well as a lack of consideration for processing speed and qualitative information. Taking into account these issues, a recently developed behavioral test with natural images manipulating face familiarity, stimulus inversion, and correct response times as a key variable appears promising. The measurement of electroencephalographic (EEG) activity in the frequency domain from fast periodic visual stimulation also appears as a particularly promising tool to complete and enhance the neuropsychological assessment of FIR.

Human Recognition: The Utilization of Face, Voice, Name and Interactions-An Extended Editorial

The many stimulating contributions to this Special Issue of Brain Science focused on some basic issues of particular interest in current research, with emphasis on human recognition using faces, voices, and names [...].

Social evaluative implications of sensory adaptation to human voices

People form social evaluations of others following brief exposure to their voices, and these impressions are calibrated based on recent perceptual experience. Participants adapted to voices with fundamental frequency (f o; the acoustic correlate of perceptual pitch) manipulated to be gender-typical (i.e. masculine men and feminine women) or gender-atypical (i.e. feminine men and masculine women) before evaluating unaltered test voices within the same sex. Adaptation resulted in contrastive aftereffects. Listening to gender-atypical voices caused female voices to sound more feminine and attractive (Study 1) and male voices to sound more masculine and attractive (Study 2). Studies 3a and 3b tested whether adaptation occurred on a conceptual or perceptual level, respectively. In Study 3a, perceivers adapted to gender-typical or gender-atypical voices for both men and women (i.e. adaptors pitch manipulated in opposite directions for men and women) before evaluating unaltered test voices. Findings showed weak evidence that evaluations differed between conditions. In Study 3b, perceivers adapted to masculinized or feminized voices for both men and women (i.e. adaptors pitch manipulated in the same direction for men and women) before evaluating unaltered test voices. In the feminized condition, participants rated male targets as more masculine and attractive. Conversely, in the masculinized condition, participants rated female targets as more feminine and attractive. Voices appear to be evaluated according to gender norms that are updated based on perceptual experience as well as conceptual knowledge.

Designing for sensory adaptation: what you see depends on what you've been looking at - Recommendations, guidelines and standards should reflect this

Sensory systems continuously recalibrate their responses according to the current stimulus environment. As a result, perception is strongly affected by the current and recent context. These adaptative changes affect both sensitivity (e.g., habituating to noise, seeing better in the dark) and appearance (e.g. how things look, what catches attention) and adjust to many perceptual properties (e.g. from light level to the characteristics of someone's face). They therefore have a profound effect on most perceptual experiences, and on how and how well the senses work in different settings. Characterizing the properties of adaptation, how it manifests, and when it influences perception in modern environments can provide insights into the diversity of human experience. Adaptation could also be leveraged both to optimize perceptual abilities (e.g. in visual inspection tasks like radiology) and to mitigate unwanted consequences (e.g. exposure to potentially unhealthy stimulus environments).

Familiarity Is Key: Exploring the Effect of Familiarity on the Face-Voice Correlation

Recent research has examined the extent to which face and voice processing are associated by virtue of the fact that both tap into a common person perception system. However, existing findings do not yet fully clarify the role of familiarity in this association. Given this, two experiments are presented that examine face-voice correlations for unfamiliar stimuli (Experiment 1) and for familiar stimuli (Experiment 2). With care being taken to use tasks that avoid floor and ceiling effects and that use realistic speech-based voice clips, the results suggested a significant positive but small-sized correlation between face and voice processing when recognizing unfamiliar individuals. In contrast, the correlation when matching familiar individuals was significant and positive, but much larger. The results supported the existing literature suggesting that face and voice processing are aligned as constituents of an overarching person perception system. However, the difference in magnitude of their association here reinforced the view that familiar and unfamiliar stimuli are processed in different ways. This likely reflects the importance of a pre-existing mental representation and cross-talk within the neural architectures when processing familiar faces and voices, and yet the reliance on more superficial stimulus-based and modality-specific analysis when processing unfamiliar faces and voices.

The effect of voice familiarity on attention to speech in a cocktail party scenario

Selective attention to one speaker in multi-talker environments can be affected by the acoustic and semantic properties of speech. One highly ecological feature of speech that has the potential to assist in selective attention is voice familiarity. Here, we tested how voice familiarity interacts with selective attention by measuring the neural speech-tracking response to both target and non-target speech in a dichotic listening "Cocktail Party" paradigm. We measured Magnetoencephalography from n = 33 participants, presented with concurrent narratives in two different voices, and instructed to pay attention to one ear ("target") and ignore the other ("non-target"). Participants were familiarized with one of the voices during the week prior to the experiment, rendering this voice familiar to them. Using multivariate speech-tracking analysis we estimated the neural responses to both stimuli and replicate their well-established modulation by selective attention. Importantly, speech-tracking was also affected by voice familiarity, showing enhanced response for target speech and reduced response for non-target speech in the contra-lateral hemisphere, when these were in a familiar vs. an unfamiliar voice. These findings offer valuable insight into how voice familiarity, and by extension, auditory-semantics, interact with goal-driven attention, and facilitate perceptual organization and speech processing in noisy environments.

Dominance of face over voice in human attractiveness judgments: ERP evidence

The attractiveness of a person, a complex, and socially relevant type of information, is transmitted in many ways, not least through face and voice. However, it is unclear how the stimulus domains carrying attractiveness information interact. The present study explored the audiovisual perception of attractiveness in a Stroop-like paradigm using event-related potentials (ERPs). Participants were presented with face-voice pairs carrying congruent or incongruent attractiveness information and, in turn, judged the attractiveness level of each domain while ignoring the other. Voice attractiveness judgments were influenced by unattended face attractiveness, in terms of both, early perceptual encoding (N170, P200) as well as later evaluative stages (N400, LPC). In contrast, effects of unattended voice attractiveness on face attractiveness judgments were confined to early perceptual encoding (N170). These results demonstrate not only the interaction of multiple domains in human attractiveness perception at different processing stages but also a relative dominance of face over voice attractiveness.

Apperceptive and Associative Forms of Phonagnosia

PURPOSE OF REVIEW

Pronagnosia is a rare acquired or developmental pathological condition that consists of a selective difficulty to recognize familiar people by their voices. It can be distinguished into two different categories: apperceptive phonagnosia, which denotes a purely perceptual form of voice recognition disorder; and associative phonagnosia, in which patients have no perceptual defects, but cannot evaluate if the voice of a known person is or not familiar. The neural substrate of these two forms of voice recognition is still controversial, but it could concern different components of the core temporal voice areas and of extratemporal voice processing areas. This article reviews recent research on the neuropsychological and anatomo-clinical aspects of this condition.

RECENT FINDINGS

Data obtained in group studies or single case reports of phonagnosic patients suggest that apperceptive phonagnosia might be due to disruption of the core temporal voice areas, bilaterally located in the posterior parts of the superior temporal gyrus, whereas associative phonagnosia might result from impaired access to structures where voice representations are stored, due to a disconnection of these areas from structures of the voice extended system. Although these results must be confirmed by further investigations, they represent an important step toward understanding the nature and neural substrate of apperceptive and associative forms of phonagnosia.

Visual Deprivation Alters Functional Connectivity of Neural Networks for Voice Recognition: A Resting-State fMRI Study

Humans recognize one another by identifying their voices and faces. For sighted people, the integration of voice and face signals in corresponding brain networks plays an important role in facilitating the process. However, individuals with vision loss primarily resort to voice cues to recognize a person's identity. It remains unclear how the neural systems for voice recognition reorganize in the blind. In the present study, we collected behavioral and resting-state fMRI data from 20 early blind (5 females; mean age = 22.6 years) and 22 sighted control (7 females; mean age = 23.7 years) individuals. We aimed to investigate the alterations in the resting-state functional connectivity (FC) among the voice- and face-sensitive areas in blind subjects in comparison with controls. We found that the intranetwork connections among voice-sensitive areas, including amygdala-posterior "temporal voice areas" (TVAp), amygdala-anterior "temporal voice areas" (TVAa), and amygdala-inferior frontal gyrus (IFG) were enhanced in the early blind. The blind group also showed increased FCs of "fusiform face area" (FFA)-IFG and "occipital face area" (OFA)-IFG but decreased FCs between the face-sensitive areas (i.e., FFA and OFA) and TVAa. Moreover, the voice-recognition accuracy was positively related to the strength of TVAp-FFA in the sighted, and the strength of amygdala-FFA in the blind. These findings indicate that visual deprivation shapes functional connectivity by increasing the intranetwork connections among voice-sensitive areas while decreasing the internetwork connections between the voice- and face-sensitive areas. Moreover, the face-sensitive areas are still involved in the voice-recognition process in blind individuals through pathways such as the subcortical-occipital or occipitofrontal connections, which may benefit the visually impaired greatly during voice processing.

The Effect of Face-Voice Gender Consistency on Impression Evaluation

Face and voice are important information cues of interpersonal interaction. Most previous studies have investigated the cross-modal perception of face and voice from the perspective of cognitive psychology, but few empirical studies have focused on the effect of gender consistency of face and voice on the impression evaluation of the target from the perspective of social cognition. Based on the two-stage model of stereotype activation and the stereotype content model, this research examined the effects of face-voice gender consistency on impression evaluation (gender categorization and warmth competence evaluation) by using a cross-modal priming paradigm (Study 1, 20 males and 23 females, M_age = 21.00, SD_age = 2.59), a sequential presentation task (Study 2a, 57 males and 70 females, M_age = 18.54, SD_age = 1.54; Study 2b, 52 males and 51 females, M_age = 18.54, SD_age = 1.36), and a simultaneous presentation task (Study 3, 51 males and 55 females, M_age = 23.58, SD_age = 3.20), respectively. The results showed that: (1) there was a face-voice gender consistency preference in gender categorization, and the response of face-voice consistent condition was faster than that of inconsistent condition; (2) compared with the face-voice gender-inconsistent individuals, the participants showed a higher and more stable evaluation of the warmth and competence of the gender-consistent individuals, indicating the effect of matching preference of the face-voice gender consistency on the impression evaluation; (3) people paid more attention to the gender information of faces in the impression evaluation, and the female face could improve people's evaluation on the target's warmth and competence; (4) males were more intolerant of face-voice gender inconsistency when presented sequentially; the "voice needs to match face" effect was stronger for females when presented simultaneously. These findings, on the one hand, enrich and expand previous theories and research on cross-modal processing of face and voice from the perspective of social cognitive impression evaluation; on the other hand, these findings have important practical implications for impression management and decision-making in social interaction.

Human voices escape the auditory attentional blink: Evidence from detections and pupil responses

Attentional selection of a second target in a rapid stream of stimuli embedding two targets tends to be briefly impaired when two targets are presented in close temporal proximity, an effect known as an attentional blink (AB). Two target sounds (T1 and T2) were embedded in a rapid serial auditory presentation of environmental sounds with a short (Lag 3) or long lag (Lag 9). Participants were to first identify T1 (bell or sine tone) and then to detect T2 (present or absent). Individual stimuli had durations of either 30 or 90 ms, and were presented in streams of 20 sounds. The T2 varied in category: human voice, cello, or dog sound. Previous research has introduced pupillometry as a useful marker of the intensity of cognitive processing and attentional allocation in the visual AB paradigm. Results suggest that the interplay of stimulus factors is critical for target detection accuracy and provides support for the hypothesis that the human voice is the least likely to show an auditory AB (in the 90 ms condition). For the other stimuli, accuracy for T2 was significantly worse at Lag 3 than at Lag 9 in the 90 ms condition, suggesting the presence of an auditory AB. When AB occurred (at Lag 3), we observed smaller pupil dilations, time-locked to the onset of T2, compared to Lag 9, reflecting lower attentional processing when 'blinking' during target detection. Taken together, these findings support the conclusion that human voices escape the AB and that the pupillary changes are consistent with the so-called T2 attentional deficit. In addition, we found some indication that salient stimuli like human voices could require a less intense allocation of attention, or noradrenergic potentiation, compared to other auditory stimuli.

How do we describe other people from voices and faces?

When seeing someone's face or hearing their voice, perceivers routinely infer information about a person's age, sex and social traits. While many experiments have explored how individual person characteristics are perceived in isolation, less is known about which person characteristics are described spontaneously from voices and faces and how descriptions may differ across modalities. In Experiment 1, participants provided free descriptions for voices and faces. These free descriptions followed similar patterns for voices and faces - and for individual identities: Participants spontaneously referred to a wide range of descriptors. Psychological descriptors, such as character traits, were used most frequently; physical characteristics, such as age and sex, were notable as they were mentioned earlier than other types of descriptors. After finding primarily similarities between modalities when analysing person descriptions across identities, Experiment 2 asked whether free descriptions encode how individual identities differ. For this purpose, the measures derived from the free descriptions were linked to voice/face discrimination judgements that are known to describe differences in perceptual properties between identity pairs. Significant relationships emerged within and across modalities, showing that free descriptions indeed encode differences between identities - information that is shared with discrimination judgements. This suggests that the two tasks tap into similar, high-level person representations. These findings show that free description data can offer valuable insights into person perception and underline that person perception is a multivariate process during which perceivers rapidly and spontaneously infer many different person characteristics to form a holistic impression of a person.

Recognition of brief sounds in rapid serial auditory presentation

Two experiments were conducted to test the role of participant factors (i.e., musical sophistication, working memory capacity) and stimulus factors (i.e., sound duration, timbre) on auditory recognition using a rapid serial auditory presentation paradigm. Participants listened to a rapid stream of very brief sounds ranging from 30 to 150 milliseconds and were tested on their ability to distinguish the presence from the absence of a target sound selected from various sound sources placed amongst the distracters. Experiment 1a established that brief exposure to stimuli (60 to 150 milliseconds) does not necessarily correspond to impaired recognition. In Experiment 1b we found evidence that 30 milliseconds of exposure to the stimuli significantly impairs recognition of single auditory targets, but the recognition for voice and sine tone targets impaired the least, suggesting that the lower limit required for successful recognition could be lower than 30 milliseconds for voice and sine tone targets. Critically, the effect of sound duration on recognition completely disappeared when differences in musical sophistication were controlled for. Participants' working memory capacities did not seem to predict their recognition performances. Our behavioral results extend the studies oriented to understand the processing of brief timbres under temporal constraint by suggesting that the musical sophistication may play a larger role than previously thought. These results can also provide a working hypothesis for future research, namely, that underlying neural mechanisms for the processing of various sound sources may have different temporal constraints.

Effects of Faces and Voices on the Encoding of Biographic Information

There are multiple forms of knowledge about people. Whether diverse person-related data interact is of interest regarding the more general issue of integration of multi-source information about the world. Our goal was to examine whether perception of a person's face or voice enhanced the encoding of their biographic data. We performed three experiments. In the first experiment, subjects learned the biographic data of a character with or without a video clip of their face. In the second experiment, they learned the character's data with an audio clip of either a generic narrator's voice or the character's voice relating the same biographic information. In the third experiment, an audiovisual clip of both the face and voice of either a generic narrator or the character accompanied the learning of biographic data. After learning, a test phase presented biographic data alone, and subjects were tested first for familiarity and second for matching of biographic data to the name. The results showed equivalent learning of biographic data across all three experiments, and none showed evidence that a character's face or voice enhanced the learning of biographic information. We conclude that the simultaneous processing of perceptual representations of people may not modulate the encoding of biographic data.

Neural correlates of perceived emotions in human insula and amygdala for auditory emotion recognition

The emotional status of a speaker is an important non-linguistic cue carried by human voice and can be perceived by a listener in vocal communication. Understanding the neural circuits involved in processing emotions carried by human voice is crucial for understanding the neural basis of social interaction. Previous studies have shown that human insula and amygdala responded more selectively to emotional sounds than non-emotional sounds. However, it is not clear whether the neural selectivity to emotional sounds in these brain structures is determined by the emotion presented by a speaker which is associated with the acoustic properties of the sounds or by the emotion perceived by a listener. In this study, we recorded intracranial electroencephalography (iEEG) responses to emotional human voices while subjects performed emotion recognition tasks. We found that the iEEG responses of Heschl's gyrus (HG) and posterior insula were determined by the presented emotion, whereas the iEEG responses of anterior insula and amygdala were driven by the perceived emotion. These results suggest that the anterior insula and amygdala play a crucial role in conscious perception of emotions carried by human voice.

Formant-invariant voice and pitch representations are pre-attentively formed from constantly varying speech and non-speech stimuli

The present study investigated whether listeners can form abstract voice representations while ignoring constantly changing phonological information and if they can use the resulting information to facilitate voice change detection. Further, the study aimed at understanding whether the use of abstraction is restricted to the speech domain or can be deployed also in non‐speech contexts. We ran an electroencephalogram (EEG) experiment including one passive and one active oddball task, each featuring a speech and a rotated speech condition. In the speech condition, participants heard constantly changing vowels uttered by a male speaker (standard stimuli) which were infrequently replaced by vowels uttered by a female speaker with higher pitch (deviant stimuli). In the rotated speech condition, participants heard rotated vowels, in which the natural formant structure of speech was disrupted. In the passive task, the mismatch negativity was elicited after the presentation of the deviant voice in both conditions, indicating that listeners could successfully group together different stimuli into a formant‐invariant voice representation. In the active task, participants showed shorter reaction times (RTs), higher accuracy and a larger P3b in the speech condition with respect to the rotated speech condition. Results showed that whereas at a pre‐attentive level the cognitive system can track pitch regularities while presumably ignoring constantly changing formant information both in speech and in rotated speech, at an attentive level the use of such information is facilitated for speech. This facilitation was also testified by a stronger synchronisation in the theta band (4–7 Hz), potentially pointing towards differences in encoding/retrieval processes.

Acoustic voice variation in spontaneous speech

This study replicates and extends the recent findings of Lee, Keating, and Kreiman [J. Acoust. Soc. Am. 146(3), 1568-1579 (2019)] on acoustic voice variation in read speech, which showed remarkably similar acoustic voice spaces for groups of female and male talkers and the individual talkers within these groups. Principal component analysis was applied to acoustic indices of voice quality measured from phone conversations for 99/100 of the same talkers studied previously. The acoustic voice spaces derived from spontaneous speech are highly similar to those based on read speech, except that unlike read speech, variability in fundamental frequency accounted for significant acoustic variability. Implications of these findings for prototype models of speaker recognition and discrimination are considered.

Representation of expression and identity by ventral prefrontal neurons

Evidence has suggested that the ventrolateral prefrontal cortex (VLPFC) processes social stimuli, including faces and vocalizations, which are essential for communication. Features embedded within audiovisual stimuli, including emotional expression and caller identity, provide abundant information about an individual's intention, emotional state, motivation, and social status, which are important to encode in a social exchange. However, it is unknown to what extent the VLPFC encodes such features. To investigate the role of VLPFC during social communication, we recorded single-unit activity while rhesus macaques (Macaca mulatta) performed a nonmatch-to-sample task using species-specific face-vocalization stimuli that differed in emotional expression or caller identity. 75% of recorded cells were task-related and of these >70% were responsive during the nonmatch period. A larger proportion of nonmatch cells encoded the stimulus rather than the context of the trial type. A subset of responsive neurons were most commonly modulated by the identity of the nonmatch stimulus and less by the emotional expression, or both features within the face-vocalization stimuli presented during the nonmatch period. Neurons encoding identity were found in VLPFC across a broader region than expression related cells which were confined to only the anterolateral portion of the recording chamber in VLPFC. These findings suggest that, within a working memory paradigm, VLPFC processes features of face and vocal stimuli, such as emotional expression and identity, in addition to task and contextual information. Thus, stimulus and contextual information may be integrated by VLPFC during social communication.

Distinct functional levels of human voice processing in the auditory cortex

Voice signaling is integral to human communication, and a cortical voice area seemed to support the discrimination of voices from other auditory objects. This large cortical voice area in the auditory cortex (AC) was suggested to process voices selectively, but its functional differentiation remained elusive. We used neuroimaging while humans processed voices and nonvoice sounds, and artificial sounds that mimicked certain voice sound features. First and surprisingly, specific auditory cortical voice processing beyond basic acoustic sound analyses is only supported by a very small portion of the originally described voice area in higher-order AC located centrally in superior Te3. Second, besides this core voice processing area, large parts of the remaining voice area in low- and higher-order AC only accessorily process voices and might primarily pick up nonspecific psychoacoustic differences between voices and nonvoices. Third, a specific subfield of low-order AC seems to specifically decode acoustic sound features that are relevant but not exclusive for voice detection. Taken together, the previously defined voice area might have been overestimated since cortical support for human voice processing seems rather restricted. Cortical voice processing also seems to be functionally more diverse and embedded in broader functional principles of the human auditory system.

Longitudinal Change in Neural Response to Vocal Emotion in Adolescence

Adolescence is associated with maturation of function within neural networks supporting the processing of social information. Previous longitudinal studies have established developmental influences on youth’s neural response to facial displays of emotion. Given the increasing recognition of the importance of non-facial cues to social communication, we build on existing work by examining longitudinal change in neural response to vocal expressions of emotion in 8- to 19-year-old youth. Participants completed a vocal emotion recognition task at two timepoints (1 year apart) while undergoing functional magnetic resonance imaging. The right inferior frontal gyrus, right dorsal striatum and right precentral gyrus showed decreases in activation to emotional voices across timepoints, which may reflect focalization of response in these areas. Activation in the dorsomedial prefrontal cortex was positively associated with age but was stable across timepoints. In addition, the slope of change across visits varied as a function of participants’ age in the right temporo-parietal junction (TPJ): this pattern of activation across timepoints and age may reflect ongoing specialization of function across childhood and adolescence. Decreased activation in the striatum and TPJ across timepoints was associated with better emotion recognition accuracy. Findings suggest that specialization of function in social cognitive networks may support the growth of vocal emotion recognition skills across adolescence.

Understanding one's character through the voice: Dimensions of personality perception from Chinese greeting word "Ni Hao"

Previous western studies revealed a two-dimensional model (valence and dominance) in voice impressions. To explore the cross-cultural validity of this model, the present study recruited Chinese participants to evaluate other people's personality from recordings of Chinese vocal greeting word "Ni Hao". Principal Component Analysis (PCA) with Varimax Rotation and Parallel Analysis was used to investigate the dimensions underlying personality judgments. The results also revealed a two-dimensional model: approachability and capability. The approachability dimension was similar to the valence dimension reported in a previous study. It indicated that the approachability/valence dimension has cross-cultural commonality. Unlike the dimension of dominance which was closely related to aggressiveness, the dimension of capability emphasized the social aspects of capability such as intellectuality, social skills, and tenacity. In addition, the acoustic parameters that were used to infer the personality of speakers, as well as the relationship between vocal attractiveness and the personality dimensions of voice, were also partially different from the findings in Western culture.

Cochlea to categories: The spatiotemporal dynamics of semantic auditory representations

How does the auditory system categorize natural sounds? Here we apply multimodal neuroimaging to illustrate the progression from acoustic to semantically dominated representations. Combining magnetoencephalographic (MEG) and functional magnetic resonance imaging (fMRI) scans of observers listening to naturalistic sounds, we found superior temporal responses beginning ∼55 ms post-stimulus onset, spreading to extratemporal cortices by ∼100 ms. Early regions were distinguished less by onset/peak latency than by functional properties and overall temporal response profiles. Early acoustically-dominated representations trended systematically toward category dominance over time (after ∼200 ms) and space (beyond primary cortex). Semantic category representation was spatially specific: Vocalizations were preferentially distinguished in frontotemporal voice-selective regions and the fusiform; scenes and objects were distinguished in parahippocampal and medial place areas. Our results are consistent with real-world events coded via an extended auditory processing hierarchy, in which acoustic representations rapidly enter multiple streams specialized by category, including areas typically considered visual cortex.

Audiovisual identity perception from naturally-varying stimuli is driven by visual information

Identity perception often takes place in multimodal settings, where perceivers have access to both visual (face) and auditory (voice) information. Despite this, identity perception is usually studied in unimodal contexts, where face and voice identity perception are modelled independently from one another. In this study, we asked whether and how much auditory and visual information contribute to audiovisual identity perception from naturally-varying stimuli. In a between-subjects design, participants completed an identity sorting task with either dynamic video-only, audio-only or dynamic audiovisual stimuli. In this task, participants were asked to sort multiple, naturally-varying stimuli from three different people by perceived identity. We found that identity perception was more accurate for video-only and audiovisual stimuli compared with audio-only stimuli. Interestingly, there was no difference in accuracy between video-only and audiovisual stimuli. Auditory information nonetheless played a role alongside visual information as audiovisual identity judgements per stimulus could be predicted from both auditory and visual identity judgements, respectively. While the relationship was stronger for visual information and audiovisual information, auditory information still uniquely explained a significant portion of the variance in audiovisual identity judgements. Our findings thus align with previous theoretical and empirical work that proposes that, compared with faces, voices are an important but relatively less salient and a weaker cue to identity perception. We expand on this work to show that, at least in the context of this study, having access to voices in addition to faces does not result in better identity perception accuracy.

Unimodal and cross-modal identity judgements using an audio-visual sorting task: Evidence for independent processing of faces and voices

Unimodal and cross-modal information provided by faces and voices contribute to identity percepts. To examine how these sources of information interact, we devised a novel audio-visual sorting task in which participants were required to group video-only and audio-only clips into two identities. In a series of three experiments, we show that unimodal face and voice sorting were more accurate than cross-modal sorting: While face sorting was consistently most accurate followed by voice sorting, cross-modal sorting was at chancel level or below. In Experiment 1, we compared performance in our novel audio-visual sorting task to a traditional identity matching task, showing that unimodal and cross-modal identity perception were overall moderately more accurate than the traditional identity matching task. In Experiment 2, separating unimodal from cross-modal sorting led to small improvements in accuracy for unimodal sorting, but no change in cross-modal sorting performance. In Experiment 3, we explored the effect of minimal audio-visual training: Participants were shown a clip of the two identities in conversation prior to completing the sorting task. This led to small, nonsignificant improvements in accuracy for unimodal and cross-modal sorting. Our results indicate that unfamiliar face and voice perception operate relatively independently with no evidence of mutual benefit, suggesting that extracting reliable cross-modal identity information is challenging.

False alarms during recognition of famous people from faces and voices in patients with unilateral temporal lobe resection and normal participants tested after anodal tDCS over the left or right ATL

Data gathered in the field of the experimental social psychology have shown that it is more difficult to recognize a person through his/her voice than through his/her face and that false alarms (FA) are produced more in voice than in face recognition. Furthermore, some neuropsychological investigations have suggested that in patients with damage to the right anterior temporal lobe (ATL) the number of FA could be higher for voice than for face recognition. In the present study we assessed FA during recognition of famous people from faces and voices in patients with unilateral ATL tumours and in normal participants tested after anodal transcranial direct current stimulation (tCDS), over the left or right ATL. The number of FA was significantly higher in patients with right than in those with left temporal tumours on both face and voice familiarity. Furthermore, lesion side did not differentially affect patient's sensitivity or response criterion when recognizing famous faces, but influenced both these measures on a voice recognition task. In fact, in this condition patients with right temporal tumours showed a lower sensitivity index and a lower response criterion than those with left-sided lesions. In normal subjects, the greater right sided involvement in voice than in face processing was confirmed by the observation that right ATL anodal stimulation significantly increased voice but only marginally influenced face sensitivity. This asymmetry between face and voice processing in the right hemisphere could be due to the greater complexity of voice processing and to the difficulty of forming stable and well-structured representations, allowing to evaluate if a presented voice matches or not with an already known voice.

Auditory cortical micro-networks show differential connectivity during voice and speech processing in humans

The temporal voice areas (TVAs) in bilateral auditory cortex (AC) appear specialized for voice processing. Previous research assumed a uniform functional profile for the TVAs which are broadly spread along the bilateral AC. Alternatively, the TVAs might comprise separate AC nodes controlling differential neural functions for voice and speech decoding, organized as local micro-circuits. To investigate micro-circuits, we modeled the directional connectivity between TVA nodes during voice processing in humans while acquiring brain activity using neuroimaging. Results show several bilateral AC nodes for general voice decoding (speech and non-speech voices) and for speech decoding in particular. Furthermore, non-hierarchical and differential bilateral AC networks manifest distinct excitatory and inhibitory pathways for voice and speech processing. Finally, while voice and speech processing seem to have distinctive but integrated neural circuits in the left AC, the right AC reveals disintegrated neural circuits for both sounds. Altogether, we demonstrate a functional heterogeneity in the TVAs for voice decoding based on local micro-circuits.

The Role of Unimodal Feedback Pathways in Gender Perception During Activation of Voice and Face Areas

Cross-modal effects provide a model framework for investigating hierarchical inter-areal processing, particularly, under conditions where unimodal cortical areas receive contextual feedback from other modalities. Here, using complementary behavioral and brain imaging techniques, we investigated the functional networks participating in face and voice processing during gender perception, a high-level feature of voice and face perception. Within the framework of a signal detection decision model, Maximum likelihood conjoint measurement (MLCM) was used to estimate the contributions of the face and voice to gender comparisons between pairs of audio-visual stimuli in which the face and voice were independently modulated. Top–down contributions were varied by instructing participants to make judgments based on the gender of either the face, the voice or both modalities (N = 12 for each task). Estimated face and voice contributions to the judgments of the stimulus pairs were not independent; both contributed to all tasks, but their respective weights varied over a 40-fold range due to top–down influences. Models that best described the modal contributions required the inclusion of two different top–down interactions: (i) an interaction that depended on gender congruence across modalities (i.e., difference between face and voice modalities for each stimulus); (ii) an interaction that depended on the within modalities’ gender magnitude. The significance of these interactions was task dependent. Specifically, gender congruence interaction was significant for the face and voice tasks while the gender magnitude interaction was significant for the face and stimulus tasks. Subsequently, we used the same stimuli and related tasks in a functional magnetic resonance imaging (fMRI) paradigm (N = 12) to explore the neural correlates of these perceptual processes, analyzed with Dynamic Causal Modeling (DCM) and Bayesian Model Selection. Results revealed changes in effective connectivity between the unimodal Fusiform Face Area (FFA) and Temporal Voice Area (TVA) in a fashion that paralleled the face and voice behavioral interactions observed in the psychophysical data. These findings explore the role in perception of multiple unimodal parallel feedback pathways.

Cortical voice processing is grounded in elementary sound analyses for vocalization relevant sound patterns

A subregion of the auditory cortex (AC) was proposed to selectively process voices. This selectivity of the temporal voice area (TVA) and its role in processing non-voice sounds however have remained elusive. For a better functional description of the TVA, we investigated its neural responses both to voice and non-voice sounds, and critically also to textural sound patterns (TSPs) that share basic features with natural sounds but that are perceptually very distant from voices. Listening to these TSPs, first, elicited activity in large subregions of the TVA, which was mainly driven by perpetual ratings of TSPs along a voice similarity scale. This similar TVA activity in response to TSPs might partially explain activation patterns typically observed during voice processing. Second, we reconstructed the TVA activity that is usually observed in voice processing with a linear combination of activation patterns from TSPs. An analysis of the reconstruction model weights demonstrated that the TVA similarly processes both natural voice and non-voice sounds as well as TSPs along their acoustic and perceptual features. The predominant factor in reconstructing the TVA pattern by TSPs were the perceptual voice similarity ratings. Third, a multi-voxel pattern analysis confirms that the TSPs contain sufficient sound information to explain TVA activity for voice processing. Altogether, rather than being restricted to higher-order voice processing only, the human "voice area" uses mechanisms to evaluate the perceptual and acoustic quality of non-voice sounds, and responds to the latter with a "voice-like" processing pattern when detecting some rudimentary perceptual similarity with voices.

Normal recognition of famous voices in developmental prosopagnosia

Developmental prosopagnosia (DP) is a condition characterised by lifelong face recognition difficulties. Recent neuroimaging findings suggest that DP may be associated with aberrant structure and function in multimodal regions of cortex implicated in the processing of both facial and vocal identity. These findings suggest that both facial and vocal recognition may be impaired in DP. To test this possibility, we compared the performance of 22 DPs and a group of typical controls, on closely matched tasks that assessed famous face and famous voice recognition ability. As expected, the DPs showed severe impairment on the face recognition task, relative to typical controls. In contrast, however, the DPs and controls identified a similar number of voices. Despite evidence of interactions between facial and vocal processing, these findings suggest some degree of dissociation between the two processing pathways, whereby one can be impaired while the other develops typically. A possible explanation for this dissociation in DP could be that the deficit originates in the early perceptual encoding of face structure, rather than at later, post-perceptual stages of face identity processing, which may be more likely to involve interactions with other modalities.

Children and adolescents' neural response to emotional faces and voices: Age-related changes in common regions of activation

The perception of facial and vocal emotional expressions engages overlapping regions of the brain. However, at a behavioral level, the ability to recognize the intended emotion in both types of nonverbal cues follows a divergent developmental trajectory throughout childhood and adolescence. The current study a) identified regions of common neural activation to facial and vocal stimuli in 8- to 19-year-old typically-developing adolescents, and b) examined age-related changes in blood-oxygen-level dependent (BOLD) response within these areas. Both modalities elicited activation in an overlapping network of subcortical regions (insula, thalamus, dorsal striatum), visual-motor association areas, prefrontal regions (inferior frontal cortex, dorsomedial prefrontal cortex), and the right superior temporal gyrus. Within these regions, increased age was associated with greater frontal activation to voices, but not faces. Results suggest that processing facial and vocal stimuli elicits activation in common areas of the brain in adolescents, but that age-related changes in response within these regions may vary by modality.

Comparing unfamiliar voice and face identity perception using identity sorting tasks

Identity sorting tasks, in which participants sort multiple naturally varying stimuli of usually two identities into perceived identities, have recently gained popularity in voice and face processing research. In both modalities, participants who are unfamiliar with the identities tend to perceive multiple stimuli of the same identity as different people and thus fail to "tell people together." These similarities across modalities suggest that modality-general mechanisms may underpin sorting behaviour. In this study, participants completed a voice sorting and a face sorting task. Taking an individual differences approach, we asked whether participants' performance on voice and face sorting of unfamiliar identities is correlated. Participants additionally completed a voice discrimination (Bangor Voice Matching Test) and a face discrimination task (Glasgow Face Matching Test). Using these tasks, we tested whether performance on sorting related to explicit identity discrimination. Performance on voice sorting and face sorting tasks was correlated, suggesting that common modality-general processes underpin these tasks. However, no significant correlations were found between sorting and discrimination performance, with the exception of significant relationships for performance on "same identity" trials with "telling people together" for voices and faces. Overall, any reported relationships were however relatively weak, suggesting the presence of additional modality-specific and task-specific processes.

Asymmetric visual representation of sex from human body shape

We efficiently infer others' states and traits from their appearance, and these inferences powerfully shape our social behaviour. One key trait is sex, which is strongly cued by the appearance of the body. What are the visual representations that link body shape to sex? Previous studies of visual sex judgment tasks find observers have a bias to report "male", particularly for ambiguous stimuli. This finding implies a representational asymmetry - that for the processes that generate a sex percept, the default output is "male", and "female" is determined by the presence of additional perceptual evidence. That is, female body shapes are positively coded by reference to a male default shape. This perspective makes a novel prediction in line with Treisman's studies of visual search asymmetries: female body targets should be more readily detected amongst male distractors than vice versa. Across 10 experiments (N = 32 each) we confirmed this prediction and ruled out alternative low-level explanations. The asymmetry was found with profile and frontal body silhouettes, frontal photographs, and schematised icons. Low-level confounds were controlled by balancing silhouette images for size and homogeneity, and by matching physical properties of photographs. The female advantage was nulled for inverted icons, but intact for inverted photographs, suggesting reliance on distinct cues to sex for different body depictions. Together, these findings demonstrate a principle of the perceptual coding that links bodily appearance with a significant social trait: the female body shape is coded as an extension of a male default. We conclude by offering a visual experience account of how these asymmetric representations arise in the first place.

Human and monkey infant attention to dynamic social and nonsocial stimuli

The present study explored behavioral norms for infant social attention in typically developing human and nonhuman primate infants. We examined the normative development of attention to dynamic social and nonsocial stimuli longitudinally in macaques (Macaca mulatta) at 1, 3, and 5 months of age (N = 75) and humans at 2, 4, 6, 8, and 13 months of age (N = 69) using eye tracking. All infants viewed concurrently played silent videos-one social video and one nonsocial video. Both macaque and human infants were faster to look to the social than the nonsocial stimulus, and both species grew faster to orient to the social stimulus with age. Further, macaque infants' social attention increased linearly from 1 to 5 months. In contrast, human infants displayed a nonlinear pattern of social interest, with initially greater attention to the social stimulus, followed by a period of greater interest in the nonsocial stimulus, and then a rise in social interest from 6 to 13 months. Overall, human infants looked longer than macaque infants, suggesting humans have more sustained attention in the first year of life. These findings highlight potential species similarities and differences, and reflect a first step in establishing baseline patterns of early social attention development.

Where Sounds Occur Matters: Context Effects Influence Processing of Salient Vocalisations

The social context in which a salient human vocalisation is heard shapes the affective information it conveys. However, few studies have investigated how visual contextual cues lead to differential processing of such vocalisations. The prefrontal cortex (PFC) is implicated in processing of contextual information and evaluation of saliency of vocalisations. Using functional Near-Infrared Spectroscopy (fNIRS), we investigated PFC responses of young adults (N = 18) to emotive infant and adult vocalisations while they passively viewed the scenes of two categories of environmental contexts: a domestic environment (DE) and an outdoors environment (OE). Compared to a home setting (DE) which is associated with a fixed mental representation (e.g., expect seeing a living room in a typical house), the outdoor setting (OE) is more variable and less predictable, thus might demand greater processing effort. From our previous study in Azhari et al. (2018) that employed the same experimental paradigm, the OE context was found to elicit greater physiological arousal compared to the DE context. Similarly, we hypothesised that greater PFC activation will be observed when salient vocalisations are paired with the OE compared to the DE condition. Our finding supported this hypothesis: the left rostrolateral PFC, an area of the brain that facilitates relational integration, exhibited greater activation in the OE than DE condition which suggests that greater cognitive resources are required to process outdoor situational information together with salient vocalisations. The result from this study bears relevance in deepening our understanding of how contextual information differentially modulates the processing of salient vocalisations.

Unconscious and Distinctive Control of Vocal Pitch and Timbre During Altered Auditory Feedback

Vocal control plays a critical role in smooth social communication. Speakers constantly monitor auditory feedback (AF) and make adjustments when their voices deviate from their intentions. Previous studies have shown that when certain acoustic features of the AF are artificially altered, speakers compensate for this alteration in the opposite direction. However, little is known about how the vocal control system implements compensations for alterations of different acoustic features, and associates them with subjective consciousness. The present study investigated whether compensations for the fundamental frequency (F0), which corresponds to perceived pitch, and formants, which contribute to perceived timbre, can be performed unconsciously and independently. Forty native Japanese speakers received two types of altered AF during vowel production that involved shifts of either only the formant frequencies (formant modification; Fm) or both the pitch and formant frequencies (pitch + formant modification; PFm). For each type, three levels of shift (slight, medium, and severe) in both directions (increase or decrease) were used. After the experiment, participants were tested for whether they had perceived a change in the F0 and/or formants. The results showed that (i) only formants were compensated for in the Fm condition, while both the F0 and formants were compensated for in the PFm condition; (ii) the F0 compensation exhibited greater precision than the formant compensation in PFm; and (iii) compensation occurred even when participants misperceived or could not explicitly perceive the alteration in AF. These findings indicate that non-experts can compensate for both formant and F0 modifications in the AF during vocal production, even when the modifications are not explicitly or correctly perceived, which provides further evidence for a dissociation between conscious perception and action in vocal control. We propose that such unconscious control of voice production may enhance rapid adaptation to changing speech environments and facilitate mutual communication.