GRAPES-Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain

Innate network mechanisms of temporal pole for semantic cognition in neonatal and adult twin studies

What are the innate neural mechanisms scaffolding the protracted development of sophisticated human cognition observable later in life? We investigate this question by focusing on the putative hub of the human semantic memory system-the temporal pole. Combining infant- and twin-based imaging analyses, we examine the ontogenetic mechanisms and network characteristics of the functional subdivisions within the temporal pole that are specialized for semantic processing of different types in adults. Our findings reveal topologically similar temporal pole parcellations in the adult and neonatal brains. Notably, the specific functional connectivity of the dorsal and ventrolateral subdivisions with semantic-related networks are evident in neonates, significantly heritable, and associated with semantic functions in adult twins. These results demonstrate the neonatal emergence of genetically programmed functional connectivity characteristics in the temporal pole parcellations that underlie its crucial role in semantic processing, highlighting the innate network mechanisms that support semantic cognition in humans.

Neural specialization for 'visual' concepts emerges in the absence of vision

The 'different-body/different-concepts hypothesis' central to some embodiment theories proposes that the sensory capacities of our bodies shape the cognitive and neural basis of our concepts. We tested this hypothesis by comparing behavioral semantic similarity judgments and neural signatures (fMRI) of 'visual' categories ('living things,' or animals, e.g., tiger, and light events, e.g., sparkle) across congenitally blind (n = 21) and sighted (n = 22) adults. Words referring to 'visual' entities/nouns and events/verbs (animals and light events) were compared to less vision-dependent categories from the same grammatical class (animal vs. place nouns, light vs. sound, mouth, and hand verbs). Within-category semantic similarity judgments about animals (e.g., sparrow vs. finch) were partially different across groups, consistent with the idea that sighted people rely on visually learned information to make such judgments about animals. However, robust neural specialization for living things in temporoparietal semantic networks, including in the precuneus, was observed in blind and sighted people alike. For light events, which are directly accessible only through vision, behavioral judgments were indistinguishable across groups. Neural responses to light events were also similar across groups: in both blind and sighted people, the left middle temporal gyrus (LMTG+) responded more to event concepts, including light events, compared to entity concepts. Multivariate patterns of neural activity in LMTG+ distinguished among different event types, including light events vs. other event types. In sum, we find that neural signatures of concepts previously attributed to visual experience do not require vision. Across a wide range of semantic types, conceptual representations develop independent of sensory experience.

Neural mechanisms for the attention-mediated propagation of conceptual information in the human brain

The visual environment is complicated, and humans and other animals accordingly prioritize some sources of information over others through the deployment of spatial attention. Cognitive theories propose that one core purpose of this is to gather information that can be used in downstream cognitive processes, including the development of concepts and categories. However, neuroscientific investigation has focused closely on the identification of the systems and algorithms that support attentional control or that instantiate the effect of attention on sensation and perception. Much less is known about how attention impacts the acquisition and activation of concepts. Here, we use machine learning of EEG and concurrently recorded EEG/MRI to temporally and anatomically characterize the neural network that abstracts from attended perceptual information to activate and construct semantic and conceptual representations. We find that variance in the amplitude of N2pc-an event-related potential (ERP) component closely linked to selective attention-predicts the emergence of conceptual information in a network including VMPFC, posterior parietal cortex, and anterior insula. This network appears to play a key role in the attention-mediated translation of perceptual information to concepts, semantics, and action plans.

Semantic language decoding across participants and stimulus modalities

Brain decoders that reconstruct language from semantic representations have the potential to improve communication for people with impaired language production. However, training a semantic decoder for a participant currently requires many hours of brain responses to linguistic stimuli, and people with impaired language production often also have impaired language comprehension. In this study, we tested whether language can be decoded from a goal participant without using any linguistic training data from that participant. We trained semantic decoders on brain responses from separate reference participants and then used functional alignment to transfer the decoders to the goal participant. Cross-participant decoder predictions were semantically related to the stimulus words, even when functional alignment was performed using movies with no linguistic content. To assess how much semantic representations are shared between language and vision, we compared functional alignment accuracy using story and movie stimuli and found that performance was comparable in most cortical regions. Finally, we tested whether cross-participant decoders could be robust to lesions by excluding brain regions from the goal participant prior to functional alignment and found that cross-participant decoders do not depend on data from any single brain region. These results demonstrate that cross-participant decoding can reduce the amount of linguistic training data required from a goal participant and potentially enable language decoding from participants who struggle with both language production and language comprehension.

Decoding the physics of observed actions in the human brain

Recognizing goal-directed actions is a computationally challenging task, requiring not only the visual analysis of body movements, but also analysis of how these movements causally impact, and thereby induce a change in, those objects targeted by an action. We tested the hypothesis that the analysis of body movements and the effects they induce relies on distinct neural representations in superior and anterior inferior parietal lobe (SPL and aIPL). In four fMRI sessions, participants observed videos of actions (e.g. breaking stick, squashing plastic bottle) along with corresponding point-light-display (PLD) stick figures, pantomimes, and abstract animations of agent-object interactions (e.g. dividing or compressing a circle). Cross-decoding between actions and animations revealed that aIPL encodes abstract representations of action effect structures independent of motion and object identity. By contrast, cross-decoding between actions and PLDs revealed that SPL is disproportionally tuned to body movements independent of visible interactions with objects. Lateral occipitotemporal cortex (LOTC) was sensitive to both action effects and body movements. These results demonstrate that parietal cortex and LOTC are tuned to physical action features, such as how body parts move in space relative to each other and how body parts interact with objects to induce a change (e.g. in position or shape/configuration). The high level of abstraction revealed by cross-decoding suggests a general neural code supporting mechanical reasoning about how entities interact with, and have effects on, each other.

What we mean when we say semantic: Toward a multidisciplinary semantic glossary

Tulving characterized semantic memory as a vast repository of meaning that underlies language and many other cognitive processes. This perspective on lexical and conceptual knowledge galvanized a new era of research undertaken by numerous fields, each with their own idiosyncratic methods and terminology. For example, "concept" has different meanings in philosophy, linguistics, and psychology. As such, many fundamental constructs used to delineate semantic theories remain underspecified and/or opaque. Weak construct specificity is among the leading causes of the replication crisis now facing psychology and related fields. Term ambiguity hinders cross-disciplinary communication, falsifiability, and incremental theory-building. Numerous cognitive subdisciplines (e.g., vision, affective neuroscience) have recently addressed these limitations via the development of consensus-based guidelines and definitions. The project to follow represents our effort to produce a multidisciplinary semantic glossary consisting of succinct definitions, background, principled dissenting views, ratings of agreement, and subjective confidence for 17 target constructs (e.g., abstractness, abstraction, concreteness, concept, embodied cognition, event semantics, lexical-semantic, modality, representation, semantic control, semantic feature, simulation, semantic distance, semantic dimension). We discuss potential benefits and pitfalls (e.g., implicit bias, prescriptiveness) of these efforts to specify a common nomenclature that other researchers might index in specifying their own theoretical perspectives (e.g., They said X, but I mean Y).

Population encoding of observed and actual somatosensations in the human posterior parietal cortex

Cognition relies on transforming sensory inputs into a generalizable understanding of the world. Mirror neurons have been proposed to underlie this process, mapping visual representations of others' actions and sensations onto neurons that mediate our own, providing a conduit for understanding. However, this theory has limitations. Here, we hypothesize that mirror-like responses represent one facet of a broader framework in which our brains engage internal models for cognition. We recorded populations of single neurons in the human posterior parietal cortex (PPC) of a brain-machine interface clinical trial participant implanted with a microelectrode array while she either experienced actual touch, or observed diverse tactile stimuli applied to other individuals. Two body locations were tested, on each of the participant and other individuals. Some neurons exhibited mirror-like properties, consistent with earlier literature. However, they were fragile, breaking with increased task complexity. Population responses were better characterized by generalizable and compositional basic-level features encoded within neural subspaces. These features enable the population to respond to diverse actual and observed touch stimuli and are recruited similarly for similar forms of touch. Mirror-like neurons belong within these subspaces, contributing more globally to compositionality and generalizability. We speculate that at a population-level, human PPC manifests an internal model for touch, and that cognition unfolds in the high-level human cortex by versatility in its representational building blocks. In a broad sense, we speculate that the population features we demonstrate support a broad mechanism by which the high-level human cortex enables understanding.

Distributed Cortical Regions for the Recall of People, Places, and Objects

The human medial parietal cortex (MPC) is recruited during multiple cognitive processes. Previously, we demonstrated regions specific to recall of people or places and proposed that the functional organization of MPC mirrors the category selectivity defining the medial-lateral axis of the ventral-temporal cortex (VTC). However, prior work considered recall of people and places only, and VTC also shows object selectivity sandwiched between face- and scene-selective regions. Here, we tested a strong prediction of our proposal: like VTC, MPC should show a region specifically recruited during object recall, and its relative cortical position should mirror the one of VTC. While responses during people and place recall showed a striking replication of prior findings, we did not observe any evidence for object-recall effects within MPC, which differentiates it from the spatial organization in VTC. Importantly, beyond MPC, robust recall effects were observed for people, places, and objects on the lateral surface of the brain. Place-recall effects were present in the angular gyrus, frontal eye fields, and peripheral portions of the early visual cortex, whereas people recall selectively drove response in the right posterior superior temporal sulcus. Object-recall effects were largely restricted to a region posterior to the left somatosensory cortex, in the vicinity of the supramarginal gyrus. Taken together, these data demonstrate that while there are distributed regions active during recall of people, places, and objects, the functional organization of MPC does not mirror the medial-lateral axis of VTC but reflects only the most salient features of that axis-namely, representations of people and places.

Evaluating semantic control with transcranial magnetic stimulation: a systematic review with meta-analysis

Background

This meta-analysis investigates the role of specific brain regions in semantic control processes using Transcranial Magnetic Stimulation (TMS). According to the Controlled Semantic Cognition framework, control processes help manage the contextually appropriate retrieval of semantic information by activating a distributed neural network, including the inferior frontal gyrus, the posterior middle temporal gyrus, and inferior parietal lobule. Lesions in these areas can lead to difficulties in manipulating weakly activated or competing semantic information. Researchers have used TMS to simulate such deficits in healthy individuals.

Method

By synthesizing results from TMS studies that targeted these regions, we aimed to evaluate whether neurostimulation over these areas can effectively impair participants' performance under high semantic control demands.

Results

Results from different meta-analytical approaches consistently showed no significant effects of TMS, especially after correcting for publication bias. Nevertheless, variability in experimental methodologies was evident.

Conclusion

These findings raise questions about the effectiveness of TMS in simulating deficits in semantic control and highlight the need for methodological improvements in future studies to enhance reliability and interpretability.

Language proficiency is associated with neural representational dimensionality of semantic concepts

Previous studies suggest that semantic concepts are characterized by high-dimensional neural representations and that language proficiency affects semantic processing. However, it is not clear whether language proficiency modulates the dimensional representations of semantic concepts at the neural level. To address this question, the present study adopted principal component analysis (PCA) and representational similarity analysis (RSA) to examine the differences in representational dimensionalities (RDs) and in semantic representations between words in highly proficient (Chinese) and less proficient (English) language. PCA results revealed that language proficiency increased the dimensions of lexical representations in the left inferior frontal gyrus, temporal pole, inferior temporal gyrus, supramarginal gyrus, angular gyrus, and fusiform gyrus. RSA results further showed that these regions represented semantic information and that higher semantic representations were observed in highly proficient language relative to less proficient language. These results suggest that language proficiency is associated with the neural representational dimensionality of semantic concepts.

Context, transparency and culture in motor resonance phenomena: Causal evidence of the motor cortex

A connection between language and movement information in metaphorical and literal expressions activates the motor system. Despite numerous studies exploring distinctions between idioms and metaphors, a notable research gap remains in the specific effect of idioms with different transparency levels concerning motor resonance. Our primary focus was analysing the functional role of the primary motor cortex (M1) in processing hand motor verbs both in literal expressions and in two idiomatic contexts, i.e., opaque and transparent idioms. Additionally, we explored a potential language and cultural effect by comparing Turkish and Spanish speakers. An overt priming task with self-paced reading was used to judge the relatedness of a primer and a target sentence. We implemented a repetitive transcranial magnetic stimulation (TMS) protocol using continuous theta-burst stimulation (cTBS) compared to sham stimulation over the M1 in both Turkish and Spanish native speakers prior to the experimental task. Our findings reveal that the performance of Turkish and Spanish participants in processing hand motor actions was facilitated after the application of cTBS over the left M1. Moreover, brain stimulation specifically facilitated the processing of only transparent-but not opaque-idioms in both Spanish and Turkish participants. Our study reports distinct motor resonance results between different types of idioms with a parallel cross-cultural effect.

Object color knowledge representation occurs in the macaque brain despite the absence of a developed language system

Animals guide their behaviors through internal representations of the world in the brain. We aimed to understand how the macaque brain stores such general world knowledge, focusing on object color knowledge. Three functional magnetic resonance imaging (fMRI) experiments were conducted in macaque monkeys: viewing chromatic and achromatic gratings, viewing grayscale images of their familiar fruits and vegetables (e.g., grayscale strawberry), and viewing true- and false-colored objects (e.g., red strawberry and green strawberry). We observed robust object knowledge representations in the color patches, especially the one located around TEO: the activity patterns could classify grayscale pictures of objects based on their memory color and response patterns in these regions could translate between chromatic grating viewing and grayscale object viewing (e.g., red grating-grayscale images of strawberry), such that classifiers trained by viewing chromatic gratings could successfully classify grayscale object images according to their memory colors. Our results showed direct positive evidence of object color memory in macaque monkeys. These results indicate the perceptually grounded knowledge representation as a conservative memory mechanism and open a new avenue to study this particular (semantic) memory representation with macaque models.

Impaired processing of conspecifics in Parkinson's disease

Experimental evidence indicates that the inferior frontal gyrus (IFG) processes emotional/affective features crucial to elaborate knowledge about social groups and that knowledge of social concepts is stored in the anterior temporal lobe (ATL).We investigated whether knowledge about social groups is impaired in Parkinson's disease (PD), in which dysfunctional connectivity between IFG and ATL has been demonstrated.PD patients (N = 20) and healthy controls (HC, N = 16) were given a lexical decision task in a semantic priming paradigm: the prime-targets included 144 words and 144 pseudowords, each preceded by three types of prime ("animals," "things," "persons"). Out of these 288 prime-targets, forty-eight were congruent (same category) and 96 incongruent (different category). Out of 48 congruent prime-targets, 24 denoted social items and 24 nonsocial items. Thus, four types of trials were obtained: congruent social; congruent nonsocial; incongruent social; incongruent nonsocial.Congruent target-words were recognized better than incongruent target-words by all groups. The semantic priming effect was preserved in PD; however, accuracy was significantly lower in PD than in HC in social items. No difference emerged between the two groups in nonsocial items.Impaired processing of words denoting social groups in PD may be due to impairment in accessing the affective/emotional features that characterize conceptual knowledge of social groups, for the functional disconnection between the IFG and the ATL.

Development and validation of a nonverbal consensus-based semantic memory paradigm in patients with epilepsy

OBJECTIVE

Brain areas implicated in semantic memory can be damaged in patients with epilepsy (PWE). However, it is challenging to delineate semantic processing deficits from acoustic, linguistic, and other verbal aspects in current neuropsychological assessments. We developed a new Visual-based Semantic Association Task (ViSAT) to evaluate nonverbal semantic processing in PWE.

METHOD

The ViSAT was adapted from similar predecessors (Pyramids & Palm Trees test, PPT; Camels & Cactus Test, CCT) comprised of 100 unique trials using real-life color pictures that avoid demographic, cultural, and other potential confounds. We obtained performance data from 23 PWE participants and 24 control participants (Control), along with crowdsourced normative data from 54 Amazon Mechanical Turk (Mturk) workers.

RESULTS

ViSAT reached a consensus >90% in 91.3% of trials compared to 83.6% in PPT and 82.9% in CCT. A deep learning model demonstrated that visual features of the stimulus images (color, shape; i.e., non-semantic) did not influence top answer choices (p = 0.577). The PWE group had lower accuracy than the Control group (p = 0.019). PWE had longer response times than the Control group in general and this was augmented for the semantic processing (trial answer) stage (both p < 0.001).

CONCLUSIONS

This study demonstrated performance impairments in PWE that may reflect dysfunction of nonverbal semantic memory circuits, such as seizure onset zones overlapping with key semantic regions (e.g., anterior temporal lobe). The ViSAT paradigm avoids confounds, is repeatable/longitudinal, captures behavioral data, and is open-source, thus we propose it as a strong alternative for clinical and research assessment of nonverbal semantic memory.

Decoding familiar visual object categories in the mu rhythm oscillatory response

Whilst previous research has linked attenuation of the mu rhythm to the observation of specific visual categories, and even to a potential role in action observation via a putative mirror neuron system, much of this work has not considered what specific type of information might be coded in this oscillatory response when triggered via vision. Here, we sought to determine whether the mu rhythm contains content-specific information about the identity of familiar (and also unfamiliar) graspable objects. In the present study, right-handed participants (N = 27) viewed images of both familiar (apple, wine glass) and unfamiliar (cubie, smoothie) graspable objects, whilst performing an orthogonal task at fixation. Multivariate pattern analysis (MVPA) revealed significant decoding of familiar, but not unfamiliar, visual object categories in the mu rhythm response. Thus, simply viewing familiar graspable objects may automatically trigger activation of associated tactile and/or motor properties in sensorimotor areas, reflected in the mu rhythm. In addition, we report significant attenuation in the central beta band for both familiar and unfamiliar visual objects, but not in the mu rhythm. Our findings highlight how analysing two different aspects of the oscillatory response - either attenuation or the representation of information content - provide complementary views on the role of the mu rhythm in response to viewing graspable object categories.

Mapping semantic space: Exploring the higher-order structure of word meaning

Multiple representation theories posit that concepts are represented via a combination of properties derived from sensorimotor, affective, and linguistic experiences. Recently, it has been proposed that information derived from social experience, or socialness, represents another key aspect of conceptual representation. How these various dimensions interact to form a coherent conceptual space has yet to be fully explored. To address this, we capitalized on openly available word property norms for 6339 words and conducted a large-scale investigation into the relationships between 18 dimensions. An exploratory factor analysis reduced the dimensions to six higher-order factors: sub-lexical, distributional, visuotactile, body action, affective and social interaction. All these factors explained unique variance in performance on lexical and semantic tasks, demonstrating that they make important contributions to the representation of word meaning. An important and novel finding was that the socialness dimension clustered with the auditory modality and with mouth and head actions. We suggest this reflects experiential learning from verbal interpersonal interactions. Moreover, formally modelling the network structure of semantic space revealed pairwise partial correlations between most dimensions and highlighted the centrality of the interoception dimension. Altogether, these findings provide new insights into the architecture of conceptual space, including the importance of inner and social experience, and highlight promising avenues for future research.

The concrete processing of Chinese action metaphors: an ERP study

The present research adopts ERP (Event-Related Potentials) technology to investigate whether there exists a concreteness effect in the processing of Chinese action verbs within metaphorical context. The mean amplitudes of N400 activated by action metaphors were compared with those activated by literal verbs and abstract verbs. The findings indicated that the Met verbs evoked a significantly larger N400 response at frontal brain region compared to the Abs verbs at a time window 200-500 ms, while the Met verbs elicited a notably greater N400 amplitude specifically at the posterior brain region in comparison to the Lit verbs at 300-500 ms time window. These results may be interpreted as indicating that the comprehension of the Met verbs is based on the concrete action semantics.

Graded and sharp transitions in semantic function in left temporal lobe

Recent work has focussed on how patterns of functional change within the temporal lobe relate to whole-brain dimensions of intrinsic connectivity variation (Margulies et al., 2016). We examined two such 'connectivity gradients' reflecting the separation of (i) unimodal versus heteromodal and (ii) visual versus auditory-motor cortex, examining visually presented verbal associative and feature judgments, plus picture-based context and emotion generation. Functional responses along the first dimension sometimes showed graded change between modality-tuned and heteromodal cortex (in the verbal matching task), and other times showed sharp functional transitions, with deactivation at the extremes and activation in the middle of this gradient (internal generation). The second gradient revealed more visual than auditory-motor activation, regardless of content (associative, feature, context, emotion) or task process (matching/generation). We also uncovered subtle differences across each gradient for content type, which predominantly manifested as differences in relative magnitude of activation or deactivation.

White matter network underlying semantic processing: evidence from stroke patients

The hub-and-spoke theory of semantic representation fractionates the neural underpinning of semantic knowledge into two essential components: the sensorimotor modality-specific regions and a crucially important semantic hub region. Our previous study in patients with semantic dementia has found that the hub region is located in the left fusiform gyrus. However, because this region is located within the brain damage in patients with semantic dementia, it is not clear whether the semantic deficit is caused by structural damage to the hub region itself or by its disconnection from other brain regions. Stroke patients do not have any damage to the left fusiform gyrus, but exhibit amodal and modality-specific deficits in semantic processing. Therefore, in this study, we validated the semantic hub region from a brain network perspective in 79 stroke patients and explored the white matter connections associated with it. First, we collected data of diffusion-weighted imaging and behavioural performance on general semantic tasks and modality-specific semantic tasks (assessing object knowledge on form, colour, motion, sound, manipulation and function). We then used correlation and regression analyses to examine the association between the nodal degree values of brain regions in the whole-brain structural network and general semantic performance in the stroke patients. The results revealed that the connectivity of the left fusiform gyrus significantly predicted general semantic performance, indicating that this region is the semantic hub. To identify the semantic-relevant connections of the semantic hub, we then correlated the white matter integrity values of each tract connected to the left fusiform gyrus separately with performance on general and modality-specific semantic processing. We found that the hub region accomplished general semantic processing through white matter connections with the left superior temporal pole, middle temporal gyrus, inferior temporal gyrus and hippocampus. The connectivity between the hub region and the left hippocampus, superior temporal pole, middle temporal gyrus, inferior temporal gyrus and parahippocampal gyrus was differentially involved in object form, colour, motion, sound, manipulation and function processing. After statistically removing the effects of potential confounding variables (i.e. whole-brain lesion volume, lesion volume of regions of interest and performance on non-semantic control tasks), the observed effects remained significant. Together, our findings support the role of the left fusiform gyrus as a semantic hub region in stroke patients and reveal its crucial connectivity in the network. This study provides new insights and evidence for the neuroanatomical organization of semantic memory in the human brain.

Syntax matters in shaping sensorimotor activation driven by nouns

Existing evidence has shown that adjectives modulate the grasp-compatibility effect elicited by object nouns. The aim of the present study was to investigate the role of syntax on the sensorimotor activation elicited by nouns in a grasp-compatibility task. We assessed two languages with different syntactic rules, Italian in Experiment 1 and English in Experiment 2. In both experiments, an adjective-noun pair was shown on the screen. The adjective was always in a pre-nominal position and denoted either a disadvantageous quality of the object graspability (e.g., sharp) or the object colour (e.g., reddish). Participants had to categorize the object nouns as natural or artifact, performing a precision or a power reach-to-grasp movement. On different trials, the grasp response was compatible or incompatible with the grip typically used to manipulate the object indicated by the noun. In Experiment 1 (Italian language) the adjective-noun order violated the syntactic order and no difference emerged between reaction times on compatible and incompatible trials (no grasp compatibility effect). In Experiment 2 (English language), the adjective-noun order followed the syntactic rule. Results showed a grasp-compatibility effect when a colour adjective was presented before a natural object noun. When a disadvantageous adjective preceded an artifact or a natural object noun, an inverted grasp-compatibility effect emerged with slower responses on compatible than incompatible trials. Taken together, these findings suggest that adjectives can shape the sensorimotor activation elicited by nouns of graspable objects only when the syntax is correct. Results are discussed with respect to embodied cognition theories.

Unraveling lexical semantics in the brain: Comparing internal, external, and hybrid language models

To explain how the human brain represents and organizes meaning, many theoretical and computational language models have been proposed over the years, varying in their underlying computational principles and in the language samples based on which they are built. However, how well they capture the neural encoding of lexical semantics remains elusive. We used representational similarity analysis (RSA) to evaluate to what extent three models of different types explained neural responses elicited by word stimuli: an External corpus-based word2vec model, an Internal free word association model, and a Hybrid ConceptNet model. Semantic networks were constructed using word relations computed in the three models and experimental stimuli were selected through a community detection procedure. The similarity patterns between language models and neural responses were compared at the community, exemplar, and word node levels to probe the potential hierarchical semantic structure. We found that semantic relations computed with the Internal model provided the closest approximation to the patterns of neural activation, whereas the External model did not capture neural responses as well. Compared with the exemplar and the node levels, community-level RSA demonstrated the broadest involvement of brain regions, engaging areas critical for semantic processing, including the angular gyrus, superior frontal gyrus and a large portion of the anterior temporal lobe. The findings highlight the multidimensional semantic organization in the brain which is better captured by Internal models sensitive to multiple modalities such as word association compared with External models trained on text corpora.

Brain encoding models based on multimodal transformers can transfer across language and vision

Encoding models have been used to assess how the human brain represents concepts in language and vision. While language and vision rely on similar concept representations, current encoding models are typically trained and tested on brain responses to each modality in isolation. Recent advances in multimodal pretraining have produced transformers that can extract aligned representations of concepts in language and vision. In this work, we used representations from multimodal transformers to train encoding models that can transfer across fMRI responses to stories and movies. We found that encoding models trained on brain responses to one modality can successfully predict brain responses to the other modality, particularly in cortical regions that represent conceptual meaning. Further analysis of these encoding models revealed shared semantic dimensions that underlie concept representations in language and vision. Comparing encoding models trained using representations from multimodal and unimodal transformers, we found that multimodal transformers learn more aligned representations of concepts in language and vision. Our results demonstrate how multimodal transformers can provide insights into the brain's capacity for multimodal processing.

No need to integrate action information during coarse semantic processing of man-made tools

Action representation of man-made tools consists of two subtypes: structural action representation concerning how to grasp an object, and functional action representation concerning the skilled use of an object. Compared to structural action representation, functional action representation plays the dominant role in fine-grained (i.e., basic level) object recognition. However, it remains unclear whether the two types of action representation are involved differently in the coarse semantic processing in which the object is recognized at a superordinate level (i.e., living/non-living). Here we conducted three experiments using the priming paradigm, in which video clips displaying structural and functional action hand gestures were used as prime stimuli and grayscale photos of man-made tools were used as target stimuli. Participants recognized the target objects at the basic level in Experiment 1 (i.e., naming task) and at the superordinate level in Experiments 2 and 3 (i.e., categorization task). We observed a significant priming effect for functional action prime-target pairs only in the naming task. In contrast, no priming effect was found in either the naming or the categorization task for the structural action prime-target pairs (Experiment 2), even when the categorization task was preceded by a preliminary action imitation of the prime gestures (Experiment 3). Our results suggest that only functional action information is retrieved during fine-grained object processing. In contrast, coarse semantic processing does not require the integration of either structural or functional action information.

Lateralized Learning to Solve Complex Boolean Problems

Modern classifier systems can effectively classify targets that consist of simple patterns. However, they can fail to detect hierarchical patterns of features that exist in many real-world problems, such as understanding speech or recognizing object ontologies. Biological nervous systems have the ability to abstract knowledge from simple and small-scale problems in order to then apply it to resolve more complex problems in similar and related domains. It is thought that lateral asymmetry of biological brains allows modular learning to occur at different levels of abstraction, which can then be transferred between tasks. This work develops a novel evolutionary machine-learning (EML) system that incorporates lateralization and modular learning at different levels of abstraction. The results of analyzable Boolean tasks show that the lateralized system has the ability to encapsulate underlying knowledge patterns in the form of building blocks of knowledge (BBK). Lateralized abstraction transforms complex problems into simple ones by reusing general patterns (e.g., any parity problem becomes a sequence of the 2-bit parity problem). By enabling abstraction in evolutionary computation, the lateralized system is able to identify complex patterns (e.g., in hierarchical multiplexer (HMux) problems) better than existing systems.

Spatiotemporal characteristics of the neural representation of event concepts

Events are a fundamentally important part of our understanding of the world. How lexical concepts denoting events are represented in the brain remains controversial. We conducted two experiments using event and object nouns matched on a range of psycholinguistic variables, including concreteness, to examine spatial and temporal characteristics of event concepts. Both experiments used magnitude and valence tasks on event and object nouns. The fMRI experiment revealed a distributed set of regions for events, including the angular gyrus, anterior temporal lobe, and posterior cingulate across tasks. In the EEG experiment, events and objects differed in amplitude within the 300-500 ms window. Together these results shed light into the spatiotemporal characteristics of event concept representation and show that event concepts are represented in the putative hubs of the semantic system. While these hubs are typically associated with object semantics, they also represent events, and have a likely role in temporal integration.

Neurobiological mechanisms for language, symbols and concepts: Clues from brain-constrained deep neural networks

Neural networks are successfully used to imitate and model cognitive processes. However, to provide clues about the neurobiological mechanisms enabling human cognition, these models need to mimic the structure and function of real brains. Brain-constrained networks differ from classic neural networks by implementing brain similarities at different scales, ranging from the micro- and mesoscopic levels of neuronal function, local neuronal links and circuit interaction to large-scale anatomical structure and between-area connectivity. This review shows how brain-constrained neural networks can be applied to study in silico the formation of mechanisms for symbol and concept processing and to work towards neurobiological explanations of specifically human cognitive abilities. These include verbal working memory and learning of large vocabularies of symbols, semantic binding carried by specific areas of cortex, attention focusing and modulation driven by symbol type, and the acquisition of concrete and abstract concepts partly influenced by symbols. Neuronal assembly activity in the networks is analyzed to deliver putative mechanistic correlates of higher cognitive processes and to develop candidate explanations founded in established neurobiological principles.

Task-based activation and resting-state connectivity predict individual differences in semantic capacity for complex semantic knowledge

Our ability to know and access complex factual information has far reaching effects, influencing our scholastic, professional and social lives. Here we employ functional MRI to assess the relationship between individual differences in semantic aptitude in the task-based activation and resting-state functional connectivity. Using psychometric and behavioural measures, we quantified the semantic and executive aptitude of individuals and had them perform a general-knowledge semantic-retrieval task (N = 41) and recorded resting-state data (N = 43). During the semantic-retrieval task, participants accessed general-knowledge facts drawn from four different knowledge-domains (people, places, objects and 'scholastic'). Individuals with greater executive capacity more strongly recruit anterior sections of prefrontal cortex (PFC) and the precuneus, and individuals with lower semantic capacity more strongly activate a posterior section of the dorsomedial PFC (dmPFC). The role of these regions in semantic processing was validated by analysis of independent resting-state data, where increased connectivity between a left anterior PFC and the precuneus predict higher semantic aptitude, and increased connectivity between left anterior PFC and posterior dmPFC predict lower semantic aptitude. Results suggest that coordination between core semantic regions in the precuneus and anterior prefrontal regions associated with executive processes support greater semantic aptitude.

Immediate neural impact and incomplete compensation after semantic hub disconnection

The human brain extracts meaning using an extensive neural system for semantic knowledge. Whether broadly distributed systems depend on or can compensate after losing a highly interconnected hub is controversial. We report intracranial recordings from two patients during a speech prediction task, obtained minutes before and after neurosurgical treatment requiring disconnection of the left anterior temporal lobe (ATL), a candidate semantic knowledge hub. Informed by modern diaschisis and predictive coding frameworks, we tested hypotheses ranging from solely neural network disruption to complete compensation by the indirectly affected language-related and speech-processing sites. Immediately after ATL disconnection, we observed neurophysiological alterations in the recorded frontal and auditory sites, providing direct evidence for the importance of the ATL as a semantic hub. We also obtained evidence for rapid, albeit incomplete, attempts at neural network compensation, with neural impact largely in the forms stipulated by the predictive coding framework, in specificity, and the modern diaschisis framework, more generally. The overall results validate these frameworks and reveal an immediate impact and capability of the human brain to adjust after losing a brain hub.

Grounded Cognition, Linguistic Relativity, and Abstract Concepts

Kemmerer's paper convincingly claims that the grounded cognition model (GCM) entails linguistic relativity. Here, we underline that tackling linguistic relativity and cultural differences is vital for GCM. First, it allows GCM to focus more on flexible rather than stable aspects of cognition. Second, it highlights the centrality of linguistic experience for human cognition. While GCM-inspired research underscored the similarity between linguistic and nonlinguistic concepts, it is now paramount to understand when and how language(s) influence knowledge. To this aim, we argue that linguistic variation might be particularly relevant for more abstract concepts-which are more debatable and open to revisions.

Extreme image transformations affect humans and machines differently

Some recent artificial neural networks (ANNs) claim to model aspects of primate neural and human performance data. Their success in object recognition is, however, dependent on exploiting low-level features for solving visual tasks in a way that humans do not. As a result, out-of-distribution or adversarial input is often challenging for ANNs. Humans instead learn abstract patterns and are mostly unaffected by many extreme image distortions. We introduce a set of novel image transforms inspired by neurophysiological findings and evaluate humans and ANNs on an object recognition task. We show that machines perform better than humans for certain transforms and struggle to perform at par with humans on others that are easy for humans. We quantify the differences in accuracy for humans and machines and find a ranking of difficulty for our transforms for human data. We also suggest how certain characteristics of human visual processing can be adapted to improve the performance of ANNs for our difficult-for-machines transforms.

Concepts require flexible grounding

Research on semantic memory has a problem. On the one hand, a robust body of evidence implicates sensorimotor regions in conceptual processing. On the other hand, a different body of evidence implicates a modality independent semantic system. The standard solution to this tension is to posit a hub-and-spoke system with modality independent hubs and modality specific spokes. In this paper, I argue in support of an alternative view of grounding which remains committed to neural reenactment but emphasizes the multimodal and multilevel nature of the semantic system. This view is built upon the recognition that abstraction is a design feature of concepts. Semantic memory employs hierarchically structured representations to capture different degrees of abstraction. Grounding does not work the way that many embodied approaches have assumed.

Bigram semantic distance as an index of continuous semantic flow in natural language: Theory, tools, and applications

Much of our understanding of word meaning has been informed through studies of single words. High-dimensional semantic space models have recently proven instrumental in elucidating connections between words. Here we show how bigram semantic distance can yield novel insights into conceptual cohesion and topic flow when computed over continuous language samples. For example, "Cats drink milk" is comprised of an ordered vector of bigrams (cat-drink, drink-milk). Each of these bigrams has a unique semantic distance. These distances in turn may provide a metric of dispersion or the flow of concepts as language unfolds. We offer an R-package ("semdistflow") that transforms any user-specified language transcript into a vector of ordered bigrams, appending two metrics of semantic distance to each pair. We validated these distance metrics on a continuous stream of simulated verbal fluency data assigning predicted switch markers between alternating semantic clusters (animals, musical instruments, fruit). We then generated bigram distance norms on a large sample of text and demonstrated applications of the technique to a classic work of short fiction, To Build a Fire (London, 1908). In one application, we showed that bigrams spanning sentence boundaries are punctuated by jumps in the semantic distance. We discuss the promise of this technique for characterizing semantic processing in real-world narratives and for bridging findings at the single word level with macroscale discourse analyses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Similar object shape representation encoded in the inferolateral occipitotemporal cortex of sighted and early blind people

We can sense an object's shape by vision or touch. Previous studies suggested that the inferolateral occipitotemporal cortex (ILOTC) implements supramodal shape representations as it responds more to seeing or touching objects than shapeless textures. However, such activation in the anterior portion of the ventral visual pathway could be due to the conceptual representation of an object or visual imagery triggered by touching an object. We addressed these possibilities by directly comparing shape and conceptual representations of objects in early blind (who lack visual experience/imagery) and sighted participants. We found that bilateral ILOTC in both groups showed stronger activation during a shape verification task than during a conceptual verification task made on the names of the same manmade objects. Moreover, the distributed activity in the ILOTC encoded shape similarity but not conceptual association among objects. Besides the ILOTC, we also found shape representation in both groups' bilateral ventral premotor cortices and intraparietal sulcus (IPS), a frontoparietal circuit relating to object grasping and haptic processing. In contrast, the conceptual verification task activated both groups' left perisylvian brain network relating to language processing and, interestingly, the cuneus in early blind participants only. The ILOTC had stronger functional connectivity to the frontoparietal circuit than to the left perisylvian network, forming a modular structure specialized in shape representation. Our results conclusively support that the ILOTC selectively implements shape representation independently of visual experience, and this unique functionality likely comes from its privileged connection to the frontoparietal haptic circuit.

Time-Lagged Multidimensional Pattern Connectivity (TL-MDPC): An EEG/MEG pattern transformation based functional connectivity metric

Functional and effective connectivity methods are essential to study the complex information flow in brain networks underlying human cognition. Only recently have connectivity methods begun to emerge that make use of the full multidimensional information contained in patterns of brain activation, rather than unidimensional summary measures of these patterns. To date, these methods have mostly been applied to fMRI data, and no method allows vertex-to-vertex transformations with the temporal specificity of EEG/MEG data. Here, we introduce time-lagged multidimensional pattern connectivity (TL-MDPC) as a novel bivariate functional connectivity metric for EEG/MEG research. TL-MDPC estimates the vertex-to-vertex transformations among multiple brain regions and across different latency ranges. It determines how well patterns in ROI X at time point tx can linearly predict patterns of ROI Y at time point ty. In the present study, we use simulations to demonstrate TL-MDPC's increased sensitivity to multidimensional effects compared to a unidimensional approach across realistic choices of number of trials and signal-to-noise ratios. We applied TL-MDPC, as well as its unidimensional counterpart, to an existing dataset varying the depth of semantic processing of visually presented words by contrasting a semantic decision and a lexical decision task. TL-MDPC detected significant effects beginning very early on, and showed stronger task modulations than the unidimensional approach, suggesting that it is capable of capturing more information. With TL-MDPC only, we observed rich connectivity between core semantic representation (left and right anterior temporal lobes) and semantic control (inferior frontal gyrus and posterior temporal cortex) areas with greater semantic demands. TL-MDPC is a promising approach to identify multidimensional connectivity patterns, typically missed by unidimensional approaches.

Decoding semantic representations in mind and brain

A key goal for cognitive neuroscience is to understand the neurocognitive systems that support semantic memory. Recent multivariate analyses of neuroimaging data have contributed greatly to this effort, but the rapid development of these novel approaches has made it difficult to track the diversity of findings and to understand how and why they sometimes lead to contradictory conclusions. We address this challenge by reviewing cognitive theories of semantic representation and their neural instantiation. We then consider contemporary approaches to neural decoding and assess which types of representation each can possibly detect. The analysis suggests why the results are heterogeneous and identifies crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theories of semantic cognition.

Distinct but cooperating brain networks supporting semantic cognition

Semantic cognition is a complex multifaceted brain function involving multiple processes including sensory, semantic, and domain-general cognitive systems. However, it remains unclear how these systems cooperate with each other to achieve effective semantic cognition. Here, we used independent component analysis (ICA) to investigate the functional brain networks that support semantic cognition. We used a semantic judgment task and a pattern-matching control task, each with 2 levels of difficulty, to disentangle task-specific networks from domain-general networks. ICA revealed 2 task-specific networks (the left-lateralized semantic network [SN] and a bilateral, extended semantic network [ESN]) and domain-general networks including the frontoparietal network (FPN) and default mode network (DMN). SN was coupled with the ESN and FPN but decoupled from the DMN, whereas the ESN was synchronized with the FPN alone and did not show a decoupling with the DMN. The degree of decoupling between the SN and DMN was associated with semantic task performance, with the strongest decoupling for the poorest performing participants. Our findings suggest that human higher cognition is achieved by the multiple brain networks, serving distinct and shared cognitive functions depending on task demands, and that the neural dynamics between these networks may be crucial for efficient semantic cognition.

Different computational relations in language are captured by distinct brain systems

A critical way for humans to acquire information is through language, yet whether and how language experience drives specific neural semantic representations is still poorly understood. We considered statistical properties captured by 3 different computational principles of language (simple co-occurrence, network-(graph)-topological relations, and neural-network-vector-embedding relations) and tested the extent to which they can explain the neural patterns of semantic representations, measured by 2 functional magnetic resonance imaging experiments that shared common semantic processes. Distinct graph-topological word relations, and not simple co-occurrence or neural-network-vector-embedding relations, had unique explanatory power for the neural patterns in the anterior temporal lobe (capturing graph-common-neighbors), inferior frontal gyrus, and posterior middle/inferior temporal gyrus (capturing graph-shortest-path). These results were relatively specific to language: they were not explained by sensory-motor similarities and the same computational relations of visual objects (based on visual image database) showed effects in the visual cortex in the picture naming experiment. That is, different topological properties within language and the same topological computations (common-neighbors) for language and visual inputs are captured by different brain regions. These findings reveal the specific neural semantic representations along graph-topological properties of language, highlighting the information type-specific and statistical property-specific manner of semantic representations in the human brain.

Temporal features of concepts are grounded in time perception neural networks: An EEG study

Experimental evidence suggests that modality-specific concept features such as action, motion, and sound partially rely on corresponding action/perception neural networks in the human brain.Little is known, however, about time-related features of concepts. We examined whether temporal features of concepts recruit networks that subserve time perception in the brain in an EEG study using event and object nouns. Results showed significantly larger ERPs for event duration vs object size judgments over right parietal electrodes, a region associated with temporal processing. Additionally, alpha/beta (10-15 Hz) neural oscillation showed a stronger desynchronization for event duration compared to object size in the right parietal electrodes. This difference was not seen in control tasks comparing event vs object valence, suggesting that it is not likely to reflect a general difference between event and object nouns. These results indicate that temporal features of words may be subserved by time perception circuits in the human brain.

Meta-analytic evidence for a novel hierarchical model of conceptual processing

Conceptual knowledge plays a pivotal role in human cognition. Grounded cognition theories propose that concepts consist of perceptual-motor features represented in modality-specific perceptual-motor cortices. However, it is unclear whether conceptual processing consistently engages modality-specific areas. Here, we performed an activation likelihood estimation (ALE) meta-analysis across 212 neuroimaging experiments on conceptual processing related to 7 perceptual-motor modalities (action, sound, visual shape, motion, color, olfaction-gustation, and emotion). We found that conceptual processing consistently engages brain regions also activated during real perceptual-motor experience of the same modalities. In addition, we identified multimodal convergence zones that are recruited for multiple modalities. In particular, the left inferior parietal lobe (IPL) and posterior middle temporal gyrus (pMTG) are engaged for three modalities: action, motion, and sound. These "trimodal" regions are surrounded by "bimodal" regions engaged for two modalities. Our findings support a novel model of the conceptual system, according to which conceptual processing relies on a hierarchical neural architecture from modality-specific to multimodal areas up to an amodal hub.

Over-reliance on English hinders cognitive science

English is the dominant language in the study of human cognition and behavior: the individuals studied by cognitive scientists, as well as most of the scientists themselves, are frequently English speakers. However, English differs from other languages in ways that have consequences for the whole of the cognitive sciences, reaching far beyond the study of language itself. Here, we review an emerging body of evidence that highlights how the particular characteristics of English and the linguistic habits of English speakers bias the field by both warping research programs (e.g., overemphasizing features and mechanisms present in English over others) and overgeneralizing observations from English speakers' behaviors, brains, and cognition to our entire species. We propose mitigating strategies that could help avoid some of these pitfalls.

Anatomical correlates of category-selective visual regions have distinctive signatures of connectivity in neonates

The ventral visual stream is shaped during development by innate proto-organization within the visual system, such as the strong input from the fovea to the fusiform face area. In adults, category-selective regions have distinct signatures of connectivity to brain regions beyond the visual system, likely reflecting cross-modal and motoric associations. We tested if this long-range connectivity is part of the innate proto-organization, or if it develops with postnatal experience, by using diffusion-weighted imaging to characterize the connectivity of anatomical correlates of category-selective regions in neonates (N = 445), 1-9 month old infants (N = 11), and adults (N = 14). Using the HCP data we identified face- and place- selective regions and a third intermediate region with a distinct profile of selectivity. Using linear classifiers, these regions were found to have distinctive connectivity at birth, to other regions in the visual system and to those outside of it. The results support an extended proto-organization that includes long-range connectivity that shapes, and is shaped by, experience-dependent development.

Has the concept of systems consolidation outlived its usefulness? Identification and evaluation of premises underlying systems consolidation

Systems consolidation has mostly been treated as a neural construct defined by the time-dependent change in memory representation from the hippocampus (HPC) to other structures, primarily the neocortex. Here, we identify and evaluate the explicit and implicit premises that underlie traditional or standard models and theories of systems consolidation based on evidence from research on humans and other animals. We use the principle that changes in neural representation over time and experience are accompanied by corresponding changes in psychological representations, and vice versa, to argue that each of the premises underlying traditional or standard models and theories of systems consolidation is found wanting. One solution is to modify or abandon the premises or theories and models. This is reflected in moderated models of systems consolidation that emphasize the early role of the HPC in training neocortical memories until they stabilize. The fault, however, may lie in the very concept of systems consolidation and its defining feature. We propose that the concept be replaced by one of memory systems reorganization, which does not carry the theoretical baggage of systems consolidation and is flexible enough to capture the dynamic nature of memory from inception to very long-term retention and retrieval at a psychological and neural level. The term "memory system reorganization" implies that memory traces are not fixed, even after they are presumably consolidated. Memories can continue to change as a result of experience and interactions among memory systems across the lifetime. As will become clear, hippocampal training of neocortical memories is only one type of such interaction, and not always the most important one, even at inception. We end by suggesting some principles of memory reorganization that can help guide research on dynamic memory processes that capture corresponding changes in memory at the psychological and neural levels.

Distinct roles for the anterior temporal lobe and angular gyrus in the spatiotemporal cortical semantic network

Semantic knowledge is supported by numerous brain regions, but the spatiotemporal configuration of the network that links these areas remains an open question. The hub-and-spokes model posits that a central semantic hub coordinates this network. In this study, we explored distinct aspects that define a semantic hub, as reflected in the spatiotemporal modulation of neural activity and connectivity by semantic variables, from the earliest stages of semantic processing. We used source-reconstructed electro/magnetoencephalography, and investigated the concreteness contrast across three tasks. In a whole-cortex analysis, the left anterior temporal lobe (ATL) was the only area that showed modulation of evoked brain activity from 100 ms post-stimulus. Furthermore, using Dynamic Causal Modeling of the evoked responses, we investigated effective connectivity amongst the candidate semantic hub regions, that is, left ATL, supramarginal/angular gyrus (SMG/AG), middle temporal gyrus, and inferior frontal gyrus. We found that models with a single semantic hub showed the highest Bayesian evidence, and the hub region was found to change from ATL (within 250 ms) to SMG/AG (within 450 ms) over time. Our results support a single semantic hub view, with ATL showing sustained modulation of neural activity by semantics, and both ATL and AG underlying connectivity depending on the stage of semantic processing.

The action-sentence compatibility effect (ACE): Meta-analysis of a benchmark finding for embodiment

The embodied account of language comprehension has been one of the most influential theoretical developments in the recent decades addressing the question how humans comprehend and represent language. To examine its assumptions, many studies have made use of behavioral paradigms involving basic compatibility effects. The action-sentence compatibility effect (ACE) is one of the most influential of these compatibility effects and is the most widely cited evidence for the embodied account of language comprehension. However, recently there have been difficulties in extending or even in reliably replicating the ACE. The conflicting findings concerning the ACE and its extensions lead to the discussion of whether the ACE is indeed a reliable effect. In a first step we conducted a meta-analysis using a random-effects model. This analysis revealed a small but significant effect size of the ACE. Furthermore, the task-parameter Delay occurred as a factor of interest in whether the ACE appears with positive or negative effect direction. A second meta-analytic approach (Fisher's method) supports these findings. Additionally, an analysis of publication bias suggests that there is bias in the ACE literature. In post-hoc analyses of the recent multi-lab investigation of the ACE (Morey et al., 2021), evidence for individual differences in the ACE was found. However, further analyses indicate that these differences are likely due to item-specific variability and the specific way in which items were assigned to conditions in the counterbalancing lists.

Brain mechanisms linking language processing and open motor skill training

Given the discovery of a distributed language and motor functional network, surprisingly few studies have explored whether language processing is related to motor skill training. To address this issue, the present study used functional magnetic resonance imaging to compare whole-brain activation between nonexperts and experts in table tennis, an open skill sport in which players make rapid decisions in response to an ever-changing environment. Whole-brain activation was assessed in 30 expert table tennis players with more than 7 years’ experience and 35 age-matched nonexpert college students while they performed both a size and a semantic judgment task of words presented on a monitor. Compared with nonexperts, expert table tennis players showed greater activation in the left middle occipital gyrus and right precuneus while judging the size of the words versus during baseline fixation. They also showed greater activation in the left lingual gyrus during the semantic judgment task versus during baseline fixation. Our findings indicate that the visual regions engaged in language processing are associated with open motor skill training.

Taste Metaphors Ground Emotion Concepts Through the Shared Attribute of Valence

“Parting is such sweet sorrow.” Taste metaphors provide a rich vocabulary for describing emotional experience, potentially serving as an adaptive mechanism for conveying abstract emotional concepts using concrete verbal references to our shared experience. We theorized that the popularity of these expressions results from the close association with hedonic valence shared by these two domains of experience. To explore the possibility that this affective quality underlies the semantic similarity of these domains, we used a behavioral “odd-one-out” task in an online sample of 1059 participants in order to examine the semantic similarity of concepts related to emotion, taste, and color, another rich source of sensory metaphors. We found that the semantic similarity of emotion and taste concepts was greater than that of emotion and color concepts. Importantly, the similarity of taste and emotion concepts was strongly related to their similarity in hedonic valence, a relationship which was also significantly greater than that present between color and emotion. These results suggest that the common core of valence between taste and emotion concepts allows us to bridge the conceptual divide between our shared sensory environment and our internal emotional experience.

The role of the anterior temporal cortex in action: evidence from fMRI multivariate searchlight analysis during real object grasping

Intelligent manipulation of handheld tools marks a major discontinuity between humans and our closest ancestors. Here we identified neural representations about how tools are typically manipulated within left anterior temporal cortex, by shifting a searchlight classifier through whole-brain real action fMRI data when participants grasped 3D-printed tools in ways considered typical for use (i.e., by their handle). These neural representations were automatically evocated as task performance did not require semantic processing. In fact, findings from a behavioural motion-capture experiment confirmed that actions with tools (relative to non-tool) incurred additional processing costs, as would be suspected if semantic areas are being automatically engaged. These results substantiate theories of semantic cognition that claim the anterior temporal cortex combines sensorimotor and semantic content for advanced behaviours like tool manipulation.