Word meaning in the ventral visual path: a perceptual to conceptual gradient of semantic coding

Similar Computational Hierarchies for Reading and Speech in the Occipital Cortex of Sighed and Blind: Converging Evidence from fMRI and Chronometric TMS

High-level perception results from interactions between hierarchical brain systems responsive to gradually increasing feature complexities. During reading, the initial evaluation of simple visual features in the early visual cortex (EVC) is followed by orthographic and lexical computations in the ventral occipitotemporal cortex (vOTC). While similar visual regions are engaged in tactile Braille reading in congenitally blind people, it is unclear whether the visual network maintains or reorganizes its hierarchy for reading in this population. Combining fMRI and chronometric transcranial magnetic stimulation (TMS), our study revealed a clear correspondence between sighted and blind individuals (both male and female) on how their occipital cortices functionally supports reading and speech processing. Using fMRI, we first observed that vOTC, but not EVC, showed an enhanced response to lexical vs nonlexical information in both groups and sensory modalities. Using TMS, we further found that, in both groups, the processing of written words and pseudowords was disrupted by the EVC stimulation at both early and late time windows. In contrast, the vOTC stimulation disrupted the processing of these written stimuli only when applied at late time windows, again in both groups. In the speech domain, we observed TMS effects only for meaningful words and only in the blind participants. Overall, our results suggest that, while the responses in the deprived visual areas might extend their functional response to other sensory modalities, the computational gradients between early and higher-order occipital regions are retained, at least for reading.

Probing the Neurodynamic Mechanisms of Cognitive Flexibility in Depressed Individuals with Autism Spectrum Disorder

Introduction: Autism spectrum disorder (ASD) is characterized by deficits in social behavior and executive function (EF), particularly in cognitive flexibility. Whether transcranial magnetic stimulation (TMS) can improve cognitive outcomes in patients with ASD remains an open question. We examined the acute effects of prefrontal TMS on cortical excitability and fluid cognition in individuals with ASD who underwent TMS for refractory major depression. Methods: We analyzed data from an open-label pilot study involving nine participants with ASD and treatment-resistant depression who received 30 sessions of accelerated theta burst stimulation of the dorsolateral prefrontal cortex, either unilaterally or bilaterally. Electroencephalography data were collected at baseline and 1, 4, and 12-weeks posttreatment and analyzed using a mixed-effects linear model to assess changes in regional cortical excitability using three models of spectral parametrization. Fluid cognition was measured using the National Institutes of Health Toolbox Cognitive Battery. Results: Prefrontal TMS led to a decrease in prefrontal cortical excitability and an increase in right temporoparietal excitability, as measured using spectral exponent analysis. This was associated with a significant improvement in the NIH Toolbox Fluid Cognition Composite score and the Dimensional Change Card Sort subtest from baseline to 12 weeks posttreatment (t = 3.79, p = 0.005, n = 9). Improvement in depressive symptomatology was significant (HDRS-17, F (3, 21) = 28.49, p < 0.001) and there was a significant correlation between cognitive improvement at week 4 and improvement in depression at week 12 (r = 0.71, p = 0.05). Conclusion: These findings link reduced prefrontal excitability in patients with ASD and improvements in cognitive flexibility. The degree to which these mechanisms can be generalized to ASD populations without Major Depressive Disorder remains a compelling question for future research.

Diverse Frontoparietal Connectivity Supports Semantic Prediction and Integration in Sentence Comprehension

Predictive processing in the parietal, temporal, frontal, and sensory cortex allows us to anticipate future meanings to maximize the efficiency of language comprehension, with the temporoparietal junction (TPJ) and inferior frontal gyrus (IFG) thought to be situated toward the top of a predictive hierarchy. Although the regions underpinning this fundamental brain function are well-documented, it remains unclear how they interact to achieve efficient comprehension. To this end, we recorded functional magnetic resonance imaging (fMRI) in 22 participants (11 males) while they comprehended sentences presented part by part, in which we manipulated the constraint provided by sentential contexts on upcoming semantic information. Using this paradigm, we examined the connectivity patterns of bilateral TPJ and IFG during anticipatory phases (i.e., before the onset of targets) and integration phases (i.e., after the onset of targets). When upcoming semantic content was highly predictable in strong constraint contexts, both the left TPJ and bilateral IFG showed stronger visual coupling, while the right TPJ showed stronger connectivity with regions within control, default mode, and visual networks, including the IFG, parahippocampal gyrus, posterior cingulate, and fusiform gyrus. These connectivity patterns were weaker when predicted semantic content appeared, in line with predictive coding theory. Conversely, for less-predictable content, these connectivity patterns were stronger during the integration phase. Overall, these results suggest that both top-down semantic prediction and bottom-up integration during predictive processing are supported by flexible coupling of frontoparietal regions with control, memory, and sensory systems.

Abnormal structural‒functional coupling patterning in progressive supranuclear palsy is associated with diverse gradients and histological features

The anatomy of the brain supports inherent processes, fostering mental abilities and eventually facilitating adaptive behavior. Recent studies have shown that progressive supranuclear palsy (PSP) is accompanied by alterations in functional and structural networks. However, how the structure and function of PSP coordinates change is not clear, and the relationships between structural‒functional coupling (SFC) and the gradient of hierarchical structure and cellular histology remain largely unknown. Here, we use neuroimaging data from two independent cohorts and a public histological dataset to investigate the relationships among the cellular histology, hierarchical structure, and SFC of PSP patients. We find that the SFC of the entire cortex in PSP is severely disrupted, with higher coupling in the visual network (VN). Moreover, coupling differences in PSP follow a macroscopic organizational principle from unimodal to transmodal gradients. Finally, we elucidate greater laminar differentiation in VN regions sensitive to SFC changes in PSP, which is related mainly to the higher cellular density and smaller size of the internal-granular layer. In conclusion, our findings provide an interpretable framework for understanding SFC changes in PSP and provide new insights into the consistency of structural and functional changes in PSP regarding hierarchical structure and cellular histology.

How the intrinsic functional connectivity patterns of the semantic network support semantic processing

Semantic processing, a core of language comprehension, involves the activation of brain regions dispersed extensively across the frontal, temporal, and parietal cortices that compose the semantic network. To comprehend the functional structure of this semantic network and how it prepares for semantic processing, we investigated its intrinsic functional connectivity (FC) and the relation between this pattern and semantic processing ability in a large sample from the Human Connectome Project (HCP) dataset. We first defined a well-studied brain network for semantic processing, and then we characterized the within-network connectivity (WNC) and the between-network connectivity (BNC) within this network using a voxel-based global brain connectivity (GBC) method based on resting-state functional magnetic resonance imaging (fMRI). The results showed that 97.73% of the voxels in the semantic network displayed considerably greater WNC than BNC, demonstrating that the semantic network is a fairly encapsulated network. Moreover, multiple connector hubs in the semantic network were identified after applying the criterion of WNC > 1 SD above the mean WNC of the semantic network. More importantly, three of these connector hubs (i.e., the left anterior temporal lobe, angular gyrus, and orbital part of the inferior frontal gyrus) were reliably associated with semantic processing ability. Our findings suggest that the three identified regions use WNC as the central mechanism for supporting semantic processing and that task-independent spontaneous connectivity in the semantic network is essential for semantic processing.

Whole-brain structural connectome asymmetry in autism

Autism spectrum disorder is a common neurodevelopmental condition that manifests as a disruption in sensory and social skills. Although it has been shown that the brain morphology of individuals with autism is asymmetric, how this differentially affects the structural connectome organization of each hemisphere remains under-investigated. We studied whole-brain structural connectivity-based brain asymmetry in individuals with autism using diffusion magnetic resonance imaging obtained from the Autism Brain Imaging Data Exchange initiative. By leveraging dimensionality reduction techniques, we constructed low-dimensional representations of structural connectivity and calculated their asymmetry index. Comparing the asymmetry index between individuals with autism and neurotypical controls, we found atypical structural connectome asymmetry in the sensory and default-mode regions, particularly showing weaker asymmetry towards the right hemisphere in autism. Network communication provided topological underpinnings by demonstrating that the inferior temporal cortex and limbic and frontoparietal regions showed reduced global network communication efficiency and decreased send-receive network navigation in the inferior temporal and lateral visual cortices in individuals with autism. Finally, supervised machine learning revealed that structural connectome asymmetry could be used as a measure for predicting communication-related autistic symptoms and nonverbal intelligence. Our findings provide insights into macroscale structural connectome alterations in autism and their topological underpinnings.

The Italian Sensorimotor Norms: Perception and action strength measures for 959 words

Neuroscience research has provided evidence that semantic information is stored in a distributed brain network involved in sensorimotor and linguistic processing. More specifically, according to the embodied cognition accounts, the representation of concepts is deemed as grounded in our bodily states. For these reasons, normative measures of words should provide relevant information about the extent to which each word embeds perceptual and action properties. In the present study, we collected ratings for 959 Italian nouns and verbs from 398 volunteers, recruited via an online platform. The words were mostly taken from the Italian adaptation of the Affective Norms for English Words (ANEW). A pool of 145 verbs was added to the original set. All the words were rated on 11 sensorimotor dimensions: six perceptual modalities (vision, audition, taste, smell, touch, and interoception) and five effectors (hand-arm, foot-leg, torso, mouth, head). The new verbs were also rated on the ANEW dimensions. Results showed good reliability and consistency with previous studies. Relations between perceptual and motor dimensions are described and interpreted, along with relations between the sensorimotor and the affective dimensions. The currently developed dataset represents an important novelty, as it includes different word classes, i.e., both nouns and verbs, and integrates ratings of both sensorimotor and affective dimensions, along with other psycholinguistic parameters; all features only partially accomplished in previous studies.

Vigilant attention mediates the association between resting EEG alpha oscillations and word learning ability

Individuals exhibit considerable variability in their capacity to learn and retain new information, including novel vocabulary. Prior research has established the importance of vigilance and electroencephalogram (EEG) alpha rhythm in the learning process. However, the interplay between vigilant attention, EEG alpha oscillations, and an individual's word learning ability (WLA) remains elusive. To address this knowledge gap, here we conducted two experiments with a total of 140 young and middle-aged adults who underwent resting EEG recordings prior to completing a paired-associate word learning task and a psychomotor vigilance test (PVT). The results of both experiments consistently revealed significant positive correlations between WLA and resting EEG alpha oscillations in the occipital and frontal regions. Furthermore, the association between resting EEG alpha oscillations and WLA was mediated by vigilant attention, as measured by the PVT. These findings provide compelling evidence supporting the crucial role of vigilant attention in linking EEG alpha oscillations to an individual's learning ability.

Task-based activation and resting-state connectivity predict individual differences in semantic capacity for complex semantic knowledge

Our ability to know and access complex factual information has far reaching effects, influencing our scholastic, professional and social lives. Here we employ functional MRI to assess the relationship between individual differences in semantic aptitude in the task-based activation and resting-state functional connectivity. Using psychometric and behavioural measures, we quantified the semantic and executive aptitude of individuals and had them perform a general-knowledge semantic-retrieval task (N = 41) and recorded resting-state data (N = 43). During the semantic-retrieval task, participants accessed general-knowledge facts drawn from four different knowledge-domains (people, places, objects and 'scholastic'). Individuals with greater executive capacity more strongly recruit anterior sections of prefrontal cortex (PFC) and the precuneus, and individuals with lower semantic capacity more strongly activate a posterior section of the dorsomedial PFC (dmPFC). The role of these regions in semantic processing was validated by analysis of independent resting-state data, where increased connectivity between a left anterior PFC and the precuneus predict higher semantic aptitude, and increased connectivity between left anterior PFC and posterior dmPFC predict lower semantic aptitude. Results suggest that coordination between core semantic regions in the precuneus and anterior prefrontal regions associated with executive processes support greater semantic aptitude.

Stimulus Repetition and Sample Size Considerations in Item-Level Representational Similarity Analysis

In studies using representational similarity analysis (RSA) of fMRI data, the reliability of the neural representational dissimilarity matrix (RDM) is a limiting factor in the ability to detect neural correlates of a model. A common strategy for boosting neural RDM reliability is to employ repeated presentations of the stimulus set across imaging runs or sessions. However, little is known about how the benefits of stimulus repetition are affected by repetition suppression, or how they compare with the benefits of increasing the number of participants. We examined the effects of these design parameters in two large data sets where participants performed a semantic decision task on visually presented words. We found that reliability gains from stimulus repetition were strongly affected by repetition suppression, both within and across scanning sessions separated by multiple weeks. The results provide new insights into these experimental design choices, particularly for item-level RSA studies of semantic cognition.

Taxonomic and thematic relations rely on different types of semantic features: Evidence from an fMRI meta-analysis and a semantic priming study

Taxonomic and thematic relations are major components of semantic representation but their neurocognitive underpinnings are still debated. We hypothesised that taxonomic relations preferentially activate parts of anterior temporal lobe (ATL) because they rely more on colour and shape features, while thematic relations preferentially activate temporoparietal cortex (TPC) because they rely more on action and location knowledge. We first conducted activation likelihood estimation (ALE) meta-analysis to assess evidence for neural specialisation in the existing fMRI literature (Study 1), then used a primed semantic judgement task to examine if the two relations are primed by different feature types (Study 2). We find that taxonomic relations show minimal feature-based specialisation but preferentially activate the lingual gyrus. Thematic relations are more dependent on action and location features and preferentially engage TPC. The meta-analysis also showed that lateral ATL is preferentially engaged by Thematic relations, which may reflect their greater reliance on verbal associations.

Typicality in the brain during semantic and episodic memory decisions

Concept typicality is a key semantic dimension supporting the categorical organization of items based on their features, such that typical items share more features with other members of their category than atypical items, which are more distinctive. Typicality effects manifest in better accuracy and faster response times during categorization tasks, but higher performance for atypical items in episodic memory tasks, due to their distinctiveness. At a neural level, typicality has been linked to the anterior temporal lobe (ATL) and the inferior frontal gyrus (IFG) in semantic decision tasks, but patterns of brain activity during episodic memory tasks remain to be understood. We investigated the neural correlates of typicality in semantic and episodic memory to determine the brain regions associated with semantic typicality and uncover effects arising when items are reinstated during retrieval. In an fMRI study, 26 healthy young subjects first performed a category verification task on words representing typical and atypical concepts (encoding), and then completed a recognition memory task (retrieval). In line with previous literature, we observed higher accuracy and faster response times for typical items in the category verification task, while atypical items were better recognized in the episodic memory task. During category verification, univariate analyses revealed a greater involvement of the angular gyrus for typical items and the inferior frontal gyrus for atypical items. During the correct recognition of old items, regions belonging to the core recollection network were activated. We then compared the similarity of the representations from encoding to retrieval (ERS) using Representation Similarity Analyses. Results showed that typical items were reinstated more than atypical ones in several regions including the left precuneus and left anterior temporal lobe (ATL). This suggests that the correct retrieval of typical items requires finer-grained processing, evidenced by greater item-specific reinstatement, which is needed to resolve their confusability with other members of the category due to their higher feature similarity. Our findings confirm the centrality of the ATL in the processing of typicality while extending it to memory retrieval.

Resting-state occipito-frontal alpha connectome is linked to differential word learning ability in adult learners

Adult language learners show distinct abilities in acquiring a new language, yet the underlying neural mechanisms remain elusive. Previous studies suggested that resting-state brain connectome may contribute to individual differences in learning ability. Here, we recorded electroencephalography (EEG) in a large cohort of 106 healthy young adults (50 males) and examined the associations between resting-state alpha band (8-12 Hz) connectome and individual learning ability during novel word learning, a key component of new language acquisition. Behavioral data revealed robust individual differences in the performance of the novel word learning task, which correlated with their performance in the language aptitude test. EEG data showed that individual resting-state alpha band coherence between occipital and frontal regions positively correlated with differential word learning performance (p = 0.001). The significant positive correlations between resting-state occipito-frontal alpha connectome and differential world learning ability were replicated in an independent cohort of 35 healthy adults. These findings support the key role of occipito-frontal network in novel word learning and suggest that resting-state EEG connectome may be a reliable marker for individual ability during new language learning.

Adolescent development of multiscale structural wiring and functional interactions in the human connectome

Adolescence is a time of profound changes in the physical wiring and function of the brain. Here, we analyzed structural and functional brain network development in an accelerated longitudinal cohort spanning 14 to 25 y (n = 199). Core to our work was an advanced in vivo model of cortical wiring incorporating MRI features of corticocortical proximity, microstructural similarity, and white matter tractography. Longitudinal analyses assessing age-related changes in cortical wiring identified a continued differentiation of multiple corticocortical structural networks in youth. We then assessed structure-function coupling using resting-state functional MRI measures in the same participants both via cross-sectional analysis at baseline and by studying longitudinal change between baseline and follow-up scans. At baseline, regions with more similar structural wiring were more likely to be functionally coupled. Moreover, correlating longitudinal structural wiring changes with longitudinal functional connectivity reconfigurations, we found that increased structural differentiation, particularly between sensory/unimodal and default mode networks, was reflected by reduced functional interactions. These findings provide insights into adolescent development of human brain structure and function, illustrating how structural wiring interacts with the maturation of macroscale functional hierarchies.

Task modulates the orthographic and phonological representations in the bilateral ventral Occipitotemporal cortex

As a key area in word reading, the left ventral occipitotemporal cortex is proposed for abstract orthographic processing, and its middle part has even been labeled as the visual word form area. Because the definition of the VWFA largely varies and the reading task differs across studies, the function of the left ventral occipitotemporal cortex in word reading is continuingly debated on whether this region is specific for orthographic processing or be involved in an interactive framework. By using representational similarity analysis (RSA), this study examined information representation in the VWFA at the individual level and the modulatory effect of reading task. Twenty-four subjects were scanned while performing the explicit (i.e., the naming task) and implicit (i.e., the perceptual task) reading tasks. Activation analysis showed that the naming task elicited greater activation in regions related to phonological processing (e.g., the bilateral prefrontal cortex and temporoparietal cortex), while the perceptual task recruited greater activation in visual cortex and default mode network (e.g., the bilateral middle frontal gyrus, angular gyrus, and the right middle temporal gyrus). More importantly, RSA also showed that task modulated information representation in the bilateral anterior occipitotemporal cortex and VWFA. Specifically, ROI-based RSA revealed enhanced orthographic and phonological representations in the bilateral anterior fusiform cortex and VWFA in the naming task relative to the perceptual task. These results suggest that lexical representation in the VWFA is influenced by the demand of phonological processing, which supports the interactive account of the VWFA.

Unique, Shared, and Dominant Brain Activation in Visual Word Form Area and Lateral Occipital Complex during Reading and Picture Naming

Identifying printed words and pictures concurrently is ubiquitous in daily tasks, and so it is important to consider the extent to which reading words and naming pictures may share a cognitive-neurophysiological functional architecture. Two functional magnetic resonance imaging (fMRI) experiments examined whether reading along the left ventral occipitotemporal region (vOT; often referred to as a visual word form area, VWFA) has activation that is overlapping with referent pictures (i.e., both conditions significant and shared, or with one significantly more dominant) or unique (i.e., one condition significant, the other not), and whether picture naming along the right lateral occipital complex (LOC) has overlapping or unique activation relative to referent words. Experiment 1 used familiar regular and exception words (to force lexical reading) and their corresponding pictures in separate naming blocks, and showed dominant activation for pictures in the LOC, and shared activation in the VWFA for exception words and their corresponding pictures (regular words did not elicit significant VWFA activation). Experiment 2 controlled for visual complexity by superimposing the words and pictures and instructing participants to either name the word or the picture, and showed primarily shared activation in the VWFA and LOC regions for both word reading and picture naming, with some dominant activation for pictures in the LOC. Overall, these results highlight the importance of including exception words to force lexical reading when comparing to picture naming, and the significant shared activation in VWFA and LOC serves to challenge specialized models of reading or picture naming.

Language Processing Differences Between Blind and Sighted Individuals and the Abstract Versus Concrete Concept Difference

In the property listing task (PLT), participants are asked to list properties for a concept (e.g., for the concept dog, "barks," and "is a pet" may be produced). In conceptual property norming (CPNs) studies, participants are asked to list properties for large sets of concepts. Here, we use a mathematical model of the property listing process to explore two longstanding issues: characterizing the difference between concrete and abstract concepts, and characterizing semantic knowledge in the blind versus sighted population. When we apply our mathematical model to a large CPN reporting properties listed by sighted and blind participants, the model uncovers significant differences between concrete and abstract concepts. Though we also find that blind individuals show many of the same processing differences between abstract and concrete concepts found in sighted individuals, our model shows that those differences are noticeably less pronounced than in sighted individuals. We discuss our results vis-a-vis theories attempting to characterize abstract concepts.

Stimulus-independent neural coding of event semantics: Evidence from cross-sentence fMRI decoding

Multivariate neuroimaging studies indicate that the brain represents word and object concepts in a format that readily generalises across stimuli. Here we investigated whether this was true for neural representations of simple events described using sentences. Participants viewed sentences describing four events in different ways. Multivariate classifiers were trained to discriminate the four events using a subset of sentences, allowing us to test generalisation to novel sentences. We found that neural patterns in a left-lateralised network of frontal, temporal and parietal regions discriminated events in a way that generalised successfully over changes in the syntactic and lexical properties of the sentences used to describe them. In contrast, decoding in visual areas was sentence-specific and failed to generalise to novel sentences. In the reverse analysis, we tested for decoding of syntactic and lexical structure, independent of the event being described. Regions displaying this coding were limited and largely fell outside the canonical semantic network. Our results indicate that a distributed neural network represents the meaning of event sentences in a way that is robust to changes in their structure and form. They suggest that the semantic system disregards the surface properties of stimuli in order to represent their underlying conceptual significance.

Representations of conceptual information during automatic and active semantic access

When we read a word or see an object, conceptual meaning is automatically accessed. However, previous research investigating non-perceptual sensitivity to semantic class has employed active tasks. In this fMRI study, we tested whether conceptual representations in regions constituting the semantic network are invoked during passive semantic access and whether these representations are modulated by the need to access deeper knowledge. Seventeen healthy subjects performed a semantically active typicality judgment task and a semantically passive phonetic decision task, in both the written and the spoken input-modalities. Stimuli consisted of one hundred forty-four concepts drawn from six semantic categories. Multivariate Pattern Analysis (MVPA) revealed that the left posterior middle temporal gyrus (pMTG), posterior ventral temporal cortex (pVTC) and pars triangularis of the left inferior frontal gyrus (IFG) showed a stronger sensitivity to semantic category when active rather than passive semantic access is required. Using a cross-task training/testing classifier, we determined that conceptual representations were not only active in these regions during passive semantic access but that the neural representation of these categories was common to both active and passive access. Collectively, these results show that while representations in the pMTG, pVTC and IFG are strongly modulated by active conceptual access, consistent representational patterns are present during active and passive conceptual access in these same regions.

Neural dynamics of semantic categorization in semantic variant of primary progressive aphasia

Semantic representations are processed along a posterior-to-anterior gradient reflecting a shift from perceptual (e.g., it has eight legs) to conceptual (e.g., venomous spiders are rare) information. One critical region is the anterior temporal lobe (ATL): patients with semantic variant primary progressive aphasia (svPPA), a clinical syndrome associated with ATL neurodegeneration, manifest a deep loss of semantic knowledge. We test the hypothesis that svPPA patients perform semantic tasks by over-recruiting areas implicated in perceptual processing. We compared MEG recordings of svPPA patients and healthy controls during a categorization task. While behavioral performance did not differ, svPPA patients showed indications of greater activation over bilateral occipital cortices and superior temporal gyrus, and inconsistent engagement of frontal regions. These findings suggest a pervasive reorganization of brain networks in response to ATL neurodegeneration: the loss of this critical hub leads to a dysregulated (semantic) control system, and defective semantic representations are seemingly compensated via enhanced perceptual processing.

Grid-like and distance codes for representing word meaning in the human brain

Relational information about items in memory is thought to be represented in our brain thanks to an internal comprehensive model, also referred to as a "cognitive map". In the human neuroimaging literature, two signatures of bi-dimensional cognitive maps have been reported: the grid-like code and the distance-dependent code. While these kinds of representation were previously observed during spatial navigation and, more recently, during processing of perceptual stimuli, it is still an open question whether they also underlie the representation of the most basic items of language: words. Here we taught human participants the meaning of novel words as arbitrary labels for a set of audiovisual objects varying orthogonally in size and sound. The novel words were therefore conceivable as points in a navigable 2D map of meaning. While subjects performed a word comparison task, we recorded their brain activity using functional magnetic resonance imaging (fMRI). By applying a combination of representational similarity and fMRI-adaptation analyses, we found evidence of (i) a grid-like code, in the right postero-medial entorhinal cortex, representing the relative angular positions of words in the word space, and (ii) a distance-dependent code, in medial prefrontal, orbitofrontal, and mid-cingulate cortices, representing the Euclidean distance between words. Additionally, we found evidence that the brain also separately represents the single dimensions of word meaning: their implied size, encoded in visual areas, and their implied sound, in Heschl's gyrus/Insula. These results support the idea that the meaning of words, when they are organized along two dimensions, is represented in the human brain across multiple maps of different dimensionality. SIGNIFICANT STATEMENT: How do we represent the meaning of words and perform comparative judgements on them in our brain? According to influential theories, concepts are conceivable as points of an internal map (where distance represents similarity) that, as the physical space, can be mentally navigated. Here we use fMRI to show that when humans compare newly learnt words, they recruit a grid-like and a distance code, the same types of neural codes that, in mammals, represent relations between locations in the environment and support physical navigation between them.

The contributions of the left fusiform subregions to successful encoding of novel words

The left fusiform cortex has been identified as a crucial structure in visual word learning and memory. Nevertheless, the specific roles of the fusiform subregions in word memory and their consistency across different writings have not been elaborated. To address these questions, the present study performed two experiments, in which study-test paradigm was used. Participants' brain activity was measured with fMRI while memorizing novel logographic words in Experiment 1 and novel alphabetic words in Experiment 2. A post-scan recognition memory test was then administered to acquire the memory performance. Results showed that, neural responses in the left anterior and middle fusiform subregions during encoding were positively correlated with recognition memory of novel words. Moreover, the positive brain-behavior correlations in the left anterior and middle fusiform cortex were evident for both logographic and alphabetic writings. The present findings clarify the relationship between the left fusiform subregions and novel word memory.

Knowing what you need to know in advance: The neural processes underpinning flexible semantic retrieval of thematic and taxonomic relations

Semantic retrieval is flexible, allowing us to focus on subsets of features and associations that are relevant to the current task or context: for example, we use taxonomic relations to locate items in the supermarket (carrots are a vegetable), but thematic associations to decide which tools we need when cooking (carrot goes with peeler). We used fMRI to investigate the neural basis of this form of semantic flexibility; in particular, we asked how retrieval unfolds differently when participants have advanced knowledge of the type of link to retrieve between concepts (taxonomic or thematic). Participants performed a semantic relatedness judgement task: on half the trials, they were cued to search for a taxonomic or thematic link, while on the remaining trials, they judged relatedness without knowing which type of semantic relationship would be relevant. Left inferior frontal gyrus showed greater activation when participants knew the trial type in advance. An overlapping region showed a stronger response when the semantic relationship between the items was weaker, suggesting this structure supports both top-down and bottom-up forms of semantic control. Multivariate pattern analysis further revealed that the neural response in left inferior frontal gyrus reflects goal information related to different conceptual relationships. Top-down control specifically modulated the response in visual cortex: when the goal was unknown, there was greater deactivation to the first word, and greater activation to the second word. We conclude that top-down control of semantic retrieval is primarily achieved through the gating of task-relevant 'spoke' regions.

Distinct Representational Structure and Localization for Visual Encoding and Recall during Visual Imagery

During memory recall and visual imagery, reinstatement is thought to occur as an echoing of the neural patterns during encoding. However, the precise information in these recall traces is relatively unknown, with previous work primarily investigating either broad distinctions or specific images, rarely bridging these levels of information. Using ultra-high-field (7T) functional magnetic resonance imaging with an item-based visual recall task, we conducted an in-depth comparison of encoding and recall along a spectrum of granularity, from coarse (scenes, objects) to mid (e.g., natural, manmade scenes) to fine (e.g., living room, cupcake) levels. In the scanner, participants viewed a trial-unique item, and after a distractor task, visually imagined the initial item. During encoding, we observed decodable information at all levels of granularity in category-selective visual cortex. In contrast, information during recall was primarily at the coarse level with fine-level information in some areas; there was no evidence of mid-level information. A closer look revealed segregation between voxels showing the strongest effects during encoding and those during recall, and peaks of encoding-recall similarity extended anterior to category-selective cortex. Collectively, these results suggest visual recall is not merely a reactivation of encoding patterns, displaying a different representational structure and localization from encoding, despite some overlap.

A multi-scale cortical wiring space links cellular architecture and functional dynamics in the human brain

The vast net of fibres within and underneath the cortex is optimised to support the convergence of different levels of brain organisation. Here, we propose a novel coordinate system of the human cortex based on an advanced model of its connectivity. Our approach is inspired by seminal, but so far largely neglected models of cortico-cortical wiring established by postmortem anatomical studies and capitalises on cutting-edge in vivo neuroimaging and machine learning. The new model expands the currently prevailing diffusion magnetic resonance imaging (MRI) tractography approach by incorporation of additional features of cortical microstructure and cortico-cortical proximity. Studying several datasets and different parcellation schemes, we could show that our coordinate system robustly recapitulates established sensory-limbic and anterior-posterior dimensions of brain organisation. A series of validation experiments showed that the new wiring space reflects cortical microcircuit features (including pyramidal neuron depth and glial expression) and allowed for competitive simulations of functional connectivity and dynamics based on resting-state functional magnetic resonance imaging (rs-fMRI) and human intracranial electroencephalography (EEG) coherence. Our results advance our understanding of how cell-specific neurobiological gradients produce a hierarchical cortical wiring scheme that is concordant with increasing functional sophistication of human brain organisation. Our evaluations demonstrate the cortical wiring space bridges across scales of neural organisation and can be easily translated to single individuals.

Neural pattern similarity across concept exemplars predicts memory after a long delay

The irregularities of the world ensure that each interaction we have with a concept is unique. In order to generalize across these unique encounters to form a high-level representation of a concept, we must draw on similarities between exemplars to form new conceptual knowledge that is maintained over a long time. Two neural similarity measures - pattern robustness and encoding-retrieval similarity - are particularly important for predicting memory outcomes. In this study, we used fMRI to measure activity patterns while people encoded and retrieved novel pairings between unfamiliar (Dutch) words and visually presented animal species. We address two underexplored questions: 1) whether neural similarity measures can predict memory outcomes, despite perceptual variability between presentations of a concept and 2) if pattern similarity measures can predict subsequent memory over a long delay (i.e., one month). Our findings indicate that pattern robustness during encoding in brain regions that include parietal and medial temporal areas is an important predictor of subsequent memory. In addition, we found significant encoding-retrieval similarity in the left ventrolateral prefrontal cortex after a month's delay. These findings demonstrate that pattern similarity is an important predictor of memory for novel word-animal pairings even when the concept includes multiple exemplars. Importantly, we show that established predictive relationships between pattern similarity and subsequent memory do not require visually identical stimuli (i.e., are not simply due to low-level visual overlap between stimulus presentations) and are maintained over a month.

What and where in the auditory systems of sighted and early blind individuals: Evidence from representational similarity analysis

Separated ventral and dorsal streams in auditory system have been proposed to process sound identification and localization respectively. Despite the popularity of the dual-pathway model, it remains controversial how much independence two neural pathways enjoy and whether visual experiences can influence the distinct cortical organizational scheme. In this study, representational similarity analysis (RSA) was used to explore the functional roles of distinct cortical regions that lay within either the ventral or dorsal auditory streams of sighted and early blind (EB) participants. We found functionally segregated auditory networks in both sighted and EB groups where anterior superior temporal gyrus (aSTG) and inferior frontal junction (IFJ) were more related to the sound identification, while posterior superior temporal gyrus (pSTG) and inferior parietal lobe (IPL) preferred the sound localization. The findings indicated visual experiences may not have an influence on this functional dissociation and the cortex of the human brain may be organized as task-specific and modality-independent strategies. Meanwhile, partial overlap of spatial and non-spatial auditory information processing was observed, illustrating the existence of interaction between the two auditory streams. Furthermore, we investigated the effect of visual experiences on the neural bases of auditory perception and observed the cortical reorganization in EB participants in whom middle occipital gyrus was recruited to process auditory information. Our findings examined the distinct cortical networks that abstractly encoded sound identification and localization, and confirmed the existence of interaction from the multivariate perspective. Furthermore, the results suggested visual experience might not impact the functional specialization of auditory regions.

Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies

Information that is shared across brains is encoded in idiosyncratic fine-scale functional topographies. Hyperalignment captures shared information by projecting pattern vectors for neural responses and connectivities into a common, high-dimensional information space, rather than by aligning topographies in a canonical anatomical space. Individual transformation matrices project information from individual anatomical spaces into the common model information space, preserving the geometry of pairwise dissimilarities between pattern vectors, and model cortical topography as mixtures of overlapping, individual-specific topographic basis functions, rather than as contiguous functional areas. The fundamental property of brain function that is preserved across brains is information content, rather than the functional properties of local features that support that content. In this Perspective, we present the conceptual framework that motivates hyperalignment, its computational underpinnings for joint modeling of a common information space and idiosyncratic cortical topographies, and discuss implications for understanding the structure of cortical functional architecture.

fMRI representational similarity analysis reveals graded preferences for chromatic and achromatic stimulus contrast across human visual cortex

Human visual cortex is partitioned into different functional areas that, from lower to higher, become increasingly selective and responsive to complex feature dimensions. Here we use a Representational Similarity Analysis (RSA) of fMRI-BOLD signals to make quantitative comparisons across LGN and multiple visual areas of the low-level stimulus information encoded in the patterns of voxel responses. Our stimulus set was picked to target the four functionally distinct subcortical channels that input visual cortex from the LGN: two achromatic sinewave stimuli that favor the responses of the high-temporal magnocellular and high-spatial parvocellular pathways, respectively, and two chromatic stimuli isolating the L/M-cone opponent and S-cone opponent pathways, respectively. Each stimulus type had three spatial extents to sample both foveal and para-central visual field. With the RSA, we compare quantitatively the response specializations for individual stimuli and combinations of stimuli in each area and how these change across visual cortex. First, our results replicate the known response preferences for motion/flicker in the dorsal visual areas. In addition, we identify two distinct gradients along the ventral visual stream. In the early visual areas (V1-V3), the strongest differential representation is for the achromatic high spatial frequency stimuli, suitable for form vision, and a very weak differentiation of chromatic versus achromatic contrast. Emerging in ventral occipital areas (V4, VO1 and VO2), however, is an increasingly strong separation of the responses to chromatic versus achromatic contrast and a decline in the high spatial frequency representation. These gradients provide new insight into how visual information is transformed across the visual cortex.

Encoding of Auditory Temporal Gestalt in the Human Brain

The perception of an acoustic rhythm is invariant to the absolute temporal intervals constituting a sound sequence. It is unknown where in the brain temporal Gestalt, the percept emerging from the relative temporal proximity between acoustic events, is encoded. Two different relative temporal patterns, each induced by three experimental conditions with different absolute temporal patterns as sensory basis, were presented to participants. A linear support vector machine classifier was trained to differentiate activation patterns in functional magnetic resonance imaging data to the two different percepts. Across the sensory constituents the classifier decoded which percept was perceived. A searchlight analysis localized activation patterns specific to the temporal Gestalt bilaterally to the temporoparietal junction, including the planum temporale and supramarginal gyrus, and unilaterally to the right inferior frontal gyrus (pars opercularis). We show that auditory areas not only process absolute temporal intervals, but also integrate them into percepts of Gestalt and that encoding of these percepts persists in high-level associative areas. The findings complement existing knowledge regarding the processing of absolute temporal patterns to the processing of relative temporal patterns relevant to the sequential binding of perceptual elements into Gestalt.

Categorical representation from sound and sight in the ventral occipito-temporal cortex of sighted and blind

Is vision necessary for the development of the categorical organization of the Ventral Occipito-Temporal Cortex (VOTC)? We used fMRI to characterize VOTC responses to eight categories presented acoustically in sighted and early blind individuals, and visually in a separate sighted group. We observed that VOTC reliably encodes sound categories in sighted and blind people using a representational structure and connectivity partially similar to the one found in vision. Sound categories were, however, more reliably encoded in the blind than the sighted group, using a representational format closer to the one found in vision. Crucially, VOTC in blind represents the categorical membership of sounds rather than their acoustic features. Our results suggest that sounds trigger categorical responses in the VOTC of congenitally blind and sighted people that partially match the topography and functional profile of the visual response, despite qualitative nuances in the categorical organization of VOTC between modalities and groups.

The world is full of rich and dynamic visual information. To avoid information overload, the human brain groups inputs into categories such as faces, houses, or tools. A part of the brain called the ventral occipito-temporal cortex (VOTC) helps categorize visual information. Specific parts of the VOTC prefer different types of visual input; for example, one part may tend to respond more to faces, whilst another may prefer houses. However, it is not clear how the VOTC characterizes information.

One idea is that similarities between certain types of visual information may drive how information is organized in the VOTC. For example, looking at faces requires using central vision, while looking at houses requires using peripheral vision. Furthermore, all faces have a roundish shape while houses tend to have a more rectangular shape. Another possibility, however, is that the categorization of different inputs cannot be explained just by vision, and is also be driven by higher-level aspects of each category. For instance, how humans use or interact with something may also influence how an input is categorized. If categories are established depending (at least partially) on these higher-level aspects, rather than purely through visual likeness, it is likely that the VOTC would respond similarly to both sounds and images representing these categories.

Now, Mattioni et al. have tested how individuals with and without sight respond to eight different categories of information to find out whether or not categorization is driven purely by visual likeness. Each category was presented to participants using sounds while measuring their brain activity. In addition, a group of participants who could see were also presented with the categories visually. Mattioni et al. then compared what happened in the VOTC of the three groups – sighted people presented with sounds, blind people presented with sounds, and sighted people presented with images – in response to each category.

The experiment revealed that the VOTC organizes both auditory and visual information in a similar way. However, there were more similarities between the way blind people categorized auditory information and how sighted people categorized visual information than between how sighted people categorized each type of input. Mattioni et al. also found that the region of the VOTC that responds to inanimate objects massively overlapped across the three groups, whereas the part of the VOTC that responds to living things was more variable.

These findings suggest that the way that the VOTC organizes information is, at least partly, independent from vision. The experiments also provide some information about how the brain reorganizes in people who are born blind. Further studies may reveal how differences in the VOTC of people with and without sight affect regions typically associated with auditory categorization, and potentially explain how the brain reorganizes in people who become blind later in life.

Brain Regions Involved in Conceptual Retrieval in Sighted and Blind People

If conceptual retrieval is partially based on the simulation of sensorimotor experience, people with a different sensorimotor experience, such as congenitally blind people, should retrieve concepts in a different way. However, studies investigating the neural basis of several conceptual domains (e.g., actions, objects, places) have shown a very limited impact of early visual deprivation. We approached this problem by investigating brain regions that encode the perceptual similarity of action and color concepts evoked by spoken words in sighted and congenitally blind people. At first, and in line with previous findings, a contrast between action and color concepts (independently of their perceptual similarity) revealed similar activations in sighted and blind people for action concepts and partially different activations for color concepts, but outside visual areas. On the other hand, adaptation analyses based on subjective ratings of perceptual similarity showed compelling differences across groups. Perceptually similar colors and actions induced adaptation in the posterior occipital cortex of sighted people only, overlapping with regions known to represent low-level visual features of those perceptual domains. Early-blind people instead showed a stronger adaptation for perceptually similar concepts in temporal regions, arguably indexing higher reliance on a lexical-semantic code to represent perceptual knowledge. Overall, our results show that visual deprivation does changes the neural bases of conceptual retrieval, but mostly at specific levels of representation supporting perceptual similarity discrimination, reconciling apparently contrasting findings in the field.

Six Challenges for Embodiment Research

Twenty years after Barsalou’s seminal perceptual-symbols article, embodied cognition, the notion that cognition involves simulations of sensory, motor, or affective states, has moved from an outlandish proposal to a mainstream position adopted by many researchers in the psychological and cognitive sciences (and neurosciences). Though it has generated productive work in the cognitive sciences as a whole, it has had a particularly strong impact on research into language comprehension. The view of a mental lexicon based on symbolic word representations, which are arbitrarily linked to sensory aspects of their referents, was generally accepted since the cognitive revolution in the 1950s. This has radically changed. Given the current status of embodiment as a main theory of cognition, it is somewhat surprising that a close look at the literature reveals that the debate about the nature of the processes involved in language comprehension is far from settled, and key questions remain unanswered. We present several suggestions for a productive way forward.

The Neural Representations of Movement across Semantic Categories

Previous evidence from neuropsychological and neuroimaging studies suggests functional specialization for tools and related semantic knowledge in a left frontoparietal network. It is still debated whether these areas are involved in the representation of rudimentary movement-relevant knowledge regardless of semantic domains (animate vs. inanimate) or categories (tools vs. nontool objects). Here, we used fMRI to record brain activity while 13 volunteers performed two semantic judgment tasks on visually presented items from three different categories: animals, tools, and nontool objects. Participants had to judge two distinct semantic features: whether two items typically move in a similar way (e.g., a fan and a windmill move in circular motion) or whether they are usually found in the same environment (e.g., a seesaw and a swing are found in a playground). We investigated differences in overall activation (which areas are involved) as well as representational content (which information is encoded) across semantic features and categories. Results of voxel-wise mass univariate analysis showed that, regardless of semantic category, a dissociation emerges between processing information on prototypical location (involving the anterior temporal cortex and the angular gyrus) and movement (linked to left inferior parietal and frontal activation). Multivoxel pattern correlation analyses confirmed the representational segregation of networks encoding task- and category-related aspects of semantic processing. Taken together, these findings suggest that the left frontoparietal network is recruited to process movement properties of items (including both biological and nonbiological motion) regardless of their semantic category.

The Cognitive Neuroscience of Stable and Flexible Semantic Typicality

Typicality effects are among the most well-studied phenomena in the study of concepts. The classical notion of typicality is that typical concepts share many features with category co-members and few features with members of contrast categories. However, this notion was challenged by evidence that typicality is highly context dependent and not always dependent on central tendency. Dieciuc and Folstein (2019) argued that there is strong evidence for both views and that the two types of typicality effects might depend on different mechanisms. A recent theoretical framework, the controlled semantic cognition framework (Lamdon Ralph et al., 2017) strongly emphasizes the classical view, but includes mechanisms that could potentially account for both kinds of typicality. In contrast, the situated cognition framework (Barsalou, 2009b) articulates the context-dependent view. Here, we review evidence from cognitive neuroscience supporting the two frameworks. We also briefly evaluate the ability of computational models associated with the CSC to account for phenomena supporting SitCog (Rogers and McClelland, 2004). Many predictions of both frameworks are borne out by recent cognitive neuroscience evidence. While the CSC framework can at least potentially account for many of the typicality phenomena reviewed, challenges remain, especially with regard to ad hoc categories.

Redefining the resolution of semantic knowledge in the brain: Advances made by the introduction of models of semantics in neuroimaging

The boundaries of our understanding of conceptual representation in the brain have been redrawn since the introduction of explicit models of semantics. These models are grounded in vast behavioural datasets acquired in healthy volunteers. Here, we review the most important techniques which have been applied to detect semantic information in neuroimaging data and argue why semantic models are possibly the most valuable addition to the research of semantics in recent years. Using multivariate analysis, predictions based on patient lesion data have been confirmed during semantic processing in healthy controls. Secondly, this new method has given rise to new research avenues, e.g. the detection of semantic processing outside of the temporal cortex. As a future line of work, the same research strategy could be useful to study neurological conditions such as the semantic variant of primary progressive aphasia, which is characterized by pathological semantic processing.

Neural activity in human visual cortex is transformed by learning real world size

The way that our brain processes visual information is directly affected by our experience. Repeated exposure to a visual stimulus triggers experience-dependent plasticity in the visual cortex of many species. Humans also have the unique ability to acquire visual knowledge through instruction. We introduced human participants to the real-world size of previously unfamiliar species, and to the functional motion of novel tools, during a functional magnetic resonance imaging scan. Using machine learning, we compared activity patterns evoked by images of the new items, before and after participants learned the animals' real-world size or tools' motion. We found that, after acquiring size information, participants' visual activity patterns for the new animals became more confusable with activity patterns evoked by similar-sized known animals in early visual cortex, but not in ventral temporal cortex, reflecting an influence of new size knowledge on posterior, but not anterior, components of the ventral stream. In contrast, learning the functional motion of new tools did not lead to an equivalent change in recorded activity. Finally, the time-points marked by evidence of new size information in early visual cortex were more likely to show size information and greater activation in the right angular gyrus, a key hub of semantic knowledge and spatial cognition. Overall, these findings suggest that learning an item's real-world size by instruction influences subsequent activity in visual cortex and in a region that is central to semantic and spatial brain systems.

Conceptual and Perceptual Dimensions of Word Meaning Are Recovered Rapidly and in Parallel during Reading

A single word (the noun “ elephant”) encapsulates a complex multidimensional meaning, including both perceptual (“ big”, “ gray”, “ trumpeting”) and conceptual (“ mammal”, “ can be found in India”) features. Opposing theories make different predictions as to whether different features (also conceivable as dimensions of the semantic space) are stored in similar neural regions and recovered with similar temporal dynamics during word reading. In this magnetoencephalography study, we tracked the brain activity of healthy human participants while reading single words varying orthogonally across three semantic dimensions: two perceptual ones (i.e., the average implied real-world size and the average strength of association with a prototypical sound) and a conceptual one (i.e., the semantic category). The results indicate that perceptual and conceptual representations are supported by partially segregated neural networks: Whereas visual and auditory dimensions are encoded in the phase coherence of low-frequency oscillations of occipital and superior temporal regions, respectively, semantic features are encoded in the power of low-frequency oscillations of anterior temporal and inferior parietal areas. However, despite the differences, these representations appear to emerge at the same latency: around 200 msec after stimulus onset. Taken together, these findings suggest that perceptual and conceptual dimensions of the semantic space are recovered automatically, rapidly, and in parallel during word reading.

Are visual processes causally involved in "perceptual simulation" effects in the sentence-picture verification task?

Many studies have shown that sentences implying an object to have a certain shape produce a robust reaction time advantage for shape-matching pictures in the sentence-picture verification task. Typically, this finding has been interpreted as evidence for perceptual simulation, i.e., that access to implicit shape information involves the activation of modality-specific visual processes. It follows from this proposal that disrupting visual processing during sentence comprehension should interfere with perceptual simulation and obliterate the match effect. Here we directly test this hypothesis. Participants listened to sentences while seeing either visual noise that was previously shown to strongly interfere with basic visual processing or a blank screen. Experiments 1 and 2 replicated the match effect but crucially visual noise did not modulate it. When an interference technique was used that targeted high-level semantic processing (Experiment 3) however the match effect vanished. Visual noise specifically targeting high-level visual processes (Experiment 4) only had a minimal effect on the match effect. We conclude that the shape match effect in the sentence-picture verification paradigm is unlikely to rely on perceptual simulation.

Lexical learning in a new language leads to neural pattern similarity with word reading in native language

Previous neuroimaging studies have suggested similar neural activations for word reading in native and second languages. However, such similarities were qualitatively determined (i.e., overlapping activation based on traditional univariate activation analysis). In this study, using representational similarity analysis and an artificial language training paradigm, we quantitatively computed cross-language neural pattern similarity to examine the modulatory effect of proficiency in the new language. Twenty-four native Chinese speakers were trained to learn 30 words in a logographic artificial language for 12 days and scanned while performing a semantic decision task after 4-day training and after 12-day training. Results showed that higher proficiency in the new language was associated with higher cross-language pattern similarity in select regions of the reading network.

Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream

A significant body of research in cognitive neuroscience is aimed at understanding how object concepts are represented in the human brain. However, it remains unknown whether and where the visual and abstract conceptual features that define an object concept are integrated. We addressed this issue by comparing the neural pattern similarities among object-evoked fMRI responses with behavior-based models that independently captured the visual and conceptual similarities among these stimuli. Our results revealed evidence for distinctive coding of visual features in lateral occipital cortex, and conceptual features in the temporal pole and parahippocampal cortex. By contrast, we found evidence for integrative coding of visual and conceptual object features in perirhinal cortex. The neuroanatomical specificity of this effect was highlighted by results from a searchlight analysis. Taken together, our findings suggest that perirhinal cortex uniquely supports the representation of fully specified object concepts through the integration of their visual and conceptual features.

What semantic dementia teaches us about the functional organization of the left posterior fusiform gyrus

After demonstrating the relative preservation of fruit and vegetable knowledge in patients with semantic dementia (SD), we sought to identify the neural substrate of this unusual category effect. Nineteen patients with SD performed a semantic sorting task and underwent a morphometric 3T MRI scan. The grey-matter volumes of five regions within the temporal lobe were bilaterally computed, as well as those of two recently described areas (FG1 and FG2) within the posterior fusiform gyrus. In contrast to the other semantic categories we tested, fruit and vegetable scores were only predicted by left FG1 volume. We therefore found a specific relationship between the volume of a subregion within the left posterior fusiform gyrus and performance on fruits and vegetables in SD. We argue that the left FG1 is a convergence zone for the features that might be critical to successfully sort fruits and vegetables. We also discuss evidence for a functional specialization of the fusiform gyrus along two axes (lateral medial and longitudinal), depending on the nature of the concepts and on the level of processing complexity required by the ongoing task.

Feature Selection Methods for Zero-Shot Learning of Neural Activity

Dimensionality poses a serious challenge when making predictions from human neuroimaging data. Across imaging modalities, large pools of potential neural features (e.g., responses from particular voxels, electrodes, and temporal windows) have to be related to typically limited sets of stimuli and samples. In recent years, zero-shot prediction models have been introduced for mapping between neural signals and semantic attributes, which allows for classification of stimulus classes not explicitly included in the training set. While choices about feature selection can have a substantial impact when closed-set accuracy, open-set robustness, and runtime are competing design objectives, no systematic study of feature selection for these models has been reported. Instead, a relatively straightforward feature stability approach has been adopted and successfully applied across models and imaging modalities. To characterize the tradeoffs in feature selection for zero-shot learning, we compared correlation-based stability to several other feature selection techniques on comparable data sets from two distinct imaging modalities: functional Magnetic Resonance Imaging and Electrocorticography. While most of the feature selection methods resulted in similar zero-shot prediction accuracies and spatial/spectral patterns of selected features, there was one exception; A novel feature/attribute correlation approach was able to achieve those accuracies with far fewer features, suggesting the potential for simpler prediction models that yield high zero-shot classification accuracy.

The neuro-cognitive representations of symbols: the case of concrete words

We live our lives surrounded by symbols (e.g., road signs, logos, but especially words and numbers), and throughout our life we use them to evoke, communicate and reflect upon ideas and things that are not currently present to our senses. Symbols are represented in our brains at different levels of complexity: at the first and most simple level, as physical entities, in the corresponding primary and secondary sensory cortices. The crucial property of symbols, however, is that, despite the simplicity of their surface forms, they have the power of evoking higher order multifaceted representations that are implemented in distributed neural networks spanning a large portion of the cortex. The rich internal states that reflect our knowledge of the meaning of symbols are what we call semantic representations. In this review paper, we summarize our current knowledge of both the cognitive and neural substrates of semantic representations, focusing on concrete words (i.e., nouns or verbs referring to concrete objects and actions), which, together with numbers, are the most-studied and well defined classes of symbols. Following a systematic descriptive approach, we will organize this literature review around two key questions: what is the content of semantic representations? And, how are semantic representations implemented in the brain, in terms of localization and dynamics? While highlighting the main current opposing perspectives on these topics, we propose that a fruitful way to make substantial progress in this domain would be to adopt a geometrical view of semantic representations as points in high dimensional space, and to operationally partition the space of concrete word meaning into motor-perceptual and conceptual dimensions. By giving concrete examples of the kinds of research that can be done within this perspective, we illustrate how we believe this framework will foster theoretical speculations as well as empirical research.

Evidence for similar patterns of neural activity elicited by picture- and word-based representations of natural scenes

A long-standing core question in cognitive science is whether different modalities and representation types (pictures, words, sounds, etc.) access a common store of semantic information. Although different input types have been shown to activate a shared network of brain regions, this does not necessitate that there is a common representation, as the neurons in these regions could still differentially process the different modalities. However, multi-voxel pattern analysis can be used to assess whether, e.g., pictures and words evoke a similar pattern of activity, such that the patterns that separate categories in one modality transfer to the other. Prior work using this method has found support for a common code, but has two limitations: they have either only examined disparate categories (e.g. animals vs. tools) that are known to activate different brain regions, raising the possibility that the pattern separation and inferred similarity reflects only large scale differences between the categories or they have been limited to individual object representations. By using natural scene categories, we not only extend the current literature on cross-modal representations beyond objects, but also, because natural scene categories activate a common set of brain regions, we identify a more fine-grained (i.e. higher spatial resolution) common representation. Specifically, we studied picture- and word-based representations of natural scene stimuli from four different categories: beaches, cities, highways, and mountains. Participants passively viewed blocks of either phrases (e.g. "sandy beach") describing scenes or photographs from those same scene categories. To determine whether the phrases and pictures evoke a common code, we asked whether a classifier trained on one stimulus type (e.g. phrase stimuli) would transfer (i.e. cross-decode) to the other stimulus type (e.g. picture stimuli). The analysis revealed cross-decoding in the occipitotemporal, posterior parietal and frontal cortices. This similarity of neural activity patterns across the two input types, for categories that co-activate local brain regions, provides strong evidence of a common semantic code for pictures and words in the brain.

Semantic attributes are encoded in human electrocorticographic signals during visual object recognition

Non-invasive neuroimaging studies have shown that semantic category and attribute information are encoded in neural population activity. Electrocorticography (ECoG) offers several advantages over non-invasive approaches, but the degree to which semantic attribute information is encoded in ECoG responses is not known. We recorded ECoG while patients named objects from 12 semantic categories and then trained high-dimensional encoding models to map semantic attributes to spectral-temporal features of the task-related neural responses. Using these semantic attribute encoding models, untrained objects were decoded with accuracies comparable to whole-brain functional Magnetic Resonance Imaging (fMRI), and we observed that high-gamma activity (70-110Hz) at basal occipitotemporal electrodes was associated with specific semantic dimensions (manmade-animate, canonically large-small, and places-tools). Individual patient results were in close agreement with reports from other imaging modalities on the time course and functional organization of semantic processing along the ventral visual pathway during object recognition. The semantic attribute encoding model approach is critical for decoding objects absent from a training set, as well as for studying complex semantic encodings without artificially restricting stimuli to a small number of semantic categories.