Decoding semantic representations in mind and brain

EEG-based neurophysiological indicators in pronoun resolution using feature analysis

BACKGROUND

Pronoun resolution is a crucial aspect of language comprehension, yet its underlying neural mechanisms remain poorly understood. While previous studies have explored individual linguistic factors, a systematic analysis of Electroencephalography (EEG)-based neurophysiological indicators across different resolution cues (gender, verb bias, and discourse focus) remains unexplored, limiting our understanding of neural-cognitive processes.

NEW METHOD

We developed an approach combining ReliefF feature selection and Linear Discriminant Analysis (LDA) to analyze EEG data from twenty participants during pronoun resolution tasks. The method examined neural indicators focusing on power spectral density (PSD) and time-domain features, including Zero-Crossing Rate and Peak-to-Peak amplitude.

RESULTS

We identified crucial neural indicators across 14 channels and 4 frequency bands, highlighting PSD features in specific channels (AF3, AF4, FC6, F4, T7, T8, and O2) across theta, beta, and gamma bands. Gender-cue processing exhibited enhanced neural responses in prefrontal and temporal regions with shorter reaction times (748.77 ms) compared to verb bias (903.20 ms) and discourse focus (948.92 ms).

COMPARISON WITH EXISTING METHODS

Unlike previous studies examining individual linguistic factors, our approach simultaneously analyzed multiple resolution cues. The method achieved significant above-chance classification accuracy (49.08 % vs. 33.33 %) across three linguistic factors. This multi-factor analysis provides a more nuanced understanding of pronoun resolution processes than traditional single-factor studies.

CONCLUSIONS

Our findings suggest a more efficient, feature-driven processing mechanism for gender-cue resolution, contrasting with more complex, reasoning-dependent processing of verb semantics and discourse cues. These insights have implications for developing computational models of language processing and potential clinical applications for language disorders.

The true colors of reading: Literacy enhances lexical-semantic processing in rapid automatized and discrete object naming

Semantic knowledge is a defining property of human cognition, profoundly influenced by cultural experiences. In this study, we investigated whether literacy enhances lexical-semantic processing independently of schooling. Three groups of neurotypical adults - unschooled illiterates, unschooled ex-illiterates, and schooled literates - from the same residential and socioeconomic background in Portugal were tested on serial rapid automatized naming (RAN) and on discrete naming of everyday objects (concrete concepts) and basic color patches (abstract concepts). The performance of readers, whether schooled literate or unschooled ex-illiterate, was not affected by stimulus category, whereas illiterates were much slower on color than object naming, irrespective of task. This naming advantage promoted by literacy was not significantly mediated by vocabulary size. We conclude that literacy per se, regardless of schooling, contributes to faster naming of depicted concepts, particularly those of more abstract categories. Our findings provide further evidence that literacy influences cognition beyond the mere accumulation of knowledge: Literacy enhances the quality and efficiency of lexical-semantic representations and processing.

Suggestions for extending the FAIR Principles based on a linguistic perspective on semantic interoperability

FAIR (meta)data presuppose their successful communication between machines and humans while preserving meaning and reference. The FAIR Guiding Principles lack specificity regarding semantic interoperability. We adopt a linguistic perspective on semantic interoperability and investigate the structures and conventions ensuring reliable communication of textual information, drawing parallels with data structures by understanding both as models. We propose a conceptual model of semantic interoperability, comprising intensional and extensional terminological interoperability, as well as logical and schema propositional interoperability. Since there cannot be a universally accepted best vocabulary and best (meta)data schema, establishing semantic interoperability necessitates the provision of comprehensive sets of intensional and extensional entity mappings and schema crosswalks. In accordance with our conceptual model, we suggest additions to the FAIR Guiding Principles that encompass the requirements for semantic interoperability. Additionally, we argue that attaining FAIRness of (meta)data requires not only their organization into FAIR Digital Objects, but also the establishment of a FAIR ecosystem of FAIR Services, that include a terminology, a schema, and an operations service.

A vectorial code for semantics in human hippocampus

As we listen to speech, our brains actively compute the meanings of individual words. Inspired by the success of large language models (LLMs), we hypothesized that the brain employs vectorial coding principles, such that meaning is reflected in distributed activity of single neurons. We recorded responses of hundreds of neurons in the human hippocampus, which has a well-established role in semantic coding, while participants listened to narrative speech. We find encoding of contextual word meaning in the simultaneous activity of neurons whose individual selectivities span multiple unrelated semantic categories. Like embedding vectors in semantic models, distance between neural population responses correlates with semantic distance; however, this effect was only observed in contextual embedding models (like BERT) and was reversed in non-contextual embedding models (like Word2Vec), suggesting that the semantic distance effect depends critically on contextualization. Moreover, for the subset of highly semantically similar words, even contextual embedders showed an inverse correlation between semantic and neural distances; we attribute this pattern to the noise-mitigating benefits of contrastive coding. Finally, in further support for the critical role of context, we find that neural response variance increases with lexical polysemy. Ultimately, these results support the hypothesis that semantic coding in the hippocampus follows vectorial principles.

A simple clustering approach to map the human brain's cortical semantic network organization during task

Constructing task-state large-scale brain networks can enhance our understanding of the organization of brain functions during cognitive tasks. The primary goal of brain network partitioning is to cluster functionally homogeneous brain regions. However, a brain region often serves multiple cognitive functions, complicating the partitioning process. This study proposes a novel clustering method for partitioning large-scale brain networks based on specific cognitive functions, selecting semantic representation as the target cognitive function to evaluate the validity of the proposed method. Specifically, we analyzed functional magnetic resonance imaging (fMRI) data from 11 subjects, each exposed to 672 concepts, and correlated this with semantic rating data related to these concepts. We identified distinct semantic networks based on the concept comprehension task and validated the robustness of our network partitioning through multiple methods. We found that the semantic networks derived from multidimensional semantic activation clustering exhibit high reliability and cross-semantic model consistency (semantic ratings and word embeddings extracted from GPT-2), particularly in networks associated with high semantic functions. Moreover, these semantic networks exhibits significant differences from the resting-state and task-based brain networks obtained using traditional methods. Further analysis revealed functional differences between semantic networks, including disparities in their multidimensional semantic representation capabilities, differences in the information modalities they rely on to acquire semantic information, and varying associations with general cognitive domains. This study introduces a novel approach for analyzing brain networks tailored to specific cognitive functions, establishing a standard semantic parcellation with seven networks for future research, potentially enriching our understanding of complex cognitive processes and their neural bases.

Tracking neural correlates of contextualized meanings with representational similarity analysis

Although it is uncontroversial that word meanings shift depending on their context, our understanding of contextualized lexical meaning remains poor. How is a contextualized semantic space organized? In this MEG study (27 human participants, 16 women, 10 men, 1 non-binary), we manipulated the semantic and syntactic contexts of word forms to query the organization of this space. All wordforms were noun/verb ambiguous and varied in the semantic distance between their noun and verb uses: unambiguous stems, polysemes with distinct but related meanings, and homonyms with completely unrelated meanings. The senses of each stem were disambiguated by a unique discourse sentence and the items were placed in syntactic contexts of varying sizes. Univariate results characterized syntactic context as a bilateral and distributed effect. A multivariate Representational Similarity Analysis correlated one-hot models of the categorical factors as well as contextualized embedding-based models with MEG activity. Of all models representing ambiguity, only a model differentiating between syntactic categories across contexts correlated with the brain. An All-Embeddings model, where each contextualized word had a distinct representation, explained distributed neural activity across the left hemisphere. Finally, a Syntactic Context model and Within-Context-Stem model were significant in left occipito-parietal regions. While the noun vs. verb contrast affected neural signals robustly, we saw no evidence of the homonym-polyseme-unambiguous contrast, over and above the evidence for fully itemized representations. These findings suggest that in contexts devoid of ambiguity, the neural representation of a word is mainly shaped by its syntactic category and its contextually informed, unique semantic representation.Significance statement A word's context can define its meaning. Context is an integral part of understanding language, yet the organization of the semantic space formed by words in context remains unclear. We used magnetoencephalography (MEG) to investigate the dynamic interaction between contextualized semantic representations, syntactic categories, ambiguity and local syntactic contexts. We find a left-lateralized network encoding a semantic space where each contextualized instance of a word has a distinct neural representation, while syntactic category had a broad bilateral representation. Our study provides a link between naturalistic multivariate studies of item/word-level semantic processing and more traditional controlled factorial investigations of lexical meaning. These findings enrich our understanding of the neural underpinnings of words in context and highlights the role of syntactic context.

Neural Representation Precision of Distance Predicts Children's Arithmetic Performance

Focusing on the distance between magnitudes as the starting point to investigate the mechanism of relation detection and its contribution to mathematical thinking, this study explores the precision of neural representations of numerical distance and their impact on children's arithmetic performance. By employing neural decoding techniques and representational similarity analysis, the present study investigates how accurately the brain represents numerical distances and how this precision relates to arithmetic skills. Twenty-nine school-aged children participated, completing a dot number comparison task during fMRI scanning and an arithmetic fluency test. Results indicated that neural activation patterns in the intra-parietal sulcus decoded the distance between the presented pair of dots, and higher precision in neural distance representation correlates with better arithmetic performance. These findings suggest that the accuracy of neural decoding can serve as an index of neural representation precision and that the ability to precisely encode numerical distances in the brain is a key factor in mathematical abilities. This provides new insights into the neural basis of mathematical cognition and learning.

Decoding semantics from natural speech using human intracranial EEG

Brain-computer interfaces (BCIs) hold promise for restoring natural language production capabilities in patients with speech impairments, potentially enabling smooth conversation that conveys meaningful information via synthesized words. While considerable progress has been made in decoding phonetic features of speech, our ability to extract lexical semantic information (i.e. the meaning of individual words) from neural activity remains largely unexplored. Moreover, most existing BCI research has relied on controlled experimental paradigms rather than natural conversation, limiting our understanding of semantic decoding in ecological contexts. Here, we investigated the feasibility of decoding lexical semantic information from stereo-electroencephalography (sEEG) recordings in 14 participants during spontaneous conversation. Using multivariate pattern analysis, we were able to decode word level semantic features during language production with an average accuracy of 21% across all participants compared to a chance level of 10%. This semantic decoding remained robust across different semantic representations while maintaining specificity to semantic features. Further, we identified a distributed left-lateralized network spanning precentral gyrus, pars triangularis, and middle temporal cortex, with low-frequency oscillations showing stronger contributions. Together, our results establish the feasibility of extracting word meanings from neural activity during natural speech production and demonstrate the potential for decoding semantic content from unconstrained speech.

Low-frequency neural activity tracks syntactic information through semantic mediation

How our brain integrates single words into larger linguistic units is a central focus in neurolinguistic studies. Previous studies mainly explored this topic at the semantic or syntactic level, with few looking at how cortical activities track word sequences with different levels of semantic correlations. In addition, prior research did not tease apart the semantic factors from the syntactic ones in the word sequences. The current study addressed these issues by conducting a speech perception EEG experiment using the frequency-tagging paradigm. Participants (N = 25, Meanage = 23;4, 16 girls) were asked to listen to different types of sequences and their neural activity was recorded by EEG. We also constructed a model simulation based on surprisal values of GPT-2. Both the EEG results and the model prediction show that low-frequency neural activity tracks syntactic information through semantic mediation. Implications of the findings were discussed in relation to the language processing mechanism.

Language proficiency is associated with neural representational dimensionality of semantic concepts

Previous studies suggest that semantic concepts are characterized by high-dimensional neural representations and that language proficiency affects semantic processing. However, it is not clear whether language proficiency modulates the dimensional representations of semantic concepts at the neural level. To address this question, the present study adopted principal component analysis (PCA) and representational similarity analysis (RSA) to examine the differences in representational dimensionalities (RDs) and in semantic representations between words in highly proficient (Chinese) and less proficient (English) language. PCA results revealed that language proficiency increased the dimensions of lexical representations in the left inferior frontal gyrus, temporal pole, inferior temporal gyrus, supramarginal gyrus, angular gyrus, and fusiform gyrus. RSA results further showed that these regions represented semantic information and that higher semantic representations were observed in highly proficient language relative to less proficient language. These results suggest that language proficiency is associated with the neural representational dimensionality of semantic concepts.

Grasping the Concept of an Object at a Glance: Category Information Accessed by Brief Dichoptic Presentation

What type of conceptual information about an object do we get at a brief glance? In two experiments, we investigated the nature of conceptual tokening-the moment at which conceptual information about an object is accessed. Using a masked picture-word congruency task with dichoptic presentations at "brief" (50-60 ms) and "long" (190-200 ms) durations, participants judged the relation between a picture (e.g., a banana) and a word representing one of four property types about the object: superordinate (fruit), basic level (banana), a high-salient (yellow), or low-salient feature (peel). In Experiment 1, stimuli were presented in black-and-white; in Experiment 2, they were presented in red and blue, with participants wearing red-blue anaglyph glasses. This manipulation allowed for the independent projection of stimuli to the left- and right-hemisphere visual areas, aiming to probe the early effects of these projections in conceptual tokening. Results showed that superordinate and basic-level properties elicited faster and more accurate responses than high- and low-salient features at both presentation times. This advantage persisted even when the objects were divided into categories (e.g., animals, vegetables, vehicles, tools), and when objects contained high-salient visual features. However, contrasts between categories show that animals, fruits, and vegetables tend to be categorized at the superordinate level, while vehicles tend to be categorized at the basic level. Also, for a restricted class of objects, high-salient features representing diagnostic color information (yellow for the picture of a banana) facilitated congruency judgments to the same extent as that of superordinate and basic-level labels. We suggest that early access to object concepts yields superordinate and basic-level information, with features only yielding effects at a later stage of processing, unless they represent diagnostic color information. We discuss these results advancing a unified theory of conceptual representation, integrating key postulates of atomism and feature-based theories.

Spatiotemporal Properties of Common Semantic Categories for Words and Pictures

The timing of semantic processing during object recognition in the brain is a topic of ongoing discussion. One way of addressing this question is by applying multivariate pattern analysis to human electrophysiological responses to object images of different semantic categories. However, although multivariate pattern analysis can reveal whether neuronal activity patterns are distinct for different stimulus categories, concerns remain on whether low-level visual features also contribute to the classification results. To circumvent this issue, we applied a cross-decoding approach to magnetoencephalography data from stimuli from two different modalities: images and their corresponding written words. We employed items from three categories and presented them in a randomized order. We show that if the classifier is trained on words, pictures are classified between 150 and 430 msec after stimulus onset, and when training on pictures, words are classified between 225 and 430 msec. The topographical map, identified using a searchlight approach for cross-modal activation in both directions, showed left lateralization, confirming the involvement of linguistic representations. These results point to semantic activation of pictorial stimuli occurring at ∼150 msec, whereas for words, the semantic activation occurs at ∼230 msec.

ChineseEEG: A Chinese Linguistic Corpora EEG Dataset for Semantic Alignment and Neural Decoding

An Electroencephalography (EEG) dataset utilizing rich text stimuli can advance the understanding of how the brain encodes semantic information and contribute to semantic decoding in brain-computer interface (BCI). Addressing the scarcity of EEG datasets featuring Chinese linguistic stimuli, we present the ChineseEEG dataset, a high-density EEG dataset complemented by simultaneous eye-tracking recordings. This dataset was compiled while 10 participants silently read approximately 13 hours of Chinese text from two well-known novels. This dataset provides long-duration EEG recordings, along with pre-processed EEG sensor-level data and semantic embeddings of reading materials extracted by a pre-trained natural language processing (NLP) model. As a pilot EEG dataset derived from natural Chinese linguistic stimuli, ChineseEEG can significantly support research across neuroscience, NLP, and linguistics. It establishes a benchmark dataset for Chinese semantic decoding, aids in the development of BCIs, and facilitates the exploration of alignment between large language models and human cognitive processes. It can also aid research into the brain's mechanisms of language processing within the context of the Chinese natural language.

Neural computations in prosopagnosia

We report an investigation of the neural processes involved in the processing of faces and objects of brain-lesioned patient PS, a well-documented case of pure acquired prosopagnosia. We gathered a substantial dataset of high-density electrophysiological recordings from both PS and neurotypicals. Using representational similarity analysis, we produced time-resolved brain representations in a format that facilitates direct comparisons across time points, different individuals, and computational models. To understand how the lesions in PS's ventral stream affect the temporal evolution of her brain representations, we computed the temporal generalization of her brain representations. We uncovered that PS's early brain representations exhibit an unusual similarity to later representations, implying an excessive generalization of early visual patterns. To reveal the underlying computational deficits, we correlated PS' brain representations with those of deep neural networks (DNN). We found that the computations underlying PS' brain activity bore a closer resemblance to early layers of a visual DNN than those of controls. However, the brain representations in neurotypicals became more akin to those of the later layers of the model compared to PS. We confirmed PS's deficits in high-level brain representations by demonstrating that her brain representations exhibited less similarity with those of a DNN of semantics.

Decoding Semantics from Dynamic Brain Activation Patterns: From Trials to Task in EEG/MEG Source Space

The temporal dynamics within the semantic brain network and its dependence on stimulus and task parameters are still not well understood. Here, we addressed this by decoding task as well as stimulus information from source-estimated EEG/MEG human data. We presented the same visual word stimuli in a lexical decision (LD) and three semantic decision (SD) tasks. The meanings of the presented words varied across five semantic categories. Source space decoding was applied over time in five ROIs in the left hemisphere (anterior and posterior temporal lobe, inferior frontal gyrus, primary visual areas, and angular gyrus) and one in the right hemisphere (anterior temporal lobe). Task decoding produced sustained significant effects in all ROIs from 50 to 100 ms, both when categorizing tasks with different semantic demands (LD-SD) as well as for similar semantic tasks (SD-SD). In contrast, a semantic word category could only be decoded in lATL, rATL, PTC, and IFG, between 250 and 500 ms. Furthermore, we compared two approaches to source space decoding: conventional ROI-by-ROI decoding and combined-ROI decoding with back-projected activation patterns. The former produced more reliable results for word category decoding while the latter was more informative for task decoding. This indicates that task effects are distributed across the whole semantic network while stimulus effects are more focal. Our results demonstrate that the semantic network is widely distributed but that bilateral anterior temporal lobes together with control regions are particularly relevant for the processing of semantic information.

Decoding face recognition abilities in the human brain

Why are some individuals better at recognizing faces? Uncovering the neural mechanisms supporting face recognition ability has proven elusive. To tackle this challenge, we used a multimodal data-driven approach combining neuroimaging, computational modeling, and behavioral tests. We recorded the high-density electroencephalographic brain activity of individuals with extraordinary face recognition abilities-super-recognizers-and typical recognizers in response to diverse visual stimuli. Using multivariate pattern analyses, we decoded face recognition abilities from 1 s of brain activity with up to 80% accuracy. To better understand the mechanisms subtending this decoding, we compared representations in the brains of our participants with those in artificial neural network models of vision and semantics, as well as with those involved in human judgments of shape and meaning similarity. Compared to typical recognizers, we found stronger associations between early brain representations of super-recognizers and midlevel representations of vision models as well as shape similarity judgments. Moreover, we found stronger associations between late brain representations of super-recognizers and representations of the artificial semantic model as well as meaning similarity judgments. Overall, these results indicate that important individual variations in brain processing, including neural computations extending beyond purely visual processes, support differences in face recognition abilities. They provide the first empirical evidence for an association between semantic computations and face recognition abilities. We believe that such multimodal data-driven approaches will likely play a critical role in further revealing the complex nature of idiosyncratic face recognition in the human brain.

Parallel cognitive maps for multiple knowledge structures in the hippocampal formation

The hippocampal-entorhinal system uses cognitive maps to represent spatial knowledge and other types of relational information. However, objects can often be characterized by different types of relations simultaneously. How does the hippocampal formation handle the embedding of stimuli in multiple relational structures that differ vastly in their mode and timescale of acquisition? Does the hippocampal formation integrate different stimulus dimensions into one conjunctive map or is each dimension represented in a parallel map? Here, we reanalyzed human functional magnetic resonance imaging data from Garvert et al. (2017) that had previously revealed a map in the hippocampal formation coding for a newly learnt transition structure. Using functional magnetic resonance imaging adaptation analysis, we found that the degree of representational similarity in the bilateral hippocampus also decreased as a function of the semantic distance between presented objects. Importantly, while both map-like structures localized to the hippocampal formation, the semantic map was located in more posterior regions of the hippocampal formation than the transition structure and thus anatomically distinct. This finding supports the idea that the hippocampal-entorhinal system forms parallel cognitive maps that reflect the embedding of objects in diverse relational structures.

Multi-Level Linguistic Alignment in a Dynamic Collaborative Problem-Solving Task

Cocreating meaning in collaboration is challenging. Success is often determined by people's abilities to coordinate their language to converge upon shared mental representations. Here we explore one set of low-level linguistic behaviors, linguistic alignment, that both emerges from, and facilitates, outcomes of high-level convergence. Linguistic alignment captures the ways people reuse, that is, "align to," the lexical, syntactic, and semantic forms of others' utterances. Our focus is on the temporal change of multi-level linguistic alignment, as well as how alignment is related to communicative outcomes within a unique collaborative problem-solving paradigm. The primary task, situated within a virtual educational video game, requires creative thinking between three people where the paths for possible solutions are highly variable. We find that over time interactions are marked by decreasing lexical and syntactic alignment, with a trade-off of increasing semantic alignment. However, greater semantic alignment does not translate into better team performance. Overall, these findings provide greater clarity on the role of linguistic coordination within complex and dynamic collaborative problem-solving tasks.