Decoding abstract and concrete concept representations based on single-trial fMRI data

Abstract sentence meanings are grounded in the sensory-motor regions in a context-dependent fashion

Sentences conveying abstract meanings are crucial tools for high-level thinking and communication. Previous research has sparked a debate on whether abstract concepts rely on the representation of the sensory-motor brain areas. We explored this issue with the assumption that abstract meanings at the sentence level could invoke the sensory-motor regions a context-dependent fashion. With a sentence comprehension task and functional MRI, we measured the neural response patterns of sentences with multimodal abstract meaning, which were presented following context sentences describing either concrete sound-related or action-related events. Multivariate pattern analyses revealed that neural responses to sentences could discriminate abstract sentences in sound- versus action-related contexts, and also context sentences describing these two types of events. The discrimination was manifested in the regions responsible for high-level auditory perception and action execution. Our finding indicates that abstract meanings in modality-specific contexts mayrequire a certain degree of grounded processing in the sensory-motor regions.

Knowledge concept recognition in the learning brain via fMRI classification

Knowledge concept recognition (KCR) aims to identify the concepts learned in the brain, which has been a longstanding area of interest for learning science and education. While many studies have investigated object recognition using brain fMRIs, there are limited research on identifying specific knowledge points within the classroom. In this paper, we propose to recognize the knowledge concepts in computer science by classifying the brain fMRIs taken when students are learning the concepts. More specifically, this study made attempts on two representation strategies, i.e., voxel and time difference. Based on the representations, we evaluated traditional classifiers and the combination of CNN and LSTM for KCR. Experiments are conducted on a public dataset collected from 25 students and teachers in a computer science course. The evaluations of classifying fMRI segments show that the used classifiers all can attain a good performance when using the time-difference representation, where the CNN-LSTM model reaches the highest accuracy. This research contributes to the understanding of human learning and supports the development of personalized learning.

Delineating Region-Specific contributions and connectivity patterns for semantic association and categorization through ROI and Granger causality analysis

The neural mechanisms supporting semantic association and categorization are examined in this study. Semantic association involves linking concepts through shared themes, events, or scenes, while semantic categorization organizes meanings hierarchically based on defining features. Twenty-three adults participated in an fMRI study performing categorization and association judgment tasks. Results showed stronger activation in the inferior frontal gyrus during association and marginally weaker activation in the posterior middle temporal gyrus (pMTG) during categorization. Granger causality analysis revealed bottom-up connectivity from the visual cortex to the hippocampus during semantic association, whereas semantic categorization exhibited strong reciprocal connections between the pMTG and frontal semantic control regions, together with information flow from the visual association area and hippocampus to the pars triangularis. We propose that demands on semantic retrieval, precision of semantic representation, perceptual experiences and world knowledge result in observable differences between these two semantic relations.

Information load dynamically modulates functional brain connectivity during narrative listening

Narratives are paradigmatic examples of natural language, where nouns represent a proxy of information. Functional magnetic resonance imaging (fMRI) studies revealed the recruitment of temporal cortices during noun processing and the existence of a noun-specific network at rest. Yet, it is unclear whether, in narratives, changes in noun density influence the brain functional connectivity, so that the coupling between regions correlates with information load. We acquired fMRI activity in healthy individuals listening to a narrative with noun density changing over time and measured whole-network and node-specific degree and betweenness centrality. Network measures were correlated with information magnitude with a time-varying approach. Noun density correlated positively with the across-regions average number of connections and negatively with the average betweenness centrality, suggesting the pruning of peripheral connections as information decreased. Locally, the degree of the bilateral anterior superior temporal sulcus (aSTS) was positively associated with nouns. Importantly, aSTS connectivity cannot be explained by changes in other parts of speech (e.g., verbs) or syllable density. Our results indicate that the brain recalibrates its global connectivity as a function of the information conveyed by nouns in natural language. Also, using naturalistic stimulation and network metrics, we corroborate the role of aSTS in noun processing.

Neural decoding of music from the EEG

Neural decoding models can be used to decode neural representations of visual, acoustic, or semantic information. Recent studies have demonstrated neural decoders that are able to decode accoustic information from a variety of neural signal types including electrocortiography (ECoG) and the electroencephalogram (EEG). In this study we explore how functional magnetic resonance imaging (fMRI) can be combined with EEG to develop an accoustic decoder. Specifically, we first used a joint EEG-fMRI paradigm to record brain activity while participants listened to music. We then used fMRI-informed EEG source localisation and a bi-directional long-term short term deep learning network to first extract neural information from the EEG related to music listening and then to decode and reconstruct the individual pieces of music an individual was listening to. We further validated our decoding model by evaluating its performance on a separate dataset of EEG-only recordings. We were able to reconstruct music, via our fMRI-informed EEG source analysis approach, with a mean rank accuracy of 71.8% ([Formula: see text], [Formula: see text]). Using only EEG data, without participant specific fMRI-informed source analysis, we were able to identify the music a participant was listening to with a mean rank accuracy of 59.2% ([Formula: see text], [Formula: see text]). This demonstrates that our decoding model may use fMRI-informed source analysis to aid EEG based decoding and reconstruction of acoustic information from brain activity and makes a step towards building EEG-based neural decoders for other complex information domains such as other acoustic, visual, or semantic information.

Neural decoding of semantic concepts: A systematic literature review

Objective Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Results Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

Detecting spontaneous deception in the brain

Deception detection can be of great value during the juristic investigation. Although the neural signatures of deception have been widely documented, most prior studies were biased by difficulty levels. That is, deceptive behavior typically required more effort, making deception detection possibly effort detection. Furthermore, no study has examined the generalizability across instructed and spontaneous responses and across participants. To explore these issues, we used a dual‐task paradigm, where the difficulty level was balanced between truth‐telling and lying, and the instructed and spontaneous truth‐telling and lying were collected independently. Using Multivoxel pattern analysis, we were able to decode truth‐telling versus lying with a balanced difficulty level. Results showed that the angular gyrus (AG), inferior frontal gyrus (IFG), and postcentral gyrus could differentiate lying from truth‐telling. Critically, linear classifiers trained to distinguish instructed truthful and deceptive responses could correctly differentiate spontaneous truthful and deceptive responses in AG and IFG with above‐chance accuracy. In addition, with a leave‐one‐participant‐out analysis, multivoxel neural patterns from AG could classify if the left‐out participant was lying or not in a trial. These results indicate the commonality of neural responses subserved instructed and spontaneous deceptive behavior as well as the feasibility of cross‐participant deception validation.

Spatiotemporal target selection for intracranial neural decoding of abstract and concrete semantics

Decoding the inner representation of a word meaning from human cortical activity is a substantial challenge in the development of speech brain-machine interfaces (BMIs). The semantic aspect of speech is a novel target of speech decoding that may enable versatile communication platforms for individuals with impaired speech ability; however, there is a paucity of electrocorticography studies in this field. We decoded the semantic representation of a word from single-trial cortical activity during an imageability-based property identification task that required participants to discriminate between the abstract and concrete words. Using high gamma activity in the language-dominant hemisphere, a support vector machine classifier could discriminate the 2-word categories with significantly high accuracy (73.1 ± 7.5%). Activities in specific time components from two brain regions were identified as significant predictors of abstract and concrete dichotomy. Classification using these feature components revealed that comparable prediction accuracy could be obtained based on a spatiotemporally targeted decoding approach. Our study demonstrated that mental representations of abstract and concrete word processing could be decoded from cortical high gamma activities, and the coverage of implanted electrodes and time window of analysis could be successfully minimized. Our findings lay the foundation for the future development of semantic-based speech BMIs.

A CNN-transformer hybrid approach for decoding visual neural activity into text

BACKGROUND AND OBJECTIVE

Most studies used neural activities evoked by linguistic stimuli such as phrases or sentences to decode the language structure. However, compared to linguistic stimuli, it is more common for the human brain to perceive the outside world through non-linguistic stimuli such as natural images, so only relying on linguistic stimuli cannot fully understand the information perceived by the human brain. To address this, an end-to-end mapping model between visual neural activities evoked by non-linguistic stimuli and visual contents is demanded.

METHODS

Inspired by the success of the Transformer network in neural machine translation and the convolutional neural network (CNN) in computer vision, here a CNN-Transformer hybrid language decoding model is constructed in an end-to-end fashion to decode functional magnetic resonance imaging (fMRI) signals evoked by natural images into descriptive texts about the visual stimuli. Specifically, this model first encodes a semantic sequence extracted by a two-layer 1D CNN from the multi-time visual neural activity into a multi-level abstract representation, then decodes this representation, step by step, into an English sentence.

RESULTS

Experimental results show that the decoded texts are semantically consistent with the corresponding ground truth annotations. Additionally, by varying the encoding and decoding layers and modifying the original positional encoding of the Transformer, we found that a specific architecture of the Transformer is required in this work.

CONCLUSIONS

The study results indicate that the proposed model can decode the visual neural activities evoked by natural images into descriptive text about the visual stimuli in the form of sentences. Hence, it may be considered as a potential computer-aided tool for neuroscientists to understand the neural mechanism of visual information processing in the human brain in the future.

The neural representation of abstract words may arise through grounding word meaning in language itself

In order to describe how humans represent meaning in the brain, one must be able to account for not just concrete words but, critically, also abstract words, which lack a physical referent. Hebbian formalism and optimization are basic principles of brain function, and they provide an appealing approach for modeling word meanings based on word co‐occurrences. We provide proof of concept that a statistical model of the semantic space can account for neural representations of both concrete and abstract words, using MEG. Here, we built a statistical model using word embeddings extracted from a text corpus. This statistical model was used to train a machine learning algorithm to successfully decode the MEG signals evoked by written words. In the model, word abstractness emerged from the statistical regularities of the language environment. Representational similarity analysis further showed that this salient property of the model co‐varies, at 280–420 ms after visual word presentation, with activity in regions that have been previously linked with processing of abstract words, namely the left‐hemisphere frontal, anterior temporal and superior parietal cortex. In light of these results, we propose that the neural encoding of word meanings can arise through statistical regularities, that is, through grounding in language itself.

Inferior parietal lobule is sensitive to different semantic similarity relations for concrete and abstract words

Similarity measures, the extent to which two concepts have similar meanings, are the key to understand how concepts are represented, with different theoretical perspectives relying on very different sources of data from which similarity can be calculated. While there is some commonality in similarity measures, the extent of their correlation is limited. Previous studies also suggested that the relative performance of different similarity measures may also vary depending on concept concreteness and that the inferior parietal lobule (IPL) may be involved in the integration of conceptual features in a multimodal system for the semantic categorization. Here, we tested for the first time whether theory-based similarity measures predict the pattern of brain activity in the IPL differently for abstract and concrete concepts. English speakers performed a semantic decision task, while we recorded their brain activity in IPL through fNIRS. Using representational similarity analysis, results indicated that the neural representational similarity in IPL conformed to the lexical co-occurrence among concrete concepts (regardless of the hemisphere) and to the affective similarity among abstract concepts in the left hemisphere only, implying that semantic representations of abstract and concrete concepts are characterized along different organizational principles in the IPL. We observed null results for the decoding accuracy. Our study suggests that the use of the representational similarity analysis as a complementary analysis to the decoding accuracy is a promising tool to reveal similarity patterns between theoretical models and brain activity recorded through fNIRS.

Functional magnetic resonance imaging multivoxel pattern analysis reveals neuronal substrates for collaboration and competition with myopic and predictive strategic reasoning

Competition and collaboration are strategies that can be used to optimize the outcomes of social interactions. Research into the neuronal substrates underlying these aspects of social behavior has been limited due to the difficulty in distinguishing complex activation via univariate analysis. Therefore, we employed multivoxel pattern analysis of functional magnetic resonance imaging to reveal the neuronal activations underlying competitive and collaborative processes when the collaborator/opponent used myopic/predictive reasoning. Twenty‐four healthy subjects participated in 2 × 2 matrix‐based sequential‐move games. Searchlight‐based multivoxel patterns were used as input for a support vector machine using nested cross‐validation to distinguish game conditions, and identified voxels were validated via the regression of the behavioral data with bootstrapping. The left anterior insula (accuracy = 78.5%) was associated with competition, and middle frontal gyrus (75.1%) was associated with predictive reasoning. The inferior/superior parietal lobules (84.8%) and middle frontal gyrus (84.7%) were associated with competition, particularly in trials with a predictive opponent. The visual/motor areas were related to response time as a proxy for visual attention and task difficulty. Our results suggest that multivoxel patterns better represent the neuronal substrates underlying the social cognition of collaboration and competition intermixed with myopic and predictive reasoning than do univariate features.

An fMRI Study of Affective Congruence across Visual and Auditory Modalities

Evaluating multisensory emotional content is a part of normal day-to-day interactions. We used fMRI to examine brain areas sensitive to congruence of audiovisual valence and their overlap with areas sensitive to valence. Twenty-one participants watched audiovisual clips with either congruent or incongruent valence across visual and auditory modalities. We showed that affective congruence versus incongruence across visual and auditory modalities is identifiable on a trial-by-trial basis across participants. Representations of affective congruence were widely distributed with some overlap with the areas sensitive to valence. Regions of overlap included bilateral superior temporal cortex and right pregenual anterior cingulate. The overlap between the regions identified here and in the emotion congruence literature lends support to the idea that valence may be a key determinant of affective congruence processing across a variety of discrete emotions.

Inter-participant consistency of language-processing networks during abstract thoughts

Human brain imaging typically employs structured and controlled tasks to avoid variable and inconsistent activation patterns. Here we expand this assumption by showing that an extremely open-ended, high-level cognitive task of thinking about an abstract content, loosely defined as "abstract thinking" - leads to highly consistent activation maps. Specifically, we show that activation maps generated during such cognitive process were precisely located relative to borders of well-known networks such as internal speech, visual and motor imagery. The activation patterns allowed decoding the thought condition at >95%. Surprisingly, the activated networks remained the same regardless of changes in thought content. Finally, we found remarkably consistent activation maps across individuals engaged in abstract thinking. This activation bordered, but strictly avoided visual and motor networks. On the other hand, it overlapped with left lateralized language networks. Activation of the default mode network (DMN) during abstract thought was similar to DMN activation during rest. These observations were supported by a quantitative neuronal distance metric analysis. Our results reveal that despite its high level, and varied content nature - abstract thinking activates surprisingly precise and consistent networks in participants' brains.

Representation of associative and affective semantic similarity of abstract words in the lateral temporal perisylvian language regions

The examination of semantic cognition has traditionally identified word concreteness as well as valence as two of the principal dimensions in the representation of conceptual knowledge. More recently, corpus-based vector space models as well as graph-theoretical analysis of large-scale task-related behavioural responses have revolutionized our insight into how the meaning of words is structured. In this fMRI study, we apply representational similarity analysis to investigate the conceptual representation of abstract words. Brain activity patterns were related to a cued-association based graph as well as to a vector-based co-occurrence model of word meaning. Twenty-six subjects (19 females and 7 males) performed an overt repetition task during fMRI. First, we performed a searchlight classification procedure to identify regions where activity is discriminable between abstract and concrete words. These regions were left inferior frontal gyrus, the upper and lower bank of the superior temporal sulcus bilaterally, posterior middle temporal gyrus and left fusiform gyrus. Representational Similarity Analysis demonstrated that for abstract words, the similarity of activity patterns in the cortex surrounding the superior temporal sulcus bilaterally and in the left anterior superior temporal gyrus reflects the similarity in word meaning. These effects were strongest for semantic similarity derived from the cued association-based graph and for affective similarity derived from either of the two models. The latter effect was mainly driven by positive valence words. This research highlights the close neurobiological link between the information structure of abstract and affective word content and the similarity in activity pattern in the lateral and anterior temporal language system.

Differential Contributions of Default and Dorsal Attention Networks to Remembering Thoughts and External Stimuli From Real-Life Events

Episodic memories are typically composed of perceptual information derived from the external environment and representations of internal states (e.g., one's thoughts during prior episodes). To date, however, research has mostly focused on the remembrance of external stimuli, such that little is known about how internal mentation is represented within episodic memory. In the present fMRI study, we examined the neural correlates of these 2 components of episodic memories using a novel method of cuing memories from photographs taken during real-life events. We found that, compared with corresponding semantic memory tasks, memories for internal thoughts and external elements were associated with activity in brain areas supporting episodic recollection. Most importantly, however, the 2 kinds of memories also showed differential activation in large-scale brain networks: the remembrance of external elements was associated with greater activity in the dorsal attention network, whereas memories of internal thoughts mainly recruited default network areas. These findings shed new light on the representation of internal and external aspects of prior experience within episodic memory. The default network may contribute to the reinstatement of thoughts experienced during past events, whereas the dorsal attention network may support the allocation of attention to visuospatial features within episodic memory representations.

A Double Dissociation in Sensitivity to Verb and Noun Semantics Across Cortical Networks

What is the neural organization of the mental lexicon? Previous research suggests that partially distinct cortical networks are active during verb and noun processing, but what information do these networks represent? We used multivoxel pattern analysis (MVPA) to investigate whether these networks are sensitive to lexicosemantic distinctions among verbs and among nouns and, if so, whether they are more sensitive to distinctions among words in their preferred grammatical class. Participants heard 4 types of verbs (light emission, sound emission, hand-related actions, mouth-related actions) and 4 types of nouns (birds, mammals, manmade places, natural places). As previously shown, the left posterior middle temporal gyrus (LMTG+), and inferior frontal gyrus (LIFG) responded more to verbs, whereas the inferior parietal lobule (LIP), precuneus (LPC), and inferior temporal (LIT) cortex responded more to nouns. MVPA revealed a double-dissociation in lexicosemantic sensitivity: classification was more accurate among verbs than nouns in the LMTG+, and among nouns than verbs in the LIP, LPC, and LIT. However, classification was similar for verbs and nouns in the LIFG, and above chance for the nonpreferred category in all regions. These results suggest that the lexicosemantic information about verbs and nouns is represented in partially nonoverlapping networks.

Neural Representations of Abstract Concepts: Identifying Underlying Neurosemantic Dimensions

The abstractness of concepts is sometimes defined indirectly as lacking concreteness, this view provides little insight into their cognitive or neural basis. Multivariate pattern analytic techniques applied to functional magnetic resonance imaging data were used to characterize the neural representations of 28 individual abstract concepts. A classifier trained on the concepts’ neural signatures reliably decoded their neural representations in an independent subset of data for each participant. There was considerable commonality of the neural representations across participants as indicated by the accurate classification of each participant’s concepts based on the neural signatures obtained in other participants. Group-level factor analysis revealed 3 semantic dimensions underlying the 28 concepts, suggesting a brain-based ontology for this set of abstract concepts. The 3 dimensions corresponded to 1) the degree a concept was Verbally Represented; 2) whether a concept was External (or Internal) to the individual, and 3) whether the concept contained Social Content. Further exploration of the Verbal Representation dimension suggests that the degree a concept is verbally represented can be construed as a point on a continuum between language faculties and perceptual faculties. A predictive model, based on independent behavioral ratings of the 28 concepts along the 3 factor dimensions, provided converging evidence for the interpretations.

Duality of Function: Activation for Meaningless Nonwords and Semantic Codes in the Same Brain Areas

Studies of the neural substrates of semantic (word meaning) processing have typically focused on semantic manipulations, with less consideration for potential differences in difficulty across conditions. While the idea that particular brain regions can support multiple functions is widely accepted, studies of specific cognitive domains rarely test for co-location with other functions. Here we start with standard univariate analyses comparing words to meaningless nonwords, replicating our recent finding that this contrast can activate task-positive regions for words, and default-mode regions in the putative semantic network for nonwords, pointing to difficulty effects. Critically, this was followed up with a multivariate analysis to test whether the same areas activated for meaningless nonwords contained semantic information sufficient to distinguish high- from low-imageability words. Indeed, this classification was performed reliably better than chance at 75% accuracy. This is compatible with two non-exclusive interpretations. Numerous areas in the default-mode network are task-negative in the sense of activating for less demanding conditions, and the same areas contain information supporting semantic cognition. Therefore, while areas of the default mode network have been hypothesized to support semantic cognition, we offer evidence that these areas can respond to both domain-general difficulty effects, and to specific aspects of semantics.

Distinguishing abstract from concrete concepts in supramodal brain regions

Concrete words have been shown to have a processing advantage over abstract words, yet theoretical accounts and neural correlates underlying the distinction between concrete and abstract concepts are still unresolved. In an fMRI study, participants performed a property verification task on abstract and concrete concepts. Property comparisons of concrete concepts were predominantly based on either visual or haptic features. Multivariate pattern analysis successfully distinguished between abstract and concrete stimulus comparisons at the whole brain level. Multivariate searchlight analyses showed that posterior and middle cingulate cortices contained information that distinguished abstract from concrete concepts regardless of feature dominance. These results support the view that supramodal convergence zones play an important role in representation of concrete and abstract concepts.

Sentential negation of abstract and concrete conceptual categories: a brain decoding multivariate pattern analysis study

We rarely use abstract and concrete concepts in isolation but rather embedded within a linguistic context. To examine the modulatory impact of the linguistic context on conceptual processing, we isolated the case of sentential negation polarity, in which an interaction occurs between the syntactic operator not and conceptual information in the negation's scope. Previous studies suggested that sentential negation of concrete action-related concepts modulates activation in the fronto-parieto-temporal action representation network. In this functional magnetic resonance imaging study, we examined the influence of negation on a wider spectrum of meanings, by factorially manipulating sentence polarity (affirmative, negative) and fine-grained abstract (mental state, emotion, mathematics) and concrete (related to mouth, hand, leg actions) conceptual categories. We adopted a multivariate pattern analysis approach, and tested the accuracy of a machine learning classifier in discriminating brain activation patterns associated to the factorial manipulation. Searchlight analysis was used to localize the discriminating patterns. Overall, the neural processing of affirmative and negative sentences with either an abstract or concrete content could be accurately predicted by means of multivariate classification. We suggest that sentential negation polarity modulates brain activation in distributed representational semantic networks, through the functional mediation of syntactic and cognitive control systems.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.

Distinct Functional Network Connectivity for Abstract and Concrete Mental Imagery

In several behavioral psycholinguistic studies, it has been shown that concrete words are processed more efficiently. They can be remembered faster, recognized better, and can be learned easier than abstract words. This fact is called concreteness effect. There are fMRI studies which compared the neural representations of concrete and abstract concepts in terms of activated regions. In the present study, a comparison has been made between the condition-specific connectivity of functional networks (obtained by group ICA) during imagery of abstract and concrete words. The obtained results revealed that the functional network connectivity between three pairs of networks during concrete imagery is significantly different from that of abstract imagery (FDR correction at the significance level of 0.05). These results suggest that abstract and concrete concepts have different representations in terms of functional network connectivity pattern. Remarkably, in all of these network pairs, the connectivity during concrete imagery is significantly higher than that of abstract imagery. These more coherent networks include both linguistic and visual regions with a higher engagement of the right hemisphere, so the results are in line with dual coding theory. Additionally, these three pairs of networks include the contrasting regions which have shown stronger activation either in concrete or abstract word processing in former studies. The findings imply that the brain is more integrated and synchronized at the time of concrete imagery and it may explain the reason of faster concrete words processing. In order to validate the results, we used functional network connectivity distributions (FNCD). Wilcoxon rank-sum test was used to check if the abstract and concrete FNCDs extracted from whole subjects are the same. The result revealed that the corresponding distributions are different which indicates two different patterns of connectivity for abstract and concrete word processing. Also, the mean of FNCD is significantly higher at the time of concrete imagery than that of abstract imagery. Furthermore, FNCDs at the single-subject level are significantly more left-skewed or equally, include more strong connectivity for concrete imagery.

Semantic Features Reveal Different Networks During Word Processing: An EEG Source Localization Study

The neural principles behind semantic category representation are still under debate. Dominant theories mostly focus on distinguishing concrete from abstract concepts but, in such theories, divisions into categories of concrete concepts are more developed than for their abstract counterparts. An encompassing theory on semantic category representation could be within reach when charting the semantic attributes that are capable of describing both concept types. A good candidate are the three semantic dimensions defined by Osgood (potency, valence, arousal). However, to show to what extent they affect semantic processing, specific neuroimaging tools are required. Electroencephalography (EEG) is on par with the temporal resolution of cognitive behavior and source reconstruction. Using high-density set-ups, it is able to yield a spatial resolution in the scale of millimeters, sufficient to identify anatomical brain parcellations that could differentially contribute to semantic category representation. Cognitive neuroscientists traditionally focus on scalp domain analysis and turn to source reconstruction when an effect in the scalp domain has been detected. Traditional methods will potentially miss out on the fine-grained effects of semantic features as they are possibly obscured by the mixing of source activity due to volume conduction. For this reason, we have developed a mass-univariate analysis in the source domain using a mixed linear effect model. Our analyses reveal distinct networks of sources for different semantic features that are active during different stages of lexico-semantic processing of single words. With our method we identified differences in the spatio-temporal activation patterns of abstract and concrete words, high and low potency words, high and low valence words, and high and low arousal words, and in this way shed light on how word categories are represented in the brain.

Brain reading and behavioral methods provide complementary perspectives on the representation of concepts

The advent of brain reading techniques has enabled new approaches to the study of concept representation, based on the analysis of multivoxel activation patterns evoked by the contemplation of individual concepts such as animal concepts. The present fMRI study characterized the representation of 30 animal concepts. Dimensionality reduction of the multivoxel activation patterns underlying the individual animal concepts indicated that the semantic building blocks of the brain's representations of the animals corresponded to intrinsic animal properties (e.g. fierceness, intelligence, size). These findings were compared to behavioral studies of concept representation, which have typically collected pairwise similarity ratings between two concepts (e.g. Henley, 1969). Behavioral similarity judgments, by contrast, indicated that the animals were organized into taxonomically defined groups (e.g. canine, feline, equine). The difference in the results between the brain reading and behavioral approaches might derive from differences in cognitive processing during judging similarities versus contemplating one animal at a time. Brain reading approaches may have an advantage in describing thoughts about an individual concept, owing to the ability to decode brain activation patterns elicited by the brief consideration of a single concept (e.g. word reading) without a complex cognitive or behavioral task (e.g. similarity judgments). On the other hand, some behavioral tasks may tend to evoke a concept from numerous perspectives, yielding a representation of the breadth and sophistication of the concept knowledge. These results suggest that neural and behavioral measures offer complementary perspectives that together characterize the content and structure of concept representations.

Just above Chance: Is It Harder to Decode Information from Prefrontal Cortex Hemodynamic Activity Patterns?

The prefrontal cortex (PFC) is central to flexible, goal-directed cognition, and understanding its representational code is an important problem in cognitive neuroscience. In humans, multivariate pattern analysis (MVPA) of fMRI blood oxygenation level-dependent (BOLD) measurements has emerged as an important approach for studying neural representations. Many previous studies have implicitly assumed that MVPA of fMRI BOLD is just as effective in decoding information encoded in PFC neural activity as it is in visual cortex. However, MVPA studies of PFC have had mixed success. Here we estimate the base rate of decoding information from PFC BOLD activity patterns from a meta-analysis of published MVPA studies. We show that PFC has a significantly lower base rate (55.4%) than visual areas in occipital (66.6%) and temporal (71.0%) cortices and one that is close to chance levels. Our results have implications for the design and interpretation of MVPA studies of PFC and raise important questions about its functional organization.

Predicting the brain activation pattern associated with the propositional content of a sentence: Modeling neural representations of events and states

Even though much has recently been learned about the neural representation of individual concepts and categories, neuroimaging research is only beginning to reveal how more complex thoughts, such as event and state descriptions, are neurally represented. We present a predictive computational theory of the neural representations of individual events and states as they are described in 240 sentences. Regression models were trained to determine the mapping between 42 neurally plausible semantic features (NPSFs) and thematic roles of the concepts of a proposition and the fMRI activation patterns of various cortical regions that process different types of information. Given a semantic characterization of the content of a sentence that is new to the model, the model can reliably predict the resulting neural signature, or, given an observed neural signature of a new sentence, the model can predict its semantic content. The models were also reliably generalizable across participants. This computational model provides an account of the brain representation of a complex yet fundamental unit of thought, namely, the conceptual content of a proposition. In addition to characterizing a sentence representation at the level of the semantic and thematic features of its component concepts, factor analysis was used to develop a higher level characterization of a sentence, specifying the general type of event representation that the sentence evokes (e.g., a social interaction versus a change of physical state) and the voxel locations most strongly associated with each of the factors. Hum Brain Mapp 38:4865-4881, 2017. © 2017 Wiley Periodicals, Inc.

A brain-based account of "basic-level" concepts

This study provides a brain-based account of how object concepts at an intermediate (basic) level of specificity are represented, offering an enriched view of what it means for a concept to be a basic-level concept, a research topic pioneered by Rosch and others (Rosch et al., 1976). Applying machine learning techniques to fMRI data, it was possible to determine the semantic content encoded in the neural representations of object concepts at basic and subordinate levels of abstraction. The representation of basic-level concepts (e.g. bird) was spatially broad, encompassing sensorimotor brain areas that encode concrete object properties, and also language and heteromodal integrative areas that encode abstract semantic content. The representation of subordinate-level concepts (robin) was less widely distributed, concentrated in perceptual areas that underlie concrete content. Furthermore, basic-level concepts were representative of their subordinates in that they were neurally similar to their typical but not atypical subordinates (bird was neurally similar to robin but not woodpecker). The findings provide a brain-based account of the advantages that basic-level concepts enjoy in everyday life over subordinate-level concepts: the basic level is a broad topographical representation that encompasses both concrete and abstract semantic content, reflecting the multifaceted yet intuitive meaning of basic-level concepts.

Neural representations of the concepts in simple sentences: Concept activation prediction and context effects

Although it has been possible to identify individual concepts from a concept's brain activation pattern, there have been significant obstacles to identifying a proposition from its fMRI signature. Here we demonstrate the ability to decode individual prototype sentences from readers' brain activation patterns, by using theory-driven regions of interest and semantic properties. It is possible to predict the fMRI brain activation patterns evoked by propositions and words which are entirely new to the model with reliably above-chance rank accuracy. The two core components implemented in the model that reflect the theory were the choice of intermediate semantic features and the brain regions associated with the neurosemantic dimensions. This approach also predicts the neural representation of object nouns across participants, studies, and sentence contexts. Moreover, we find that the neural representation of an agent-verb-object proto-sentence is more accurately characterized by the neural signatures of its components as they occur in a similar context than by the neural signatures of these components as they occur in isolation.

Neural Representations of Physics Concepts

We used functional MRI (fMRI) to assess neural representations of physics concepts (momentum, energy, etc.) in juniors, seniors, and graduate students majoring in physics or engineering. Our goal was to identify the underlying neural dimensions of these representations. Using factor analysis to reduce the number of dimensions of activation, we obtained four physics-related factors that were mapped to sets of voxels. The four factors were interpretable as causal motion visualization, periodicity, algebraic form, and energy flow. The individual concepts were identifiable from their fMRI signatures with a mean rank accuracy of .75 using a machine-learning (multivoxel) classifier. Furthermore, there was commonality in participants' neural representation of physics; a classifier trained on data from all but one participant identified the concepts in the left-out participant (mean accuracy = .71 across all nine participant samples). The findings indicate that abstract scientific concepts acquired in an educational setting evoke activation patterns that are identifiable and common, indicating that science education builds abstract knowledge using inherent, repurposed brain systems.

The neural basis of a deficit in abstract thinking in patients with schizophrenia

Abnormal abstract thinking is a major cause of social dysfunction in patients with schizophrenia, but little is known about its neural basis. In this study, we aimed to determine the characteristic abstract thinking-related brain responses in patients using a task reflecting social situations. We conducted functional magnetic resonance imaging while 16 patients with schizophrenia and 16 healthy controls performed a theme-identification task, in which various emotional pictures depicting social situations were presented. Compared with healthy controls, the patients showed significantly decreased activity in the left frontopolar and right orbitofrontal cortices during theme identification. Activity in these two regions correlated well in the controls, but not in patients. Instead, the patients exhibited a close correlation between activity in both sides of the frontopolar cortex, and a positive correlation between the right orbitofrontal cortex activity and degrees of theme identification. Reduced activity in the left frontopolar and right orbitofrontal cortices and the underlying aberrant connectivity may be implicated in the patients' deficits in abstract thinking. These newly identified features of the neural basis of abnormal abstract thinking are important as they have implications for the impaired social behavior of patients with schizophrenia during real-life situations.

Remembering Preservation in Hippocampal Amnesia

The lesion-deficit model dominates neuropsychology. This is unsurprising given powerful demonstrations that focal brain lesions can affect specific aspects of cognition. Nowhere is this more evident than in patients with bilateral hippocampal damage. In the past 60 years, the amnesia and other impairments exhibited by these patients have helped to delineate the functions of the hippocampus and shape the field of memory. We do not question the value of this approach. However, less prominent are the cognitive processes that remain intact following hippocampal lesions. Here, we collate the piecemeal reports of preservation of function following focal bilateral hippocampal damage, highlighting a wealth of information often veiled by the field's focus on deficits. We consider how a systematic understanding of what is preserved as well as what is lost could add an important layer of precision to models of memory and the hippocampus.

Classification of self-driven mental tasks from whole-brain activity patterns

During wakefulness, a constant and continuous stream of complex stimuli and self-driven thoughts permeate the human mind. Here, eleven participants were asked to count down numbers and remember negative or positive autobiographical episodes of their personal lives, for 32 seconds at a time, during which they could freely engage in the execution of those tasks. We then examined the possibility of determining from a single whole-brain functional magnetic resonance imaging scan which one of the two mental tasks each participant was performing at a given point in time. Linear support-vector machines were used to build within-participant classifiers and across-participants classifiers. The within-participant classifiers could correctly discriminate scans with an average accuracy as high as 82%, when using data from all individual voxels in the brain. These results demonstrate that it is possible to accurately classify self-driven mental tasks from whole-brain activity patterns recorded in a time interval as short as 2 seconds.

Discriminating Taxonomic Categories and Domains in Mental Simulations of Concepts of Varying Concreteness

Most studies of conceptual knowledge in the brain focus on a narrow range of concrete conceptual categories, rely on the researchers' intuitions about which object belongs to these categories, and assume a broadly taxonomic organization of knowledge. In this fMRI study, we focus on concepts with a variety of concreteness levels; we use a state of the art lexical resource (WordNet 3.1) as the source for a relatively large number of category distinctions and compare a taxonomic style of organization with a domain-based model (an example domain is Law). Participants mentally simulated situations associated with concepts when cued by text stimuli. Using multivariate pattern analysis, we find evidence that all Taxonomic categories and Domains can be distinguished from fMRI data and also observe a clear concreteness effect: Tools and Locations can be reliably predicted for unseen participants, but less concrete categories (e.g., Attributes, Communications, Events, Social Roles) can only be reliably discriminated within participants. A second concreteness effect relates to the interaction of Domain and Taxonomic category membership: Domain (e.g., relation to Law vs. Music) can be better predicted for less concrete categories. We repeated the analysis within anatomical regions, observing discrimination between all/most categories in the left mid occipital and left mid temporal gyri, and more specialized discrimination for concrete categories Tool and Location in the left precentral and fusiform gyri, respectively. Highly concrete/abstract Taxonomic categories and Domain were segregated in frontal regions. We conclude that both Taxonomic and Domain class distinctions are relevant for interpreting neural structuring of concrete and abstract concepts.

Representations of modality-specific affective processing for visual and auditory stimuli derived from functional magnetic resonance imaging data

There is converging evidence that people rapidly and automatically encode affective dimensions of objects, events, and environments that they encounter in the normal course of their daily routines. An important research question is whether affective representations differ with sensory modality. This research examined the nature of the dependency of affect and sensory modality at a whole-brain level of analysis in an incidental affective processing paradigm. Participants were presented with picture and sound stimuli that differed in positive or negative valence in an event-related functional magnetic resonance imaging experiment. Global statistical tests, applied at a level of the individual, demonstrated significant sensitivity to valence within modality, but not valence across modalities. Modality-general and modality-specific valence hypotheses predict distinctly different multidimensional patterns of the stimulus conditions. Examination of lower dimensional representation of the data demonstrated separable dimensions for valence processing within each modality. These results provide support for modality-specific valence processing in an incidental affective processing paradigm at a whole-brain level of analysis. Future research should further investigate how stimulus-specific emotional decoding may be mediated by the physical properties of the stimuli.

Examining similarity structure: multidimensional scaling and related approaches in neuroimaging

This paper covers similarity analyses, a subset of multivariate pattern analysis techniques that are based on similarity spaces defined by multivariate patterns. These techniques offer several advantages and complement other methods for brain data analyses, as they allow for comparison of representational structure across individuals, brain regions, and data acquisition methods. Particular attention is paid to multidimensional scaling and related approaches that yield spatial representations or provide methods for characterizing individual differences. We highlight unique contributions of these methods by reviewing recent applications to functional magnetic resonance imaging data and emphasize areas of caution in applying and interpreting similarity analysis methods.

Limiting factors for mapping corpus-based semantic representations to brain activity

To help understand how semantic information is represented in the human brain, a number of previous studies have explored how a linear mapping from corpus derived semantic representations to corresponding patterns of fMRI brain activations can be learned. They have demonstrated that such a mapping for concrete nouns is able to predict brain activations with accuracy levels significantly above chance, but the more recent elaborations have achieved relatively little performance improvement over the original study. In fact, the absolute accuracies of all these models are still currently rather limited, and it is not clear which aspects of the approach need improving in order to achieve performance levels that might lead to better accounts of human capabilities. This paper presents a systematic series of computational experiments designed to identify the limiting factors of the approach. Two distinct series of artificial brain activation vectors with varying levels of noise are introduced to characterize how the brain activation data restricts performance, and improved corpus based semantic vectors are developed to determine how the word set and model inputs affect the results. These experiments lead to the conclusion that the current state-of-the-art input semantic representations are already operating nearly perfectly (at least for non-ambiguous concrete nouns), and that it is primarily the quality of the fMRI data that is limiting what can be achieved with this approach. The results allow the study to end with empirically informed suggestions about the best directions for future research in this area.