Multivariate Pattern Analysis Reveals Category-Related Organization of Semantic Representations in Anterior Temporal Cortex

A double dissociation between semantic and spatial cognition in visual to default network pathways

Processing pathways between sensory and default mode network (DMN) regions support recognition, navigation, and memory but their organisation is not well understood. We show that functional subdivisions of visual cortex and DMN sit at opposing ends of parallel streams of information processing that support visually mediated semantic and spatial cognition, providing convergent evidence from univariate and multivariate task responses, intrinsic functional and structural connectivity. Participants learned virtual environments consisting of buildings populated with objects, drawn from either a single semantic category or multiple categories. Later, they made semantic and spatial context decisions about these objects and buildings during functional magnetic resonance imaging. A lateral ventral occipital to fronto-temporal DMN pathway was primarily engaged by semantic judgements, while a medial visual to medial temporal DMN pathway supported spatial context judgements. These pathways had distinctive locations in functional connectivity space: the semantic pathway was both further from unimodal systems and more balanced between visual and auditory-motor regions compared with the spatial pathway. When semantic and spatial context information could be integrated (in buildings containing objects from a single category), regions at the intersection of these pathways responded, suggesting that parallel processing streams interact at multiple levels of the cortical hierarchy to produce coherent memory-guided cognition.

From long-term to short-term: Distinct neural networks underlying semantic knowledge and its recruitment in working memory

Although numerous studies suggest that working memory (WM) and semantic long-term knowledge interact, the nature and underlying neural mechanisms of this intervention remain poorly understood. Using functional magnetic resonance imaging (fMRI), this study investigated the extent to which neural markers of semantic knowledge in long-term memory (LTM) are activated during the WM maintenance stage in 32 young adults. First, the multivariate neural patterns associated with four semantic categories were determined via an implicit semantic activation task. Next, the participants maintained words - the names of the four semantic categories implicitly activated in the first task - in a verbal WM task. Multi-voxel pattern analyses showed reliable neural decoding of the four semantic categories in the implicit semantic activation and the verbal WM tasks. Critically, however, no between-task classification of semantic categories was observed. Searchlight analyses showed that for the WM task, semantic category information could be decoded in anterior temporal areas associated with abstract semantic category knowledge. In the implicit semantic activation task, semantic category information was decoded in superior temporal, occipital and frontal cortices associated with domain-specific semantic feature representations. These results indicate that item-level semantic activation during verbal WM involves shallow rather than deep semantic information.

Graded and sharp transitions in semantic function in left temporal lobe

Recent work has focussed on how patterns of functional change within the temporal lobe relate to whole-brain dimensions of intrinsic connectivity variation (Margulies et al., 2016). We examined two such 'connectivity gradients' reflecting the separation of (i) unimodal versus heteromodal and (ii) visual versus auditory-motor cortex, examining visually presented verbal associative and feature judgments, plus picture-based context and emotion generation. Functional responses along the first dimension sometimes showed graded change between modality-tuned and heteromodal cortex (in the verbal matching task), and other times showed sharp functional transitions, with deactivation at the extremes and activation in the middle of this gradient (internal generation). The second gradient revealed more visual than auditory-motor activation, regardless of content (associative, feature, context, emotion) or task process (matching/generation). We also uncovered subtle differences across each gradient for content type, which predominantly manifested as differences in relative magnitude of activation or deactivation.

Evidence for a Spoken Word Lexicon in the Auditory Ventral Stream

The existence of a neural representation for whole words (i.e., a lexicon) is a common feature of many models of speech processing. Prior studies have provided evidence for a visual lexicon containing representations of whole written words in an area of the ventral visual stream known as the visual word form area. Similar experimental support for an auditory lexicon containing representations of spoken words has yet to be shown. Using functional magnetic resonance imaging rapid adaptation techniques, we provide evidence for an auditory lexicon in the auditory word form area in the human left anterior superior temporal gyrus that contains representations highly selective for individual spoken words. Furthermore, we show that familiarization with novel auditory words sharpens the selectivity of their representations in the auditory word form area. These findings reveal strong parallels in how the brain represents written and spoken words, showing convergent processing strategies across modalities in the visual and auditory ventral streams.

Context free and context-dependent conceptual representation in the brain

How concepts are coded in the brain is a core issue in cognitive neuroscience. Studies have focused on how individual concepts are processed, but the way in which conceptual representation changes to suit the context is unclear. We parametrically manipulated the association strength between words, presented in pairs one word at a time using a slow event-related fMRI design. We combined representational similarity analysis and computational linguistics to probe the neurocomputational content of these trials. Individual word meaning was maintained in supramarginal gyrus (associated with verbal short-term memory) when items were judged to be unrelated, but not when a linking context was retrieved. Context-dependent meaning was instead represented in left lateral prefrontal gyrus (associated with controlled retrieval), angular gyrus, and ventral temporal lobe (regions associated with integrative aspects of memory). Analyses of informational connectivity, examining the similarity of activation patterns across trials between sites, showed that control network regions had more similar multivariate responses across trials when association strength was weak, reflecting a common controlled retrieval state when the task required more unusual associations. These findings indicate that semantic control and representational sites amplify contextually relevant meanings in trials judged to be related.

False memory confidence depends on the prefrontal reinstatement of true memory

For confidence of memory, a neural basis such as traces of stored memories should be required. However, because false memories have never been stored, the neural basis for false memory confidence remains unclear. Here we monitored the brain activity in participants while they viewed learned or novel objects, subsequently decided whether each presented object was learned and assessed their confidence levels. We found that when novel objects are presented, false memory confidence significantly depends on the shared representations with learned objects in the prefrontal cortex. However, such a tendency was not found in posterior regions including the visual cortex, which may be involved in the processing of perceptual gist. Furthermore, the confidence-dependent shared representations were not observed when participants correctly answered novel objects as non-learned objects. These results demonstrate that false memory confidence is critically based on the reinstatement of high-level semantic gist of stored memories in the prefrontal cortex.

A Data-Driven Functional Mapping of the Anterior Temporal Lobes

Although the anterior temporal lobe (ATL) comprises several anatomic and functional subdivisions, it is often reduced to a homogeneous theoretical entity, such as a domain-general convergence zone, or "hub," for semantic information. Methodological limitations are largely to blame for the imprecise mapping of function to structure in the ATL. There are two major obstacles to using fMRI to identify the precise functional organization of the ATL: the difficult choice of stimuli and tasks to activate, and dissociate, specific regions within the ATL; and poor signal quality because of magnetic field distortions near the sinuses. To circumvent these difficulties, we developed a data-driven parcellation routine using resting-state fMRI data (24 females, 64 males) acquired using a sequence that was optimized to enhance signal in the ATL. Focusing on patterns of functional connectivity between each ATL voxel and the rest of the brain, we found that the ATL comprises at least 34 distinct functional parcels that are arranged into bands along the lateral and ventral cortical surfaces, extending from the posterior temporal lobes into the temporal poles. In addition, the anterior region of the fusiform gyrus, most often cited as the location of the semantic hub, was found to be part of a domain-specific network associated with face and social processing, rather than a domain-general semantic hub. These findings offer a fine-grained functional map of the ATL and offer an initial step toward using more precise language to describe the locations of functional responses in this heterogeneous region of human cortex.SIGNIFICANCE STATEMENT The functional role of the anterior aspects of the temporal lobes (ATL) is a contentious issue. While it is likely that different regions within the ATL subserve unique cognitive functions, most studies revert to vaguely referring to particular functional regions as "the ATL," and, thus, the mapping of function to anatomy remains unclear. We used resting-state fMRI connectivity patterns between the ATL and the rest of the brain to reveal that the ATL comprises at least 34 distinct functional parcels that are organized into a three-level functional hierarchy. These results provide a detailed functional map of the anterior temporal lobes that can guide future research on how distinct regions within the ATL support diverse cognitive functions.

Leveraging Prior Concept Learning Improves Generalization From Few Examples in Computational Models of Human Object Recognition

Humans quickly and accurately learn new visual concepts from sparse data, sometimes just a single example. The impressive performance of artificial neural networks which hierarchically pool afferents across scales and positions suggests that the hierarchical organization of the human visual system is critical to its accuracy. These approaches, however, require magnitudes of order more examples than human learners. We used a benchmark deep learning model to show that the hierarchy can also be leveraged to vastly improve the speed of learning. We specifically show how previously learned but broadly tuned conceptual representations can be used to learn visual concepts from as few as two positive examples; reusing visual representations from earlier in the visual hierarchy, as in prior approaches, requires significantly more examples to perform comparably. These results suggest techniques for learning even more efficiently and provide a biologically plausible way to learn new visual concepts from few examples.

From shape to meaning: Evidence for multiple fast feedforward hierarchies of concept processing in the human brain

A number of fMRI studies have provided support for the existence of multiple concept representations in areas of the brain such as the anterior temporal lobe (ATL) and inferior parietal lobule (IPL). However, the interaction among different conceptual representations remains unclear. To better understand the dynamics of how the brain extracts meaning from sensory stimuli, we conducted a human high-density electroencephalography (EEG) study in which we first trained participants to associate pseudowords with various animal and tool concepts. After training, multivariate pattern classification of EEG signals in sensor and source space revealed the representation of both animal and tool concepts in the left ATL and tool concepts within the left IPL within 250 ​ms. Finally, we used Granger Causality analyses to show that orthography-selective sensors directly modulated activity in the parietal-tool selective cluster. Together, our results provide evidence for distinct but parallel "perceptual-to-conceptual" feedforward hierarchies in the brain.

Neural pattern similarity across concept exemplars predicts memory after a long delay

The irregularities of the world ensure that each interaction we have with a concept is unique. In order to generalize across these unique encounters to form a high-level representation of a concept, we must draw on similarities between exemplars to form new conceptual knowledge that is maintained over a long time. Two neural similarity measures - pattern robustness and encoding-retrieval similarity - are particularly important for predicting memory outcomes. In this study, we used fMRI to measure activity patterns while people encoded and retrieved novel pairings between unfamiliar (Dutch) words and visually presented animal species. We address two underexplored questions: 1) whether neural similarity measures can predict memory outcomes, despite perceptual variability between presentations of a concept and 2) if pattern similarity measures can predict subsequent memory over a long delay (i.e., one month). Our findings indicate that pattern robustness during encoding in brain regions that include parietal and medial temporal areas is an important predictor of subsequent memory. In addition, we found significant encoding-retrieval similarity in the left ventrolateral prefrontal cortex after a month's delay. These findings demonstrate that pattern similarity is an important predictor of memory for novel word-animal pairings even when the concept includes multiple exemplars. Importantly, we show that established predictive relationships between pattern similarity and subsequent memory do not require visually identical stimuli (i.e., are not simply due to low-level visual overlap between stimulus presentations) and are maintained over a month.

Neural Components of Reading Revealed by Distributed and Symbolic Computational Models

Determining how the cognitive components of reading - orthographic, phonological, and semantic representations - are instantiated in the brain has been a longstanding goal of psychology and human cognitive neuroscience. The two most prominent computational models of reading instantiate different cognitive processes, implying different neural processes. Artificial neural network (ANN) models of reading posit non-symbolic, distributed representations. The dual-route cascaded (DRC) model instead suggests two routes of processing, one representing symbolic rules of spelling-sound correspondence, the other representing orthographic and phonological lexicons. These models are not adjudicated by behavioral data and have never before been directly compared in terms of neural plausibility. We used representational similarity analysis to compare the predictions of these models to neural data from participants reading aloud. Both the ANN and DRC model representations corresponded with neural activity. However, ANN model representations correlated to more reading-relevant areas of cortex. When contributions from the DRC model were statistically controlled, partial correlations revealed that the ANN model accounted for significant variance in the neural data. The opposite analysis, examining the variance explained by the DRC model with contributions from the ANN model factored out, revealed no correspondence to neural activity. Our results suggest that ANNs trained using distributed representations provide a better correspondence between cognitive and neural coding. Additionally, this framework provides a principled approach for comparing computational models of cognitive function to gain insight into neural representations.

Association of anticholinergic medications and AD biomarkers with incidence of MCI among cognitively normal older adults

OBJECTIVE

To determine the cognitive consequences of anticholinergic medications (aCH) in cognitively normal older adults as well as interactive effects of genetic and CSF Alzheimer disease (AD) risk factors.

METHODS

A total of 688 cognitively normal participants from the Alzheimer's Disease Neuroimaging Initiative were evaluated (mean age 73.5 years, 49.6% female). Cox regression examined risk of progression to mild cognitive impairment (MCI) over a 10-year period and linear mixed effects models examined 3-year rates of decline in memory, executive function, and language as a function of aCH. Interactions with APOE ε4 genotype and CSF biomarker evidence of AD pathology were also assessed.

RESULTS

aCH+ participants had increased risk of progression to MCI (hazard ratio [HR] 1.47, p = 0.02), and there was a significant aCH × AD risk interaction such that aCH+/ε4+ individuals showed greater than 2-fold increased risk (HR 2.69, p < 0.001) for incident MCI relative to aCH-/ε4-), while aCH+/CSF+) individuals demonstrated greater than 4-fold (HR 4.89, p < 0.001) increased risk relative to aCH-/CSF-. Linear mixed effects models revealed that aCH predicted a steeper slope of decline in memory (t = -2.35, p = 0.02) and language (t = -2.35, p = 0.02), with effects exacerbated in individuals with AD risk factors.

CONCLUSIONS

aCH increased risk of incident MCI and cognitive decline, and effects were significantly enhanced among individuals with genetic risk factors and CSF-based AD pathophysiologic markers. Findings underscore the adverse impact of aCH medications on cognition and the need for deprescribing trials, particularly among individuals with elevated risk for AD.

Dynamic activity patterns in the anterior temporal lobe represents object semantics

The anterior temporal lobe (ATL) is considered a crucial area for the representation of transmodal concepts. Recent evidence suggests that specific regions within the ATL support the representation of individual object concepts, as shown by studies combining multivariate analysis methods and explicit measures of semantic knowledge. This research looks to further our understanding by probing conceptual representations at a spatially and temporally resolved neural scale. Representational similarity analysis was applied to human intracranial recordings from anatomically defined lateral to medial ATL sub-regions. Neural similarity patterns were tested against semantic similarity measures, where semantic similarity was defined by a hybrid corpus-based and feature-based approach. Analyses show that the perirhinal cortex, in the medial ATL, significantly related to semantic effects around 200 to 400 ms, and were greater than more lateral ATL regions. Further, semantic effects were present in low frequency (theta and alpha) oscillatory phase signals. These results provide converging support that more medial regions of the ATL support the representation of basic-level visual object concepts within the first 400 ms, and provide a bridge between prior fMRI and MEG work by offering detailed evidence for the presence of conceptual representations within the ATL.

A systematic review of brain functional connectivity patterns involved in episodic and semantic memory

The study of functional connectivity and declarative memory has lately been focused on finding biomarkers of neuropsychological diseases. However, little is known about its patterns in healthy brains. Thus, in this systematic review we analyze and integrate the findings of 81 publications regarding functional connectivity (measured by fMRI during both task and resting-state) and semantic and episodic memory in healthy adults. Moreover, we discriminate and analyze the main areas and links found in specific memory phases (encoding, storage or retrieval) based on several criteria, such as time length, depth of processing, rewarding value of the information, vividness and amount or kind of details retrieved. There is a certain degree of overlap between the networks of episodic and semantic memory and between the encoding and retrieval stages. Although several differences are pointed out during the article, this calls to attention the need for further empirical studies that actively compare both types of memory, particularly using other baseline conditions apart from the traditional resting state. Indeed, the active involvement of the default mode network in both declarative memory and resting condition suggests the possibility that during rest there is an on-going memory processing. We find support for the 'attention to memory' hypothesis, the memory differentiation model and the appropriate transfer hypothesis, but some evidence is inconsistent with the traditional hub-and-spoke model.

Neural representation of visual concepts in people born blind

How do we represent information without sensory features? How are abstract concepts like “freedom”, devoid of external perceptible referents, represented in the brain? Here, to address the role of sensory information in the neural representation of concepts, we used fMRI to investigate how people born blind process concepts whose referents are imperceptible to them because of their visual nature (“rainbow”, “red”). Activity for these concepts was compared to that of sensorially-perceptible referents (“rain”), classical abstract concepts (“justice”) and concrete concepts (“cup”), providing a gradient between fully concrete and fully abstract concepts in the blind. We find that anterior temporal lobe (ATL) responses track concept perceptibility and objecthood: preference for imperceptible object concepts was found in dorsal ATL, for abstract (non-object, non-referential) concepts in lateral ATL, and for perceptible concepts in medial ATL. These findings point to a new division-of-labor among aspects of ATL in representing conceptual properties that are abstract in different ways.

How are abstract, imperceptible concepts such as ‘freedom’ represented in the brain? Here, the authors use fMRI in people born blind to compare the neural responses for abstract concepts, concrete concepts like ‘rainbow’ which in blind people lack sensory qualities, and concrete concepts sensorily accessible to the blind.

Multivariate classification of neuroimaging data with nested subclasses: Biased accuracy and implications for hypothesis testing

Biological data sets are typically characterized by high dimensionality and low effect sizes. A powerful method for detecting systematic differences between experimental conditions in such multivariate data sets is multivariate pattern analysis (MVPA), particularly pattern classification. However, in virtually all applications, data from the classes that correspond to the conditions of interest are not homogeneous but contain subclasses. Such subclasses can for example arise from individual subjects that contribute multiple data points, or from correlations of items within classes. We show here that in multivariate data that have subclasses nested within its class structure, these subclasses introduce systematic information that improves classifiability beyond what is expected by the size of the class difference. We analytically prove that this subclass bias systematically inflates correct classification rates (CCRs) of linear classifiers depending on the number of subclasses as well as on the portion of variance induced by the subclasses. In simulations, we demonstrate that subclass bias is highest when between-class effect size is low and subclass variance high. This bias can be reduced by increasing the total number of subclasses. However, we can account for the subclass bias by using permutation tests that explicitly consider the subclass structure of the data. We illustrate our result in several experiments that recorded human EEG activity, demonstrating that parametric statistical tests as well as typical trial-wise permutation fail to determine significance of classification outcomes correctly.

When data are analyzed using multivariate pattern classification, any systematic similarities between subsets of trials (e.g. shared physical properties among a subgroup of stimuli, trials belonging to the same session or subject, etc.) form distinct nested subclasses within each class. Pattern classification is sensitive to this kind of structure in the data and uses such groupings to increase classification accuracies even when data from both conditions are sampled from the same distribution, i.e. the null hypothesis is true. Here, we show that the bias is higher for larger subclass variances and that it is directly related to the number of subclasses and the intraclass correlation (ICC). Because the increased classification accuracy in such data sets is not based on class differences, the null distribution should be adjusted to account for this type of bias. To do so, we propose to use blocked permutation testing on subclass levels and show that it can confine the false positive rate to the predefined α-levels.

Cognitive consequences of the left-right asymmetry of atrophy in semantic dementia

Semantic dementia (SD) is a condition in which atrophy to the anterior temporal lobes (ATL) produces a selective deterioration of conceptual knowledge. As this atrophy is always bilateral but usually asymmetrical, differences in performance of the two SD subgroups-with left > right (L > R) versus right > left (R > L) atrophy-constitute a major source of evidence regarding the roles of the left and right sides of this region. We explored this issue using large scale case-series methodology, with a pool of 216 observations of neuropsychological data from 72 patients with SD. Anomia was significantly more severe in the L > R subgroup, even when cases from the two subgroups were matched on severity of comprehension deficits. For subgroups matched on the degree of anomia, we show that asymmetry of atrophy also affected both the nature of the naming errors produced, and the degree of a semantic category effect (living things vs artefacts). A comparison across tasks varying in their loading on verbal and visual processing revealed a greater deficit in object naming for L > R cases and in a picture-based semantic association test for R > L cases; this held true whether severity across subgroups was controlled using pairwise matching or statistically via principal components analysis. Importantly, the size of our sample allowed us to demonstrate considerable individual variation within each of the L > R and R > L subgroups, with consequent overlap between them. Our results paint a clear picture of how asymmetry of atrophy affects cognitive performance in SD, and we discuss the results in terms of two mechanisms that could contribute to these differences: variation in the information involved in semantic representations in the left and right ATL, and preferential connectivity between each ATL and other more modality specific intra-hemispheric regions.

On the partnership between neural representations of object categories and visual features in the ventral visual pathway

A dominant view in the cognitive neuroscience of object vision is that regions of the ventral visual pathway exhibit some degree of category selectivity. However, recent findings obtained with multivariate pattern analyses (MVPA) suggest that apparent category selectivity in these regions is dependent on more basic visual features of stimuli. In which case a rethinking of the function and organization of the ventral pathway may be in order. We suggest that addressing this issue of functional specificity requires clear coding hypotheses, about object category and visual features, which make contrasting predictions about neuroimaging results in ventral pathway regions. One way to differentiate between categorical and featural coding hypotheses is to test for residual categorical effects: effects of category selectivity that cannot be accounted for by visual features of stimuli. A strong method for testing these effects, we argue, is to make object category and target visual features orthogonal in stimulus design. Recent studies that adopt this approach support a feature-based categorical coding hypothesis according to which regions of the ventral stream do indeed code for object category, but in a format at least partially based on the visual features of stimuli.

What semantic dementia teaches us about the functional organization of the left posterior fusiform gyrus

After demonstrating the relative preservation of fruit and vegetable knowledge in patients with semantic dementia (SD), we sought to identify the neural substrate of this unusual category effect. Nineteen patients with SD performed a semantic sorting task and underwent a morphometric 3T MRI scan. The grey-matter volumes of five regions within the temporal lobe were bilaterally computed, as well as those of two recently described areas (FG1 and FG2) within the posterior fusiform gyrus. In contrast to the other semantic categories we tested, fruit and vegetable scores were only predicted by left FG1 volume. We therefore found a specific relationship between the volume of a subregion within the left posterior fusiform gyrus and performance on fruits and vegetables in SD. We argue that the left FG1 is a convergence zone for the features that might be critical to successfully sort fruits and vegetables. We also discuss evidence for a functional specialization of the fusiform gyrus along two axes (lateral medial and longitudinal), depending on the nature of the concepts and on the level of processing complexity required by the ongoing task.

Enhanced verbal abilities in the congenitally blind

Numerous studies have found that congenitally blind individuals have better verbal memory than their normally sighted counterparts. However, it is not known whether this reflects superiority of verbal or memory abilities. In order to distinguish between these possibilities, we tested congenitally blind participants and normally sighted control participants, matched for age and education, on a range of verbal and spatial tasks. Congenitally blind participants were significantly better than sighted controls on all the verbal tasks but the groups did not differ significantly on the spatial tasks. Thus, the congenitally blind appear to have superior verbal, but not spatial, abilities. This may reflect greater reliance on verbal information and the involvement of visual cortex in language processing in the congenitally blind.