Object-specific semantic coding in human perirhinal cortex

Connectivity analyses for task-based fMRI

Functional connectivity is conventionally defined by measuring the similarity between brain signals from two regions. The technique has become widely adopted in the analysis of functional magnetic resonance imaging (fMRI) data, where it has provided cognitive neuroscientists with abundant information on how brain regions interact to support complex cognition. However, in the past decade the notion of "connectivity" has expanded in both the complexity and heterogeneity of its application to cognitive neuroscience, resulting in greater difficulty of interpretation, replication, and cross-study comparisons. In this paper, we begin with the canonical notions of functional connectivity and then introduce recent methodological developments that either estimate some alternative form of connectivity or extend the analytical framework, with the hope of bringing better clarity for cognitive neuroscience researchers.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organisation of memories remains unclear. Here, we designed a novel task - the Emotion Boundary Task - wherein participants viewed sequences of negative and neutral images surrounded by a colour border. We manipulated perceptual context (border colour), emotional-picture valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create events with a shared perceptual and/or emotional context. We measured memory for temporal order and temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distances were remembered as longer for images spanning neutral-to-negative shifts - suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently-presented negative (versus neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral-to-negative events produce temporal dilation in memory, with implications for adaptive emotional functioning.

How does the "default mode" network contribute to semantic cognition?

This review examines whether and how the "default mode" network (DMN) contributes to semantic processing. We review evidence implicating the DMN in the processing of individual word meanings and in sentence- and discourse-level semantics. Next, we argue that the areas comprising the DMN contribute to semantic processing by coordinating and integrating the simultaneous activity of local neuronal ensembles across multiple unimodal and multimodal cortical regions, creating a transient, global neuronal ensemble. The resulting ensemble implements an integrated simulation of phenomenological experience - that is, an embodied situation model - constructed from various modalities of experiential memory traces. These situation models, we argue, are necessary not only for semantic processing but also for aspects of cognition that are not traditionally considered semantic. Although many aspects of this proposal remain provisional, we believe it provides new insights into the relationships between semantic and non-semantic cognition and into the functions of the DMN.

Preliminary Evidence for Global Properties in Human Listeners During Natural Auditory Scene Perception

Theories of auditory and visual scene analysis suggest the perception of scenes relies on the identification and segregation of objects within it, resembling a detail-oriented processing style. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. It is our understanding that a similar line of research has not been explored in the auditory domain; therefore, we evaluated the contributions of high-level global and low-level acoustic information to auditory scene perception. An additional aim was to increase the field's ecological validity by using and making available a new collection of high-quality auditory scenes. Participants rated scenes on 8 global properties (e.g., open vs. enclosed) and an acoustic analysis evaluated which low-level features predicted the ratings. We submitted the acoustic measures and average ratings of the global properties to separate exploratory factor analyses (EFAs). The EFA of the acoustic measures revealed a seven-factor structure explaining 57% of the variance in the data, while the EFA of the global property measures revealed a two-factor structure explaining 64% of the variance in the data. Regression analyses revealed each global property was predicted by at least one acoustic variable (R2 = 0.33-0.87). These findings were extended using deep neural network models where we examined correlations between human ratings of global properties and deep embeddings of two computational models: an object-based model and a scene-based model. The results support that participants' ratings are more strongly explained by a global analysis of the scene setting, though the relationship between scene perception and auditory perception is multifaceted, with differing correlation patterns evident between the two models. Taken together, our results provide evidence for the ability to perceive auditory scenes from a global perspective. Some of the acoustic measures predicted ratings of global scene perception, suggesting representations of auditory objects may be transformed through many stages of processing in the ventral auditory stream, similar to what has been proposed in the ventral visual stream. These findings and the open availability of our scene collection will make future studies on perception, attention, and memory for natural auditory scenes possible.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organization of memories remains unclear. Here, we designed a novel task-the Emotion Boundary Task-wherein participants viewed sequences of negative and neutral images surrounded by a color border. We manipulated perceptual context (border color), emotional valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create encoding events comprised of image sequences with a shared perceptual and/or emotional context. We measured memory for temporal order and subjectively remembered temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distance was remembered as longer for images spanning neutral-to-negative shifts-suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of this negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently presented negative (compared to neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral to negative events produce temporal dilation in memory, which may be relevant for adaptive emotional functioning.

White matter network underlying semantic processing: evidence from stroke patients

The hub-and-spoke theory of semantic representation fractionates the neural underpinning of semantic knowledge into two essential components: the sensorimotor modality-specific regions and a crucially important semantic hub region. Our previous study in patients with semantic dementia has found that the hub region is located in the left fusiform gyrus. However, because this region is located within the brain damage in patients with semantic dementia, it is not clear whether the semantic deficit is caused by structural damage to the hub region itself or by its disconnection from other brain regions. Stroke patients do not have any damage to the left fusiform gyrus, but exhibit amodal and modality-specific deficits in semantic processing. Therefore, in this study, we validated the semantic hub region from a brain network perspective in 79 stroke patients and explored the white matter connections associated with it. First, we collected data of diffusion-weighted imaging and behavioural performance on general semantic tasks and modality-specific semantic tasks (assessing object knowledge on form, colour, motion, sound, manipulation and function). We then used correlation and regression analyses to examine the association between the nodal degree values of brain regions in the whole-brain structural network and general semantic performance in the stroke patients. The results revealed that the connectivity of the left fusiform gyrus significantly predicted general semantic performance, indicating that this region is the semantic hub. To identify the semantic-relevant connections of the semantic hub, we then correlated the white matter integrity values of each tract connected to the left fusiform gyrus separately with performance on general and modality-specific semantic processing. We found that the hub region accomplished general semantic processing through white matter connections with the left superior temporal pole, middle temporal gyrus, inferior temporal gyrus and hippocampus. The connectivity between the hub region and the left hippocampus, superior temporal pole, middle temporal gyrus, inferior temporal gyrus and parahippocampal gyrus was differentially involved in object form, colour, motion, sound, manipulation and function processing. After statistically removing the effects of potential confounding variables (i.e. whole-brain lesion volume, lesion volume of regions of interest and performance on non-semantic control tasks), the observed effects remained significant. Together, our findings support the role of the left fusiform gyrus as a semantic hub region in stroke patients and reveal its crucial connectivity in the network. This study provides new insights and evidence for the neuroanatomical organization of semantic memory in the human brain.

Using deep neural networks to disentangle visual and semantic information in human perception and memory

Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual-semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual-semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

Semantic representations in inferior frontal and lateral temporal cortex during picture naming, reading, and repetition

Reading, naming, and repetition are classical neuropsychological tasks widely used in the clinic and psycholinguistic research. While reading and repetition can be accomplished by following a direct or an indirect route, pictures can be named only by means of semantic mediation. By means of fMRI multivariate pattern analysis, we evaluated whether this well-established fundamental difference at the cognitive level is associated at the brain level with a difference in the degree to which semantic representations are activated during these tasks. Semantic similarity between words was estimated based on a word association model. Twenty subjects participated in an event-related fMRI study where the three tasks were presented in pseudo-random order. Linear discriminant analysis of fMRI patterns identified a set of regions that allow to discriminate between words at a high level of word-specificity across tasks. Representational similarity analysis was used to determine whether semantic similarity was represented in these regions and whether this depended on the task performed. The similarity between neural patterns of the left Brodmann area 45 (BA45) and of the superior portion of the left supramarginal gyrus correlated with the similarity in meaning between entities during picture naming. In both regions, no significant effects were seen for repetition or reading. The semantic similarity effect during picture naming was significantly larger than the similarity effect during the two other tasks. In contrast, several regions including left anterior superior temporal gyrus and left ventral BA44/frontal operculum, among others, coded for semantic similarity in a task-independent manner. These findings provide new evidence for the dynamic, task-dependent nature of semantic representations in the left BA45 and a more task-independent nature of the representational activation in the lateral temporal cortex and ventral BA44/frontal operculum.

Parallel cognitive maps for multiple knowledge structures in the hippocampal formation

The hippocampal-entorhinal system uses cognitive maps to represent spatial knowledge and other types of relational information. However, objects can often be characterized by different types of relations simultaneously. How does the hippocampal formation handle the embedding of stimuli in multiple relational structures that differ vastly in their mode and timescale of acquisition? Does the hippocampal formation integrate different stimulus dimensions into one conjunctive map or is each dimension represented in a parallel map? Here, we reanalyzed human functional magnetic resonance imaging data from Garvert et al. (2017) that had previously revealed a map in the hippocampal formation coding for a newly learnt transition structure. Using functional magnetic resonance imaging adaptation analysis, we found that the degree of representational similarity in the bilateral hippocampus also decreased as a function of the semantic distance between presented objects. Importantly, while both map-like structures localized to the hippocampal formation, the semantic map was located in more posterior regions of the hippocampal formation than the transition structure and thus anatomically distinct. This finding supports the idea that the hippocampal-entorhinal system forms parallel cognitive maps that reflect the embedding of objects in diverse relational structures.

Representation of event and object concepts in ventral anterior temporal lobe and angular gyrus

Semantic knowledge includes understanding of objects and their features and also understanding of the characteristics of events. The hub-and-spoke theory holds that these conceptual representations rely on multiple information sources that are integrated in a central hub in the ventral anterior temporal lobes. The dual-hub theory expands this framework with the claim that the ventral anterior temporal lobe hub is specialized for object representation, while a second hub in angular gyrus is specialized for event representation. To test these ideas, we used representational similarity analysis, univariate and psychophysiological interaction analyses of fMRI data collected while participants processed object and event concepts (e.g. "an apple," "a wedding") presented as images and written words. Representational similarity analysis showed that angular gyrus encoded event concept similarity more than object similarity, although the left angular gyrus also encoded object similarity. Bilateral ventral anterior temporal lobes encoded both object and event concept structure, and left ventral anterior temporal lobe exhibited stronger coding for events. Psychophysiological interaction analysis revealed greater connectivity between left ventral anterior temporal lobe and right pMTG, and between right angular gyrus and bilateral ITG and middle occipital gyrus, for event concepts compared to object concepts. These findings support the specialization of angular gyrus for event semantics, though with some involvement in object coding, but do not support ventral anterior temporal lobe specialization for object concepts.

Hippocampal Functions Modulate Transfer-Appropriate Cortical Representations Supporting Subsequent Memory

The hippocampus plays a central role as a coordinate system or index of information stored in neocortical loci. Nonetheless, it remains unclear how hippocampal processes integrate with cortical information to facilitate successful memory encoding. Thus, the goal of the current study was to identify specific hippocampal-cortical interactions that support object encoding. We collected fMRI data while 19 human participants (7 female and 12 male) encoded images of real-world objects and tested their memory for object concepts and image exemplars (i.e., conceptual and perceptual memory). Representational similarity analysis revealed robust representations of visual and semantic information in canonical visual (e.g., occipital cortex) and semantic (e.g., angular gyrus) regions in the cortex, but not in the hippocampus. Critically, hippocampal functions modulated the mnemonic impact of cortical representations that are most pertinent to future memory demands, or transfer-appropriate representations Subsequent perceptual memory was best predicted by the strength of visual representations in ventromedial occipital cortex in coordination with hippocampal activity and pattern information during encoding. In parallel, subsequent conceptual memory was best predicted by the strength of semantic representations in left inferior frontal gyrus and angular gyrus in coordination with either hippocampal activity or semantic representational strength during encoding. We found no evidence for transfer-incongruent hippocampal-cortical interactions supporting subsequent memory (i.e., no hippocampal interactions with cortical visual/semantic representations supported conceptual/perceptual memory). Collectively, these results suggest that diverse hippocampal functions flexibly modulate cortical representations of object properties to satisfy distinct future memory demands.Significance Statement The hippocampus is theorized to index pieces of information stored throughout the cortex to support episodic memory. Yet how hippocampal processes integrate with cortical representation of stimulus information remains unclear. Using fMRI, we examined various forms of hippocampal-cortical interactions during object encoding in relation to subsequent performance on conceptual and perceptual memory tests. Our results revealed novel hippocampal-cortical interactions that utilize semantic and visual representations in transfer-appropriate manners: conceptual memory supported by hippocampal modulation of frontoparietal semantic representations, and perceptual memory supported by hippocampal modulation of occipital visual representations. These findings provide important insights into the neural mechanisms underlying the formation of information-rich episodic memory and underscore the value of studying the flexible interplay between brain regions for complex cognition.

Recurrent connectivity supports higher-level visual and semantic object representations in the brain

Visual object recognition has been traditionally conceptualised as a predominantly feedforward process through the ventral visual pathway. While feedforward artificial neural networks (ANNs) can achieve human-level classification on some image-labelling tasks, it's unclear whether computational models of vision alone can accurately capture the evolving spatiotemporal neural dynamics. Here, we probe these dynamics using a combination of representational similarity and connectivity analyses of fMRI and MEG data recorded during the recognition of familiar, unambiguous objects. Modelling the visual and semantic properties of our stimuli using an artificial neural network as well as a semantic feature model, we find that unique aspects of the neural architecture and connectivity dynamics relate to visual and semantic object properties. Critically, we show that recurrent processing between the anterior and posterior ventral temporal cortex relates to higher-level visual properties prior to semantic object properties, in addition to semantic-related feedback from the frontal lobe to the ventral temporal lobe between 250 and 500 ms after stimulus onset. These results demonstrate the distinct contributions made by semantic object properties in explaining neural activity and connectivity, highlighting it as a core part of object recognition not fully accounted for by current biologically inspired neural networks.

Effects of Aging on Successful Object Encoding: Enhanced Semantic Representations Compensate for Impaired Visual Representations

Although episodic memory and visual processing decline substantially with healthy aging, semantic knowledge is generally spared. There is evidence that older adults' spared semantic knowledge can support episodic memory. Here, we used functional magnetic resonance imaging (fMRI) combined with representational similarity analyses (RSAs) to examine how novel visual and preexisting semantic representations at encoding predict subjective memory vividness at retrieval. Eighteen young and seventeen older adults (female and male participants) encoded images of objects during fMRI scanning and recalled these images while rating the vividness of their memories. After scanning, participants discriminated between studied images and similar lures. RSA based on a deep convolutional neural network and normative concept feature data were used to link patterns of neural activity during encoding to visual and semantic representations. Relative to young adults, the specificity of activation patterns for visual features was reduced in older adults, consistent with dedifferentiation. However, the specificity of activation patterns for semantic features was enhanced in older adults, consistent with hyperdifferentiation. Despite dedifferentiation, visual representations in early visual cortex (EVC) predicted high memory vividness in both age groups. In contrast, semantic representations in lingual gyrus (LG) and fusiform gyrus (FG) were associated with high memory vividness only in the older adults. Intriguingly, data suggests that older adults with lower specificity of visual representations in combination with higher specificity of semantic representations tended to rate their memories as more vivid. Our findings suggest that memory vividness in aging relies more on semantic representations over anterior regions, potentially compensating for age-related dedifferentiation of visual information in posterior regions.SIGNIFICANCE STATEMENT Normal aging is associated with impaired memory for events while semantic knowledge might even improve. We investigated the effects of aging on the specificity of visual and semantic information in the brain when viewing common objects and how this information enables subsequent memory vividness for these objects. Using functional magnetic resonance imaging (fMRI) combined with modeling of the stimuli we found that visual information was represented with less specificity in older than young adults while still supporting memory vividness. In contrast semantic information supported memory vividness only in older adults and especially in those individuals that had the lowest specificity of visual information. These findings provide evidence for a spared semantic memory system increasingly recruited to compensate for degraded visual representations in older age.

Complex visual discrimination is impaired after right, but not left, anterior temporal lobectomy

The prevailing view in human cognitive neuroscience associates the medial temporal lobes (MTLs) with declarative memory. Compelling experimental evidence has, however, demonstrated that these regions are specialized according to the representations processed, irrespective of the cognitive domain assessed. This account was supported by the study of patients with bilateral medial temporal amnesia, who exhibit impairments in perceptual tasks involving complex visual stimuli. Yet, little is known regarding the impact of unilateral MTL damage on complex visual abilities. To address this issue, we administered a visual matching task to 20 patients who underwent left (N = 12) or right (N = 8) anterior temporal lobectomy for drug-resistant epilepsy and to 38 healthy controls. Presentation viewpoint was manipulated to increase feature ambiguity, as this is critical to reveal impairments in perceptual tasks. Similar to control participants, patients with left-sided damage succeeded in all task conditions. In contrast, patients with right-sided damage had decreased accuracy compared with that of the other two groups, as well as increased response time. Notably, the accuracy of those with right-sided damage did not exceed chance level when feature ambiguity was high (i.e., when stimuli were presented from different viewpoints) for the most complex classes of stimuli (i.e., scenes and buildings, compared with single objects). The pattern reported in bilateral patients in previous studies was therefore reproduced in patients with right, but not left, resection. These results suggest that the complex visual-representation functions supported by the MTL are right-lateralized, and raise the question as to how the representational account of these regions applies to representations supported by left MTL regions.

Perirhinal cortex is associated with fine-grained discrimination of conceptually confusable objects in Alzheimer's disease

The perirhinal cortex (PrC) stands among the first brain areas to deteriorate in Alzheimer's disease (AD). This study tests to what extent the PrC is involved in representing and discriminating confusable objects based on the conjunction of their perceptual and conceptual features. To this aim, AD patients and control counterparts performed 3 tasks: a naming, a recognition memory, and a conceptual matching task, where we manipulated conceptual and perceptual confusability. A structural MRI of the antero-lateral parahippocampal subregions was obtained for each participant. We found that the sensitivity to conceptual confusability was associated with the left PrC volume in both AD patients and control participants for the recognition memory task, while it was specifically associated with the volume of the left PrC in AD patients for the conceptual matching task. This suggests that a decreased volume of the PrC is related to the ability to disambiguate conceptually confusable items. Therefore, testing recognition memory or conceptual matching of easily conceptually confusable items can provide a potential cognitive marker of PrC atrophy.

Multisensory perception constrains the formation of object categories: a review of evidence from sensory-driven and predictive processes on categorical decisions

Although object categorization is a fundamental cognitive ability, it is also a complex process going beyond the perception and organization of sensory stimulation. Here we review existing evidence about how the human brain acquires and organizes multisensory inputs into object representations that may lead to conceptual knowledge in memory. We first focus on evidence for two processes on object perception, multisensory integration of redundant information (e.g. seeing and feeling a shape) and crossmodal, statistical learning of complementary information (e.g. the 'moo' sound of a cow and its visual shape). For both processes, the importance attributed to each sensory input in constructing a multisensory representation of an object depends on the working range of the specific sensory modality, the relative reliability or distinctiveness of the encoded information and top-down predictions. Moreover, apart from sensory-driven influences on perception, the acquisition of featural information across modalities can affect semantic memory and, in turn, influence category decisions. In sum, we argue that both multisensory processes independently constrain the formation of object categories across the lifespan, possibly through early and late integration mechanisms, respectively, to allow us to efficiently achieve the everyday, but remarkable, ability of recognizing objects. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Perirhinal cortex automatically tracks multiple types of familiarity regardless of task-relevance

Perirhinal cortex (PrC) has long been implicated in familiarity assessment for objects and corresponding concepts. However, extant studies have focused mainly on changes in familiarity induced by recent exposure in laboratory settings. There is an increasing appreciation of other types of familiarity signals, in particular graded familiarity accumulated throughout one's lifetime. In prior work (Duke et al., 2017, Cortex, 89, 61-70), PrC has been shown to track lifetime familiarity ratings when participants make related judgements. A theoretically important characteristic of familiarity is its proposed automaticity. Support for automaticity comes from a documented impact of recent stimulus exposure on behavioral performance, and on PrC signals, under conditions in which this exposure is not task relevant. In the current fMRI study, we tested whether PrC also tracks lifetime familiarity of object concepts automatically, and whether this type of familiarity influences behavior even when it is not task relevant. During scanning, neurotypical participants (N = 30, age range 18-40, 7 males) provided animacy judgements about concrete object concepts presented at differing frequencies in an initial study phase. In a subsequent test phase, they made graded judgements of recent or lifetime familiarity. Behavioral performance showed sensitivity to lifetime familiarity even when it was not relevant for the task at hand. Across five sets of fMRI analyses, we found that PrC consistently tracked recent and lifetime familiarity of object concepts regardless of the task at hand. Critically, while several other temporal-lobe regions also showed isolated familiarity effects, none of them tracked familiarity with the same consistency. These findings demonstrate that PrC automatically tracks multiple types of familiarity. They support models that assign a broad role in the representation of information about object concepts to this structure.

Episodic Memory formation: A review of complex Hippocampus input pathways

Memories of everyday experiences involve the encoding of a rich and dynamic representation of present objects and their contextual features. Traditionally, the resulting mnemonic trace is referred to as Episodic Memory, i.e. the "what", "where" and "when" of a lived episode. The journey for such memory trace encoding begins with the perceptual data of an experienced episode handled in sensory brain regions. The information is then streamed to cortical areas located in the ventral Medio Temporal Lobe, which produces multi-modal representations concerning either the objects (in the Perirhinal cortex) or the spatial and contextual features (in the parahippocampal region) of the episode. Then, this high-level data is gated through the Entorhinal Cortex and forwarded to the Hippocampal Formation, where all the pieces get bound together. Eventually, the resulting encoded neural pattern is relayed back to the Neocortex for a stable consolidation. This review will detail these different stages and provide a systematic overview of the major cortical streams toward the Hippocampus relevant for Episodic Memory encoding.

Semantic cognition in healthy ageing: Neural signatures of representation and control mechanisms in naming typical and atypical objects

Effective use of conceptual knowledge engages semantic representation and control processes to access information in a goal-driven manner. Neuropsychological findings of patients presenting either degraded knowledge (e.g., semantic dementia) or disrupted control (e.g., semantic aphasia) converge with neuroimaging evidence from young adults, and delineate the neural segregation of representation and control mechanisms. However, there is still scarce research on the neurofunctional underpinnings of such mechanisms in healthy ageing. To address this, we conducted an fMRI study, wherein young and older adults performed a covert naming task of typical and atypical objects. Three main age-related differences were found. As shown by age group and typicality interactions, older adults exhibited overactivation during naming of atypical (e.g., avocado) relative to typical concepts in brain regions associated to semantic representation, including anterior and medial portions of left temporal lobe (respectively, ATL and MTG). This provides evidence for the reorganization of neural activity in these brain regions contingent to the enrichment of semantic repositories in older ages. The medial orbitofrontal gyrus was also overactivated, indicating that the processing of atypical concepts (relative to typical items) taxes additional control resources in the elderly. Increased activation in the inferior frontal gyrus (IFG) was observed in naming typical items (relative to atypical ones), but only for young adults. This suggests that naming typical items (e.g., strawberry) taxes more on control processes in younger ages, presumably due to the semantic competition set by other items that share multiple features with the target (e.g., raspberry, blackberry, cherry). Together, these results reveal the dynamic nature of semantic control interplaying with conceptual representations as people grow older, by indicating that distinct neural bases uphold semantic performance from young to older ages. These findings may be explained by neural compensation mechanisms coming into play to support neurocognitive changes in healthy ageing.

Typicality in the brain during semantic and episodic memory decisions

Concept typicality is a key semantic dimension supporting the categorical organization of items based on their features, such that typical items share more features with other members of their category than atypical items, which are more distinctive. Typicality effects manifest in better accuracy and faster response times during categorization tasks, but higher performance for atypical items in episodic memory tasks, due to their distinctiveness. At a neural level, typicality has been linked to the anterior temporal lobe (ATL) and the inferior frontal gyrus (IFG) in semantic decision tasks, but patterns of brain activity during episodic memory tasks remain to be understood. We investigated the neural correlates of typicality in semantic and episodic memory to determine the brain regions associated with semantic typicality and uncover effects arising when items are reinstated during retrieval. In an fMRI study, 26 healthy young subjects first performed a category verification task on words representing typical and atypical concepts (encoding), and then completed a recognition memory task (retrieval). In line with previous literature, we observed higher accuracy and faster response times for typical items in the category verification task, while atypical items were better recognized in the episodic memory task. During category verification, univariate analyses revealed a greater involvement of the angular gyrus for typical items and the inferior frontal gyrus for atypical items. During the correct recognition of old items, regions belonging to the core recollection network were activated. We then compared the similarity of the representations from encoding to retrieval (ERS) using Representation Similarity Analyses. Results showed that typical items were reinstated more than atypical ones in several regions including the left precuneus and left anterior temporal lobe (ATL). This suggests that the correct retrieval of typical items requires finer-grained processing, evidenced by greater item-specific reinstatement, which is needed to resolve their confusability with other members of the category due to their higher feature similarity. Our findings confirm the centrality of the ATL in the processing of typicality while extending it to memory retrieval.

Decoding semantic representations in mind and brain

A key goal for cognitive neuroscience is to understand the neurocognitive systems that support semantic memory. Recent multivariate analyses of neuroimaging data have contributed greatly to this effort, but the rapid development of these novel approaches has made it difficult to track the diversity of findings and to understand how and why they sometimes lead to contradictory conclusions. We address this challenge by reviewing cognitive theories of semantic representation and their neural instantiation. We then consider contemporary approaches to neural decoding and assess which types of representation each can possibly detect. The analysis suggests why the results are heterogeneous and identifies crucial links between cognitive theory, data collection, and analysis that can help to better connect neuroimaging to mechanistic theories of semantic cognition.

Proper and common names in the semantic system

Proper names are an important part of language and communication. They are thought to have a special status due to their neuropsychological and psycholinguistic profile. To what extent proper names rely on the same semantic system as common names is not clear. In an fMRI study, we presented the same group of participants with both proper and common names to compare the associated activations. Both person and place names, as well as personally familiar and famous names were used, and compared with words representing concrete and abstract concepts. A whole-brain analysis was followed by a detailed analysis of subdivisions of four regions of interest known to play a central role in the semantic system: angular gyrus, anterior temporal lobe, posterior cingulate complex, and medial temporal lobe. We found that most subdivisions within these regions bilaterally were activated by both proper names and common names. The bilateral perirhinal and right entorhinal cortex showed a response specific to proper names, suggesting an item-specific role in retrieving person and place related information. While activation to person and place names overlapped greatly, place names were differentiated by activating areas associated with spatial memory and navigation. Person names showed greater right hemisphere involvement compared to places, suggesting a wider range of associations. Personally familiar names showed stronger activation bilaterally compared to famous names, indicating representations that are enhanced by autobiographic and episodic details. Both proper and common names are processed in the wider semantic system that contains associative, episodic, and spatial components. Processing of proper names is characterized by a somewhat stronger involvement these components, rather than by a fundamentally different system.

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Individual differences in behavioral and electrophysiological signatures of familiarity- and recollection-based recognition memory

Our everyday memories can vary in terms of accuracy and phenomenology. According to one theoretical account, these differences hinge on whether the memories contain information about both an item itself as well as associated details (remember) versus those that are devoid of these associated contextual details (familiar). This distinction has been supported by computational modeling of behavior, studies in patients, and neuroimaging work including differences both in electrophysiological and functional magnetic resonance imaging. At present, however, little evidence has emerged to suggest that neurophysiological measures track individual differences in estimates of recollection and familiarity. Here, we conducted electrophysiological recordings of brain activity during a recognition memory task designed to differentiate between behavioral indices of recollection and familiarity. Non-parametric cluster-based permutation analyses revealed associations between electrophysiological signatures of familiarity and recollection with their respective behavioral estimates. These results support the idea that recollection and familiarity are distinct phenomena and is the first, to our knowledge, to identify distinct electrophysiological signatures that track individual differences in these processes.

Damage to the human dentate gyrus impairs the perceptual discrimination of complex, novel objects

The hippocampus (HPC), and the dentate gyrus (DG) subregion in particular, is purported to be a pattern separator, orthogonally representing similar information so that distinct memories may be formed. The HPC may also be involved in complex perceptual discrimination. It is unclear if this role is limited to spatial/scene stimuli or extends to the discrimination of objects. Also unclear is whether the DG itself contributes to pattern separation beyond memory. BL, an individual with bilateral DG lesions, was previously shown to have poor discrimination of similar, everyday objects in memory. Here, we demonstrate that BL's deficit extends to complex perceptual discrimination of novel objects. Specifically, BL was presented with closely matched possible and impossible objects, which give rise to fundamentally different 3D perceptual representations despite being visually similar. BL performed significantly worse than controls when asked to select an odd object (e.g., impossible) amongst three identical counterpart objects (e.g., possible) presented at different rotations. His deficit was also evident in an atypical eye fixation pattern during this task. In contrast, BL's performance was indistinguishable from that of controls on other tasks involving the same objects, indicating that he could visually differentiate the object pairs, that he perceived the objects holistically in 3D, and that he has only a mild weakness in categorizing object possibility. Furthermore, his performance on standardized neuropsychological measures indicated intact mental rotation, visual-spatial attention, and working memory (visual and auditory). Collectively, these results provide evidence that the DG is necessary for complex perceptual discrimination of novel objects, indicating that the DG might function as a generic pattern separator of a wide range of stimuli within high-level perception, and that its role is not limited to memory.

Semantic Expectation Effects on Object Detection: Using Figure Assignment to Elucidate Mechanisms

Recent evidence suggesting that object detection is improved following valid rather than invalid labels implies that semantics influence object detection. It is not clear, however, whether the results index object detection or feature detection. Further, because control conditions were absent and labels and objects were repeated multiple times, the mechanisms are unknown. We assessed object detection via figure assignment, whereby objects are segmented from backgrounds. Masked bipartite displays depicting a portion of a mono-oriented object (a familiar configuration) on one side of a central border were shown once only for 90 or 100 ms. Familiar configuration is a figural prior. Accurate detection was indexed by reports of an object on the familiar configuration side of the border. Compared to control experiments without labels, valid labels improved accuracy and reduced response times (RTs) more for upright than inverted objects (Studies 1 and 2). Invalid labels denoting different superordinate-level objects (DSC; Study 1) or same superordinate-level objects (SSC; Study 2) reduced accuracy for upright displays only. Orientation dependency indicates that effects are mediated by activated object representations rather than features which are invariant over orientation. Following invalid SSC labels (Study 2), accurate detection RTs were longer than control for both orientations, implicating conflict between semantic representations that had to be resolved before object detection. These results demonstrate that object detection is not just affected by semantics, it entails semantics.

Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision

Vision neuroscience has made great strides in understanding the hierarchical organization of object representations along the ventral visual stream (VVS). How VVS representations capture fine-grained visual similarities between objects that observers subjectively perceive has received limited examination so far. In the current study, we addressed this question by focussing on perceived visual similarities among subordinate exemplars of real-world categories. We hypothesized that these perceived similarities are reflected with highest fidelity in neural activity patterns downstream from inferotemporal regions, namely in perirhinal (PrC) and anterolateral entorhinal cortex (alErC) in the medial temporal lobe. To address this issue with functional magnetic resonance imaging (fMRI), we administered a modified 1-back task that required discrimination between category exemplars as well as categorization. Further, we obtained observer-specific ratings of perceived visual similarities, which predicted behavioural discrimination performance during scanning. As anticipated, we found that activity patterns in PrC and alErC predicted the structure of perceived visual similarity relationships among category exemplars, including its observer-specific component, with higher precision than any other VVS region. Our findings provide new evidence that subjective aspects of object perception that rely on fine-grained visual differentiation are reflected with highest fidelity in the medial temporal lobe.

Moving in Semantic Space in Prodromal and Very Early Alzheimer's Disease: An Item-Level Characterization of the Semantic Fluency Task

The semantic fluency task is a widely used clinical tool in the diagnostic process of Alzheimer's disease. The task requires efficient mapping of the semantic space to produce as many items as possible within a semantic category. We examined whether healthy volunteers (n = 42) and patients with early Alzheimer's disease (24 diagnosed with amnestic Mild Cognitive Impairment and 18 with early Alzheimer's dementia) take advantage of and travel in the semantic space differently. With focus on the animal fluency task, we sought to emulate the detailed structure of the multidimensional semantic space by utilizing word2vec-method from the natural language processing domain. To render the resulting multidimensional semantic space visually comprehensible, we applied a dimensionality reduction algorithm (t-SNE), which enabled a straightforward division of the semantic space into sub-categories. Moving in semantic space was quantified with the number of items created, sub-categories visited, and switches and returns to these sub-categories. Multinomial logistic regression models were used to predict the diagnostic group with these independent variables. We found that returning to a sub-category provided additional information, besides the number of words produced in the task, to differentiate patients with Alzheimer's dementia from both amnestic Mild Cognitive Impairment patients and healthy controls. The results suggest that the frequency of returning to a sub-category may serve as an additional aid for clinicians in diagnosing early Alzheimer's disease. Moreover, our results imply that the combination of word2vec and subsequent t-SNE-visualization may offer a valuable tool for examining the semantic space and its sub-categories.

Between automatic and control processes: How relationships between problem elements interact to facilitate or impede insight

Solving a problem requires relating the pieces of information available to each other and to the solution. We investigated how the strength of these relationships determines the likelihood of solving insight tasks based on remote associates. In these tasks, the solver is provided with several cues (e.g., drop, coat, summer) and has to find the solution that matches those cues (e.g., rain). We measured the semantic similarity between the cues and the solution (cue-solution similarity) as well as between cues (cue-cue similarity). We assume those relationships modulate two basic processes underlying insight problem-solving. First, there is an automatic activation process whereby conceptual activation spreads across a semantic network from each cue node to their associated nodes, potentially reaching the node of the solution. Thus, in general, the higher cue-solution similarity, the more likely the solution will be found (Prediction 1). Second, there is a controlled search process focused on an area in semantic space whose radius depends on competing cue-cue similarity. High cue-cue similarity will bias a search for the solution close to the provided cues because the associated nodes shared by both cues are highly coactivated. Therefore, high cue-cue similarity will have a beneficial effect when the cue-solution similarity is high but a detrimental effect when cue-solution similarity is low (Prediction 2). Our two predictions were confirmed using both verbal and pictorial remote association tasks, supporting the view that insight is dependent on an interaction of meaningful relationships between cues and solutions, and clarify the mechanisms of insight problem solving in remote associates.

The characterization of actions at the superordinate, basic and subordinate level

Objects can be categorized at different levels of abstraction, ranging from the superordinate (e.g., fruit) and the basic (e.g., apple) to the subordinate level (e.g., golden delicious). The basic level is assumed to play a key role in categorization, e.g., in terms of the number of features used to describe these actions and the speed of processing. To which degree do these principles also apply to the categorization of observed actions? To address this question, we first selected a range of actions at the superordinate (e.g., locomotion), basic (e.g., to swim) and subordinate level (e.g., to swim breaststroke), using verbal material (Experiments 1-3). Experiments 4-6 aimed to determine the characteristics of these actions across the three taxonomic levels. Using a feature listing paradigm (Experiment 4), we determined the number of features that were provided by at least six out of twenty participants (common features), separately for the three different levels. In addition, we examined the number of shared (i.e., provided for more than one category) and distinct (i.e., provided for one category only) features. Participants produced the highest number of common features for actions at the basic level. Actions at the subordinate level shared more features with other actions at the same level than those at the superordinate level. Actions at the superordinate and basic level were described with more distinct features compared to those provided at the subordinate level. Using an auditory priming paradigm (Experiment 5), we observed that participants responded faster to action images preceded by a matching auditory cue corresponding to the basic and subordinate level, but not for superordinate level cues, suggesting that the basic level is the most abstract level at which verbal cues facilitate the processing of an upcoming action. Using a category verification task (Experiment 6), we found that participants were faster and more accurate to verify action categories (depicted as images) at the basic and subordinate level in comparison to the superordinate level. Together, in line with the object categorization literature, our results suggest that information about action categories is maximized at the basic level.

Distributed learning episodes create a context fear memory outside the hippocampus that depends on perirhinal and anterior cingulate cortices

Damage to the hippocampus (HPC) typically causes retrograde amnesia for contextual fear conditioning. Repeating the conditioning over several sessions, however, can eliminate the retrograde amnesic effects. This form of reinstatement thus permits modifications to networks that can support context memory retrieval in the absence of the HPC. The present study aims to identify cortical regions that support the nonHPC context memory. Specifically, the contribution of the perirhinal cortex (PRH) and the anterior cingulate cortex (ACC) were examined because of their established importance to context memory. The findings show that context memories established through distributed reinstatement survive damage limited only to the HPC, PRH, or ACC. Combined lesions of the HPC and PRH, as well as the HPC and ACC, caused retrograde amnesia, suggesting that network modifications in the PRH and ACC enable context fear memories to become resistant to HPC damage.

Two 'what' pathways for action and object recognition

The ventral visual stream is conceived as a pathway for object recognition. However, we also recognize the actions an object can be involved in. Here, we show that action recognition critically depends on a pathway in lateral occipitotemporal cortex, partially overlapping and topographically aligned with object representations that are precursors for action recognition. By contrast, object features that are more relevant for object recognition, such as color and texture, are typically found in ventral occipitotemporal cortex. We argue that occipitotemporal cortex contains similarly organized lateral and ventral 'what' pathways for action and object recognition, respectively. This account explains a number of observed phenomena, such as the duplication of object domains and the specific representational profiles in lateral and ventral cortex.

Perceptual and Semantic Representations at Encoding Contribute to True and False Recognition of Objects

When encoding new episodic memories, visual and semantic processing is proposed to make distinct contributions to accurate memory and memory distortions. Here, we used fMRI and preregistered representational similarity analysis to uncover the representations that predict true and false recognition of unfamiliar objects. Two semantic models captured coarse-grained taxonomic categories and specific object features, respectively, while two perceptual models embodied low-level visual properties. Twenty-eight female and male participants encoded images of objects during fMRI scanning, and later had to discriminate studied objects from similar lures and novel objects in a recognition memory test. Both perceptual and semantic models predicted true memory. When studied objects were later identified correctly, neural patterns corresponded to low-level visual representations of these object images in the early visual cortex, lingual, and fusiform gyri. In a similar fashion, alignment of neural patterns with fine-grained semantic feature representations in the fusiform gyrus also predicted true recognition. However, emphasis on coarser taxonomic representations predicted forgetting more anteriorly in the anterior ventral temporal cortex, left inferior frontal gyrus and, in an exploratory analysis, left perirhinal cortex. In contrast, false recognition of similar lure objects was associated with weaker visual analysis posteriorly in early visual and left occipitotemporal cortex. The results implicate multiple perceptual and semantic representations in successful memory encoding and suggest that fine-grained semantic as well as visual analysis contributes to accurate later recognition, while processing visual image detail is critical for avoiding false recognition errors.SIGNIFICANCE STATEMENT People are able to store detailed memories of many similar objects. We offer new insights into the encoding of these specific memories by combining fMRI with explicit models of how image properties and object knowledge are represented in the brain. When people processed fine-grained visual properties in occipital and posterior temporal cortex, they were more likely to recognize the objects later and less likely to falsely recognize similar objects. In contrast, while object-specific feature representations in fusiform gyrus predicted accurate memory, coarse-grained categorical representations in frontal and temporal regions predicted forgetting. The data provide the first direct tests of theoretical assumptions about encoding true and false memories, suggesting that semantic representations contribute to specific memories as well as errors.

Autobiographical Memory Fluency Reductions in Cognitively Unimpaired Middle-Aged and Older Adults at Increased Risk for Alzheimer's Disease Dementia

OBJECTIVE

Recent research has revealed that cognitively unimpaired older adults who are at higher risk for developing Alzheimer's disease (AD) dementia often exhibit subtle cognitive alterations in their neuropsychological profiles. Emerging evidence suggests that autobiographical memory, which is memory for personal events and knowledge, may be sensitive to early AD-related cognitive alterations. In the present study, we investigated whether the rapid generation of autobiographical memory category exemplars, a retrieval process that taxes the neural network that is vulnerable to early AD, is compromised in cognitively unimpaired middle-aged and older carriers of the e4 allele of the apolipoprotein E gene (APOE4), which increases risk for AD dementia.

METHODS

In addition to standard neuropsychological tests, we administered a fluency task that requires generating exemplars for two types of autobiographical memory, namely episodic memories and personal semantics, to a group of cognitively unimpaired middle-aged and older adults (n = 45) enriched with APOE4 carriers (n = 20).

RESULTS

While no APOE4 deficits were found on standard neuropsychological tests, episodic and personal semantic exemplar generation was reduced in the APOE4 group.

DISCUSSION

Autobiographical memory aberrations associated with a higher risk for AD are evident in fluency and affect both episodic memory and personal semantics.

A category-selective semantic memory deficit for animate objects in semantic variant primary progressive aphasia

Data are mixed on whether patients with semantic variant primary progressive aphasia exhibit a category-selective semantic deficit for animate objects. Moreover, there is little consensus regarding the neural substrates of this category-selective semantic deficit, though prior literature has suggested that the perirhinal cortex and the lateral posterior fusiform gyrus may support semantic memory functions important for processing animate objects. In this study, we investigated whether patients with semantic variant primary progressive aphasia exhibited a category-selective semantic deficit for animate objects in a word-picture matching task, controlling for psycholinguistic features of the stimuli, including frequency, familiarity, typicality and age of acquisition. We investigated the neural bases of this category selectivity by examining its relationship with cortical atrophy in two primary regions of interest: bilateral perirhinal cortex and lateral posterior fusiform gyri. We analysed data from 20 patients with semantic variant primary progressive aphasia (mean age = 64 years, S.D. = 6.94). For each participant, we calculated an animacy index score to denote the magnitude of the category-selective semantic deficit for animate objects. Multivariate regression analysis revealed a main effect of animacy (β = 0.52, t = 4.03, P < 0.001) even after including all psycholinguistic variables in the model, such that animate objects were less likely to be identified correctly relative to inanimate objects. Inspection of each individual patient's data indicated the presence of a disproportionate impairment in animate objects in most patients. A linear regression analysis revealed a relationship between the right perirhinal cortex thickness and animacy index scores (β = -0.57, t = -2.74, P = 0.015) such that patients who were more disproportionally impaired for animate relative to inanimate objects exhibited thinner right perirhinal cortex. A vertex-wise general linear model analysis restricted to the temporal lobes revealed additional associations between positive animacy index scores (i.e. a disproportionately poorer performance on animate objects) and cortical atrophy in the right perirhinal and entorhinal cortex, superior, middle, and inferior temporal gyri, and the anterior fusiform gyrus, as well as the left anterior fusiform gyrus. Taken together, our results indicate that a category-selective semantic deficit for animate objects is a characteristic feature of semantic variant primary progressive aphasia that is detectable in most individuals. Our imaging findings provide further support for the role of the right perirhinal cortex and other temporal lobe regions in the semantic processing of animate objects.

Untangling the Animacy Organization of Occipitotemporal Cortex

Some of the most impressive functional specializations in the human brain are found in the occipitotemporal cortex (OTC), where several areas exhibit selectivity for a small number of visual categories, such as faces and bodies, and spatially cluster based on stimulus animacy. Previous studies suggest this animacy organization reflects the representation of an intuitive taxonomic hierarchy, distinct from the presence of face- and body-selective areas in OTC. Using human functional magnetic resonance imaging, we investigated the independent contribution of these two factors-the face-body division and taxonomic hierarchy-in accounting for the animacy organization of OTC and whether they might also be reflected in the architecture of several deep neural networks that have not been explicitly trained to differentiate taxonomic relations. We found that graded visual selectivity, based on animal resemblance to human faces and bodies, masquerades as an apparent animacy continuum, which suggests that taxonomy is not a separate factor underlying the organization of the ventral visual pathway.SIGNIFICANCE STATEMENT Portions of the visual cortex are specialized to determine whether types of objects are animate in the sense of being capable of self-movement. Two factors have been proposed as accounting for this animacy organization: representations of faces and bodies and an intuitive taxonomic continuum of humans and animals. We performed an experiment to assess the independent contribution of both of these factors. We found that graded visual representations, based on animal resemblance to human faces and bodies, masquerade as an apparent animacy continuum, suggesting that taxonomy is not a separate factor underlying the organization of areas in the visual cortex.

Sensitive Measures of Cognition in Mild Cognitive Impairment

BACKGROUND

Sensitive measures of cognition are needed in preclinical and prodromal Alzheimer's disease (AD) to track cognitive change and evaluate potential interventions. Neurofibrillary tangle pathology in AD is first observed in Brodmann Area 35 (BA35), the medial portion of the perirhinal cortex. The importance of the perirhinal cortex for semantic memory may explain early impairments of semantics in preclinical AD. Additionally, our research has tied figurative language impairment to neurodegenerative disease.

OBJECTIVE

We aim to identify tasks that are sensitive to cognitive impairment in individuals with mild cognitive impairment (MCI), and that are sensitive to atrophy in BA35.

METHODS

Individuals with MCI and cognitively normal participants (CN) were tested on productive and receptive experimental measures of semantic memory and experimental tests of figurative language comprehension (including metaphor and verbal analogy). Performance was related to structural imaging and standard neuropsychological assessment.

RESULTS

On the experimental tests of semantics and figurative language, people with MCI performed worse than CN participants. The experimental semantic memory tasks are sensitive and specific; performance on the experimental semantic memory tasks related to medial temporal lobe structural integrity, including BA35, while standard neuropsychological assessments of semantic memory did not, demonstrating the sensitivity of these experimental measures. A visuo-spatial analogy task did not differentiate groups, confirming the specificity of semantic and figurative language tasks.

CONCLUSION

These experimental measures appear sensitive to cognitive change and neurodegeneration early in the AD trajectory and may prove useful in tracking cognitive change in clinical trials aimed at early intervention.

Distinct cortical systems reinstate the content and context of episodic memories

Episodic recall depends upon the reinstatement of cortical activity present during the formation of a memory. Evidence from functional neuroimaging and invasive recordings in humans suggest that reinstatement organizes our memories by time or content, yet the neural systems involved in reinstating these unique types of information remain unclear. Here, combining computational modeling and intracranial recordings from 69 epilepsy patients, we show that two cortical systems uniquely reinstate the semantic content and temporal context of previously studied items during free recall. Examining either the posterior medial or anterior temporal networks, we find that forward encoding models trained on the brain's response to the temporal and semantic attributes of items can predict the serial position and semantic category of unseen items. During memory recall, these models uniquely link reinstatement of temporal context and semantic content to these posterior and anterior networks, respectively. These findings demonstrate how specialized cortical systems enable the human brain to target specific memories.

The neural representation of abstract words may arise through grounding word meaning in language itself

In order to describe how humans represent meaning in the brain, one must be able to account for not just concrete words but, critically, also abstract words, which lack a physical referent. Hebbian formalism and optimization are basic principles of brain function, and they provide an appealing approach for modeling word meanings based on word co‐occurrences. We provide proof of concept that a statistical model of the semantic space can account for neural representations of both concrete and abstract words, using MEG. Here, we built a statistical model using word embeddings extracted from a text corpus. This statistical model was used to train a machine learning algorithm to successfully decode the MEG signals evoked by written words. In the model, word abstractness emerged from the statistical regularities of the language environment. Representational similarity analysis further showed that this salient property of the model co‐varies, at 280–420 ms after visual word presentation, with activity in regions that have been previously linked with processing of abstract words, namely the left‐hemisphere frontal, anterior temporal and superior parietal cortex. In light of these results, we propose that the neural encoding of word meanings can arise through statistical regularities, that is, through grounding in language itself.

Representations of conceptual information during automatic and active semantic access

When we read a word or see an object, conceptual meaning is automatically accessed. However, previous research investigating non-perceptual sensitivity to semantic class has employed active tasks. In this fMRI study, we tested whether conceptual representations in regions constituting the semantic network are invoked during passive semantic access and whether these representations are modulated by the need to access deeper knowledge. Seventeen healthy subjects performed a semantically active typicality judgment task and a semantically passive phonetic decision task, in both the written and the spoken input-modalities. Stimuli consisted of one hundred forty-four concepts drawn from six semantic categories. Multivariate Pattern Analysis (MVPA) revealed that the left posterior middle temporal gyrus (pMTG), posterior ventral temporal cortex (pVTC) and pars triangularis of the left inferior frontal gyrus (IFG) showed a stronger sensitivity to semantic category when active rather than passive semantic access is required. Using a cross-task training/testing classifier, we determined that conceptual representations were not only active in these regions during passive semantic access but that the neural representation of these categories was common to both active and passive access. Collectively, these results show that while representations in the pMTG, pVTC and IFG are strongly modulated by active conceptual access, consistent representational patterns are present during active and passive conceptual access in these same regions.

Reconciling the object and spatial processing views of the perirhinal cortex through task-relevant unitization

The perirhinal cortex is situated on the border between sensory association cortex and the hippocampal formation. It serves an important function as a transition area between the sensory neocortex and the medial temporal lobe. While the perirhinal cortex has traditionally been associated with object coding and the "what" pathway of the temporal lobe, current evidence suggests a broader function of the perirhinal cortex in solving feature ambiguity and processing complex stimuli. Besides fulfilling functions in object coding, recent neurophysiological findings in freely moving rodents indicate that the perirhinal cortex also contributes to spatial and contextual processing beyond individual sensory modalities. Here, we address how these two opposing views on perirhinal cortex-the object-centered and spatial-contextual processing hypotheses-may be reconciled. The perirhinal cortex is consistently recruited when different features can be merged perceptually or conceptually into a single entity. Features that are unitized in these entities include object information from multiple sensory domains, reward associations, semantic features and spatial/contextual associations. We propose that the same perirhinal network circuits can be flexibly deployed for multiple cognitive functions, such that the perirhinal cortex performs similar unitization operations on different types of information, depending on behavioral demands and ranging from the object-related domain to spatial, contextual and semantic information.

Age-related dedifferentiation and hyperdifferentiation of perceptual and mnemonic representations

Preliminary evidence indicates that occipito-temporal activation patterns for different visual stimuli are less distinct in older (OAs) than younger (YAs) adults, suggesting a dedifferentiation of visual representations with aging. Yet, it is unclear if this deficit (1) affects only sensory or also categorical aspects of representations during visual perception (perceptual representations), and (2) affects only perceptual or also mnemonic representations. To investigate these issues, we fMRI-scanned YAs and OAs viewing and then remembering visual scenes. First, using representational similarity analyses, we distinguished sensory vs. categorical features of perceptual representations. We found that, compared to YAs, sensory features in early visual cortex were less differentiated in OAs (i.e., age-related dedifferentiation), replicating previous research, whereas categorical features in anterior temporal lobe (ATL) were more differentiated in OAs. This is, to our knowledge, the first report of an age-related hyperdifferentiation. Second, we assessed the quality of mnemonic representations by measuring encoding-retrieval similarity (ERS) in activation patterns. We found that aging impaired mnemonic representations in early visual cortex and hippocampus but enhanced mnemonic representations in ATL. Thus, both perceptual and mnemonic representations in ATL were enhanced by aging. In sum, our findings suggest that aging impairs visual and mnemonic representations in posterior brain regions but enhances them in anterior regions.

Semantic Knowledge of Famous People and Places Is Represented in Hippocampus and Distinct Cortical Networks

Studies have found that anterior temporal lobe (ATL) is critical for detailed knowledge of object categories, suggesting that it has an important role in semantic memory. However, in addition to information about entities, such as people and objects, semantic memory also encompasses information about places. We tested predictions stemming from the PMAT model, which proposes there are distinct systems that support different kinds of semantic knowledge: an anterior temporal (AT) network, which represents information about entities; and a posterior medial (PM) network, which represents information about places. We used representational similarity analysis to test for activation of semantic features when human participants viewed pictures of famous people and places, while controlling for visual similarity. We used machine learning techniques to quantify the semantic similarity of items based on encyclopedic knowledge in the Wikipedia page for each item and found that these similarity models accurately predict human similarity judgments. We found that regions within the AT network, including ATL and inferior frontal gyrus, represented detailed semantic knowledge of people. In contrast, semantic knowledge of places was represented within PM network areas, including precuneus, posterior cingulate cortex, angular gyrus, and parahippocampal cortex. Finally, we found that hippocampus, which has been proposed to serve as an interface between the AT and PM networks, represented fine-grained semantic similarity for both individual people and places. Our results provide evidence that semantic knowledge of people and places is represented separately in AT and PM areas, whereas hippocampus represents semantic knowledge of both categories.SIGNIFICANCE STATEMENT Humans acquire detailed semantic knowledge about people (e.g., their occupation and personality) and places (e.g., their cultural or historical significance). While research has demonstrated that brain regions preferentially respond to pictures of people and places, less is known about whether these regions preferentially represent semantic knowledge about specific people and places. We used machine learning techniques to develop a model of semantic similarity based on information available from Wikipedia, validating the model against similarity ratings from human participants. Using our computational model, we found that semantic knowledge about people and places is represented in distinct anterior temporal and posterior medial brain networks, respectively. We further found that hippocampus, an important memory center, represented semantic knowledge for both types of stimuli.

Clarifying the role of higher-level cortices in resolving perceptual ambiguity using ultra high field fMRI

The brain is organized into distinct, flexible networks. Within these networks, cognitive variables such as attention can modulate sensory representations in accordance with moment-to-moment behavioral requirements. These modulations can be studied by varying task demands; however, the tasks employed are often incongruent with the postulated functions of a sensory system, limiting the characterization of the system in relation to natural behaviors. Here we combine domain-specific task manipulations and ultra-high field fMRI to study the nature of top-down modulations. We exploited faces, a visual category underpinned by a complex cortical network, and instructed participants to perform either a stimulus-relevant/domain-specific or a stimulus-irrelevant task in the scanner. We found that 1. perceptual ambiguity (i.e. difficulty of achieving a stable percept) is encoded in top-down modulations from higher-level cortices; 2. the right inferior-temporal lobe is active under challenging conditions and uniquely encodes trial-by-trial variability in face perception.

Visual and Semantic Representations Predict Subsequent Memory in Perceptual and Conceptual Memory Tests

It is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used functional magnetic resonance imaging (fMRI) and representational similarity analysis to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative observed ("is round"), taxonomic ("is a fruit"), and encyclopedic features ("is sweet"). We identified brain regions where each representation type predicted later perceptual memory, conceptual memory, or both (general memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent perceptual memory in visual cortices, but also facilitated conceptual and general memory in more anterior regions. Semantic representations, in turn, predicted perceptual memory in visual cortex, conceptual memory in the perirhinal and inferior prefrontal cortex, and general memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

Knowing what you need to know in advance: The neural processes underpinning flexible semantic retrieval of thematic and taxonomic relations

Semantic retrieval is flexible, allowing us to focus on subsets of features and associations that are relevant to the current task or context: for example, we use taxonomic relations to locate items in the supermarket (carrots are a vegetable), but thematic associations to decide which tools we need when cooking (carrot goes with peeler). We used fMRI to investigate the neural basis of this form of semantic flexibility; in particular, we asked how retrieval unfolds differently when participants have advanced knowledge of the type of link to retrieve between concepts (taxonomic or thematic). Participants performed a semantic relatedness judgement task: on half the trials, they were cued to search for a taxonomic or thematic link, while on the remaining trials, they judged relatedness without knowing which type of semantic relationship would be relevant. Left inferior frontal gyrus showed greater activation when participants knew the trial type in advance. An overlapping region showed a stronger response when the semantic relationship between the items was weaker, suggesting this structure supports both top-down and bottom-up forms of semantic control. Multivariate pattern analysis further revealed that the neural response in left inferior frontal gyrus reflects goal information related to different conceptual relationships. Top-down control specifically modulated the response in visual cortex: when the goal was unknown, there was greater deactivation to the first word, and greater activation to the second word. We conclude that top-down control of semantic retrieval is primarily achieved through the gating of task-relevant 'spoke' regions.

Deficits in recent but not lifetime familiarity in amnestic mild cognitive impairment

People with amnestic mild cognitive impairment (aMCI) repeat questions, seemingly without any sense of familiarity (i.e., recognition of prior occurrence without recollection of episodic context). Accumulation of neurofibrillary tau in preclinical Alzheimer's disease begins in perirhinal cortex, a medial temporal lobe region linked to familiarity. Both observations would predict impaired familiarity assessment in aMCI; however, the extant evidence is mixed. To reveal familiarity impairments, it may be necessary to minimize the influence of recollection. In the current study, older adults with aMCI and healthy controls were administered two tasks on which a well-characterized patient (NB) with selective familiarity impairments due to surgical left temporal lobe excision sparing the hippocampus showed abnormal performance: frequency judgments for words exposed to in a recent study phase and judgments of cumulative lifetime familiarity for object concepts denoted by words. We also administered a process dissociation procedure (PDP) task that previously revealed spared familiarity in aMCI. We predicted that familiarity would be spared in aMCI on the PDP task, but impaired when assessed by frequency judgments for recent laboratory exposures and lifetime familiarity judgments. Familiarity was spared on the PDP task, but was impaired when probed with frequency judgments for recently exposed words in aMCI. Lifetime familiarity was also not impaired in aMCI. These results highlight the benefits of studying familiarity under conditions that minimize recollection and the value of frequency judgments in revealing familiarity deficits, and suggest that perirhinal cortex may not be necessary for accessing familiarity accumulated over a lifetime of experience.

Mapping the Scene and Object Processing Networks by Intracranial EEG

Human perception and cognition are based predominantly on visual information processing. Much of the information regarding neuronal correlates of visual processing has been derived from functional imaging studies, which have identified a variety of brain areas contributing to visual analysis, recognition, and processing of objects and scenes. However, only two of these areas, namely the parahippocampal place area (PPA) and the lateral occipital complex (LOC), were verified and further characterized by intracranial electroencephalogram (iEEG). iEEG is a unique measurement technique that samples a local neuronal population with high temporal and anatomical resolution. In the present study, we aimed to expand on previous reports and examine brain activity for selectivity of scenes and objects in the broadband high-gamma frequency range (50–150 Hz). We collected iEEG data from 27 epileptic patients while they watched a series of images, containing objects and scenes, and we identified 375 bipolar channels responding to at least one of these two categories. Using K-means clustering, we delineated their brain localization. In addition to the two areas described previously, we detected significant responses in two other scene-selective areas, not yet reported by any electrophysiological studies; namely the occipital place area (OPA) and the retrosplenial complex. Moreover, using iEEG we revealed a much broader network underlying visual processing than that described to date, using specialized functional imaging experimental designs. Here, we report the selective brain areas for scene processing include the posterior collateral sulcus and the anterior temporal region, which were already shown to be related to scene novelty and landmark naming. The object-selective responses appeared in the parietal, frontal, and temporal regions connected with tool use and object recognition. The temporal analyses specified the time course of the category selectivity through the dorsal and ventral visual streams. The receiver operating characteristic analyses identified the PPA and the fusiform portion of the LOC as being the most selective for scenes and objects, respectively. Our findings represent a valuable overview of visual processing selectivity for scenes and objects based on iEEG analyses and thus, contribute to a better understanding of visual processing in the human brain.

Distinct fronto-temporal substrates of distributional and taxonomic similarity among words: evidence from RSA of BOLD signals

A class of semantic theories defines concepts in terms of statistical distributions of lexical items, basing meaning on vectors of word co-occurrence frequencies. A different approach emphasizes abstract hierarchical taxonomic relationships among concepts. However, the functional relevance of these different accounts and how they capture information-encoding of lexical meaning in the brain still remains elusive. We investigated to what extent distributional and taxonomic models explained word-elicited neural responses using cross-validated representational similarity analysis (RSA) of functional magnetic resonance imaging (fMRI) and model comparisons. Our findings show that the brain encodes both types of semantic information, but in distinct cortical regions. Posterior middle temporal regions reflected lexical-semantic similarity based on hierarchical taxonomies, in coherence with the action-relatedness of specific semantic word categories. In contrast, distributional semantics best predicted the representational patterns in left inferior frontal gyrus (LIFG, BA 47). Both representations coexisted in the angular gyrus supporting semantic binding and integration. These results reveal that neuronal networks with distinct cortical distributions across higher-order association cortex encode different representational properties of word meanings. Taxonomy may shape long-term lexical-semantic representations in memory consistently with the sensorimotor details of semantic categories, whilst distributional knowledge in the LIFG (BA 47) may enable semantic combinatorics in the context of language use. Our approach helps to elucidate the nature of semantic representations essential for understanding human language.