The role of visual representations during the lexical access of spoken words

Phase-dependent word perception emerges from region-specific sensitivity to the statistics of language

Neural oscillations reflect fluctuations in excitability, which biases the percept of ambiguous sensory input. Why this bias occurs is still not fully understood. We hypothesized that neural populations representing likely events are more sensitive, and thereby become active on earlier oscillatory phases, when the ensemble itself is less excitable. Perception of ambiguous input presented during less-excitable phases should therefore be biased toward frequent or predictable stimuli that have lower activation thresholds. Here, we show such a frequency bias in spoken word recognition using psychophysics, magnetoencephalography (MEG), and computational modelling. With MEG, we found a double dissociation, where the phase of oscillations in the superior temporal gyrus and medial temporal gyrus biased word-identification behavior based on phoneme and lexical frequencies, respectively. This finding was reproduced in a computational model. These results demonstrate that oscillations provide a temporal ordering of neural activity based on the sensitivity of separable neural populations.

Auditory Word Comprehension Is Less Incremental in Isolated Words

Partial speech input is often understood to trigger rapid and automatic activation of successively higher-level representations of words, from sound to meaning. Here we show evidence from magnetoencephalography that this type of incremental processing is limited when words are heard in isolation as compared to continuous speech. This suggests a less unified and automatic word recognition process than is often assumed. We present evidence from isolated words that neural effects of phoneme probability, quantified by phoneme surprisal, are significantly stronger than (statistically null) effects of phoneme-by-phoneme lexical uncertainty, quantified by cohort entropy. In contrast, we find robust effects of both cohort entropy and phoneme surprisal during perception of connected speech, with a significant interaction between the contexts. This dissociation rules out models of word recognition in which phoneme surprisal and cohort entropy are common indicators of a uniform process, even though these closely related information-theoretic measures both arise from the probability distribution of wordforms consistent with the input. We propose that phoneme surprisal effects reflect automatic access of a lower level of representation of the auditory input (e.g., wordforms) while the occurrence of cohort entropy effects is task sensitive, driven by a competition process or a higher-level representation that is engaged late (or not at all) during the processing of single words.

Stimulus-independent neural coding of event semantics: Evidence from cross-sentence fMRI decoding

Multivariate neuroimaging studies indicate that the brain represents word and object concepts in a format that readily generalises across stimuli. Here we investigated whether this was true for neural representations of simple events described using sentences. Participants viewed sentences describing four events in different ways. Multivariate classifiers were trained to discriminate the four events using a subset of sentences, allowing us to test generalisation to novel sentences. We found that neural patterns in a left-lateralised network of frontal, temporal and parietal regions discriminated events in a way that generalised successfully over changes in the syntactic and lexical properties of the sentences used to describe them. In contrast, decoding in visual areas was sentence-specific and failed to generalise to novel sentences. In the reverse analysis, we tested for decoding of syntactic and lexical structure, independent of the event being described. Regions displaying this coding were limited and largely fell outside the canonical semantic network. Our results indicate that a distributed neural network represents the meaning of event sentences in a way that is robust to changes in their structure and form. They suggest that the semantic system disregards the surface properties of stimuli in order to represent their underlying conceptual significance.

Phase synchronization varies systematically with linguistic structure composition

Computation in neuronal assemblies is putatively reflected in the excitatory and inhibitory cycles of activation distributed throughout the brain. In speech and language processing, coordination of these cycles resulting in phase synchronization has been argued to reflect the integration of information on different timescales (e.g. segmenting acoustics signals to phonemic and syllabic representations; (Giraud and Poeppel 2012 Nat. Neurosci. 15, 511 (doi:10.1038/nn.3063)). A natural extension of this claim is that phase synchronization functions similarly to support the inference of more abstract higher-level linguistic structures (Martin 2016 Front. Psychol. 7, 120; Martin and Doumas 2017 PLoS Biol. 15, e2000663 (doi:10.1371/journal.pbio.2000663); Martin and Doumas. 2019 Curr. Opin. Behav. Sci. 29, 77-83 (doi:10.1016/j.cobeha.2019.04.008)). Hale et al. (Hale et al. 2018 Finding syntax in human encephalography with beam search. arXiv 1806.04127 (http://arxiv.org/abs/1806.04127)) showed that syntactically driven parsing decisions predict electroencephalography (EEG) responses in the time domain; here we ask whether phase synchronization in the form of either inter-trial phrase coherence or cross-frequency coupling (CFC) between high-frequency (i.e. gamma) bursts and lower-frequency carrier signals (i.e. delta, theta), changes as the linguistic structures of compositional meaning (viz., bracket completions, as denoted by the onset of words that complete phrases) accrue. We use a naturalistic story-listening EEG dataset from Hale et al. to assess the relationship between linguistic structure and phase alignment. We observe increased phase synchronization as a function of phrase counts in the delta, theta, and gamma bands, especially for function words. A more complex pattern emerged for CFC as phrase count changed, possibly related to the lack of a one-to-one mapping between 'size' of linguistic structure and frequency band-an assumption that is tacit in recent frameworks. These results emphasize the important role that phase synchronization, desynchronization, and thus, inhibition, play in the construction of compositional meaning by distributed neural networks in the brain. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Names and their meanings: A dual-process account of proper-name encoding and retrieval

The ability to pick out a unique entity with a proper name is an important component of human language. It has been a primary focus of research in the philosophy of language since the nineteenth century. Brain-based evidence has shed new light on this capacity, and an extensive literature indicates the involvement of distinct fronto-temporal and temporo-occipito-parietal association cortices in proper-name retrieval. However, comparatively few efforts have sought to explain how memory encoding processes lead to the later recruitment of these distinct regions at retrieval. Here, we provide a unified account of proper-name encoding and retrieval, reviewing evidence that socio-emotional and unitized encoding subserve the retrieval of proper names via anterior-temporal-prefrontal activations. Meanwhile, non-unitized item-item and item-context encoding support subsequent retrieval, largely dependent on the temporo-occipito-parietal cortex. We contend that this well-established divergence in encoding systems can explain how proper names are later retrieved from distinct neural structures. Furthermore, we explore how evidence reviewed here can inform a century-and-a-half-old debate about proper names and the meanings they pick out.

The Neural Dynamics of Novel Scene Imagery

Retrieval of long-term episodic memories is characterized by synchronized neural activity between hippocampus and ventromedial prefrontal cortex (vmPFC), with additional evidence that vmPFC activity leads that of the hippocampus. It has been proposed that the mental generation of scene imagery is a crucial component of episodic memory processing. If this is the case, then a comparable interaction between the two brain regions should exist during the construction of novel scene imagery. To address this question, we leveraged the high temporal resolution of MEG to investigate the construction of novel mental imagery. We tasked male and female humans with imagining scenes and single isolated objects in response to one-word cues. We performed source-level power, coherence, and causality analyses to characterize the underlying interregional interactions. Both scene and object imagination resulted in theta power changes in the anterior hippocampus. However, higher theta coherence was observed between the hippocampus and vmPFC in the scene compared with the object condition. This interregional theta coherence also predicted whether imagined scenes were subsequently remembered. Dynamic causal modeling of this interaction revealed that vmPFC drove activity in hippocampus during novel scene construction. Additionally, theta power changes in the vmPFC preceded those observed in the hippocampus. These results constitute the first evidence in humans that episodic memory retrieval and scene imagination rely on similar vmPFC–hippocampus neural dynamics. Furthermore, they provide support for theories emphasizing similarities between both cognitive processes and perspectives that propose the vmPFC guides the construction of context-relevant representations in the hippocampus.

SIGNIFICANCE STATEMENT Episodic memory retrieval is characterized by a dialog between hippocampus and ventromedial prefrontal cortex (vmPFC). It has been proposed that the mental generation of scene imagery is a crucial component of episodic memory processing. An ensuing prediction would be of a comparable interaction between the two brain regions during the construction of novel scene imagery. Here, we leveraged the high temporal resolution of MEG and combined it with a scene imagination task. We found that a hippocampal–vmPFC dialog existed and that it took the form of vmPFC driving the hippocampus. We conclude that episodic memory and scene imagination share fundamental neural dynamics and the process of constructing vivid, spatially coherent, contextually appropriate scene imagery is strongly modulated by vmPFC.

Compounding matters: Event-related potential evidence for early semantic access to compound words

Reading words in a native language triggers a largely obligatory cognitive process that we accept as leading to comprehension of the word - we cannot suppress our understanding of word meaning. In this study, we investigated the early stages of this comprehension process by means of event-related potentials (ERPs) to identify when this processing of meaning - that is, semantic processing - first occurs. We report that, when processing visually presented compound words, semantic access at some level occurs as early as the P100 and persists through to the N400. Specifically, we focused on the P100 ERP component, and utilized the unique features of compound words (i.e. variation in the transparency of meaning) to investigate the speed with which we gain access to information about meaning (i.e. semantic access). Twenty-two participants performed a lexical decision task including 40 English compounds, which varied with respect to their constituent semantic transparency. Compounds ranged from full constituent semantic transparency (e.g. grapeseed) to partial transparency (e.g. grapefruit) to full opacity (e.g. hogwash). Regression analyses predicted ERP components from compound constituent transparency, adjusting for word frequency. Word frequency and transparency of both the first and second constituents each uniquely predicted P100 amplitude. Transparency of the second constituent, but not word frequency, predicted later component amplitudes, including that of the N400. The findings suggest that some level of semantic access occurs as early as the P100. Overall, these results support models which emphasize simultaneous processing of form and meaning as opposed to serial or hierarchical approaches.

Are visual processes causally involved in "perceptual simulation" effects in the sentence-picture verification task?

Many studies have shown that sentences implying an object to have a certain shape produce a robust reaction time advantage for shape-matching pictures in the sentence-picture verification task. Typically, this finding has been interpreted as evidence for perceptual simulation, i.e., that access to implicit shape information involves the activation of modality-specific visual processes. It follows from this proposal that disrupting visual processing during sentence comprehension should interfere with perceptual simulation and obliterate the match effect. Here we directly test this hypothesis. Participants listened to sentences while seeing either visual noise that was previously shown to strongly interfere with basic visual processing or a blank screen. Experiments 1 and 2 replicated the match effect but crucially visual noise did not modulate it. When an interference technique was used that targeted high-level semantic processing (Experiment 3) however the match effect vanished. Visual noise specifically targeting high-level visual processes (Experiment 4) only had a minimal effect on the match effect. We conclude that the shape match effect in the sentence-picture verification paradigm is unlikely to rely on perceptual simulation.

Electrophysiology Reveals the Neural Dynamics of Naturalistic Auditory Language Processing: Event-Related Potentials Reflect Continuous Model Updates

The recent trend away from ANOVA-based analyses places experimental investigations into the neurobiology of cognition in more naturalistic and ecologically valid designs within reach. Using mixed-effects models for epoch-based regression, we demonstrate the feasibility of examining event-related potentials (ERPs), and in particular the N400, to study the neural dynamics of human auditory language processing in a naturalistic setting. Despite the large variability between trials during naturalistic stimulation, we replicated previous findings from the literature: the effects of frequency, animacy, and word order and find previously unexplored interaction effects. This suggests a new perspective on ERPs, namely, as a continuous modulation reflecting continuous stimulation instead of a series of discrete and essentially sequential processes locked to discrete events.

Cortical entrainment to music and its modulation by expertise

Recent studies establish that cortical oscillations track naturalistic speech in a remarkably faithful way. Here, we test whether such neural activity, particularly low-frequency (<8 Hz; delta-theta) oscillations, similarly entrain to music and whether experience modifies such a cortical phenomenon. Music of varying tempi was used to test entrainment at different rates. In three magnetoencephalography experiments, we recorded from nonmusicians, as well as musicians with varying years of experience. Recordings from nonmusicians demonstrate cortical entrainment that tracks musical stimuli over a typical range of tempi, but not at tempi below 1 note per second. Importantly, the observed entrainment correlates with performance on a concurrent pitch-related behavioral task. In contrast, the data from musicians show that entrainment is enhanced by years of musical training, at all presented tempi. This suggests a bidirectional relationship between behavior and cortical entrainment, a phenomenon that has not previously been reported. Additional analyses focus on responses in the beta range (∼15-30 Hz)-often linked to delta activity in the context of temporal predictions. Our findings provide evidence that the role of beta in temporal predictions scales to the complex hierarchical rhythms in natural music and enhances processing of musical content. This study builds on important findings on brainstem plasticity and represents a compelling demonstration that cortical neural entrainment is tightly coupled to both musical training and task performance, further supporting a role for cortical oscillatory activity in music perception and cognition.

Decomposition, lookup, and recombination: MEG evidence for the full decomposition model of complex visual word recognition

There is much evidence that visual recognition of morphologically complex words (e.g., teacher) proceeds via a decompositional route, first involving recognition of their component morphemes (teach + -er). According to the Full Decomposition model, after the visual decomposition stage, followed by morpheme lookup, there is a final "recombination" stage, in which the decomposed morphemes are combined and the well-formedness of the complex form is evaluated. Here, we use MEG to provide evidence for the temporally-differentiated stages of this model. First, we demonstrate an early effect of derivational family entropy, corresponding to the stem lookup stage; this is followed by a surface frequency effect, corresponding to the later recombination stage. We also demonstrate a late effect of a novel statistical measure, semantic coherence, which quantifies the gradient semantic well-formedness of complex words. Our findings illustrate the usefulness of corpus measures in investigating the component processes within visual word recognition.

Cortical activity predicts which older adults recognize speech in noise and when

Speech recognition in noise can be challenging for older adults and elicits elevated activity throughout a cingulo-opercular network that is hypothesized to monitor and modify behaviors to optimize performance. A word recognition in noise experiment was used to test the hypothesis that cingulo-opercular engagement provides performance benefit for older adults. Healthy older adults (N = 31; 50-81 years of age; mean pure tone thresholds <32 dB HL from 0.25 to 8 kHz, best ear; species: human) performed word recognition in multitalker babble at 2 signal-to-noise ratios (SNR = +3 or +10 dB) during a sparse sampling fMRI experiment. Elevated cingulo-opercular activity was associated with an increased likelihood of correct recognition on the following trial independently of SNR and performance on the preceding trial. The cingulo-opercular effect increased for participants with the best overall performance. These effects were lower for older adults compared with a younger, normal-hearing adult sample (N = 18). Visual cortex activity also predicted trial-level recognition for the older adults, which resulted from discrete decreases in activity before errors and occurred for the oldest adults with the poorest recognition. Participants demonstrating larger visual cortex effects also had reduced fractional anisotropy in an anterior portion of the left inferior frontal-occipital fasciculus, which projects between frontal and occipital regions where activity predicted word recognition. Together, the results indicate that older adults experience performance benefit from elevated cingulo-opercular activity, but not to the same extent as younger adults, and that declines in attentional control can limit word recognition.

The neural bases of taxonomic and thematic conceptual relations: an MEG study

Converging evidence from behavioral and neuroimaging studies of human concepts indicate distinct neural systems for taxonomic and thematic knowledge. A recent study of naming in aphasia found involvement of the anterior temporal lobe (ATL) during taxonomic (feature-based) processing, and involvement of the temporoparietal junction (TPJ) during thematic (function-based) processing. We conducted an online magnetoencephalography (MEG) study to examine the spatio-temporal nature of taxonomic and thematic relations. We measured participants' brain responses to words preceded by either a taxonomically or thematically related item (e.g., cottage→castle, king→castle). In a separate experiment we collected relatedness ratings of the word pairs from participants. We examined effects of relatedness and relation type on activation in ATL and TPJ regions of interest (ROIs) using permutation t-tests to identify differences in ROI activation between conditions as well as single-trial correlational analyses to examine the millisecond-by-millisecond influence of the stimulus variables on the ROIs. Taxonomic relations strongly predicted ATL activation, and both kinds of relations influenced the TPJ. Our results further strengthen the view of the ATL's importance to taxonomic knowledge. Moreover, they provide a nuanced view of thematic relations as involving taxonomic knowledge.

The neuroanatomic and neurophysiological infrastructure for speech and language

New tools and new ideas have changed how we think about the neurobiological foundations of speech and language processing. This perspective focuses on two areas of progress. First, focusing on spatial organization in the human brain, the revised functional anatomy for speech and language is discussed. The complexity of the network organization undermines the well-regarded classical model and suggests looking for more granular computational primitives, motivated both by linguistic theory and neural circuitry. Second, focusing on recent work on temporal organization, a potential role of cortical oscillations for speech processing is outlined. Such an implementational-level mechanism suggests one way to deal with the computational challenge of segmenting natural speech.