Dimensionality and Ramping: Signatures of Sentence Integration in the Dynamics of Brains and Deep Language Models

Hierarchical dynamic coding coordinates speech comprehension in the human brain

Speech comprehension involves transforming an acoustic waveform into meaning. To do so, the human brain generates a hierarchy of features that converts the sensory input into increasingly abstract language properties. However, little is known about how rapid incoming sequences of hierarchical features are continuously coordinated. Here, we propose that each language feature is supported by a dynamic neural code, which represents the sequence history of hierarchical features in parallel. To test this 'Hierarchical Dynamic Coding' (HDC) hypothesis, we use time-resolved decoding of brain activity to track the construction, maintenance, and update of a comprehensive hierarchy of language features spanning phonetic, word form, lexical-syntactic, syntactic and semantic representations. For this, we recorded 21 native English participants with magnetoencephalography (MEG), while they listened to two hours of short stories in English. Our analyses reveal three main findings. First, the brain represents and simultaneously maintains a sequence of hierarchical features. Second, the duration of these representations depends on their level in the language hierarchy. Third, each representation is maintained by a dynamic neural code, which evolves at a speed commensurate with its corresponding linguistic level. This HDC preserves the maintenance of information over time while limiting destructive interference between successive features. Overall, HDC reveals how the human brain maintains and updates the continuously unfolding language hierarchy during natural speech comprehension, thereby anchoring linguistic theories to their biological implementations.

Animal models of the human brain: Successes, limitations, and alternatives

The last three decades of research in human cognitive neuroscience have given us an initial "parts list" for the human mind in the form of a set of cortical regions with distinct and often very specific functions. But current neuroscientific methods in humans have limited ability to reveal exactly what these regions represent and compute, the causal role of each in behavior, and the interactions among regions that produce real-world cognition. Animal models can help to answer these questions when homologues exist in other species, like the face system in macaques. When homologues do not exist in animals, for example for speech and music perception, and understanding of language or other people's thoughts, intracranial recordings in humans play a central role, along with a new alternative to animal models: artificial neural networks.

The human auditory cortex concurrently tracks syllabic and phonemic timescales via acoustic spectral flux

Dynamical theories of speech processing propose that the auditory cortex parses acoustic information in parallel at the syllabic and phonemic timescales. We developed a paradigm to independently manipulate both linguistic timescales, and acquired intracranial recordings from 11 patients who are epileptic listening to French sentences. Our results indicate that (i) syllabic and phonemic timescales are both reflected in the acoustic spectral flux; (ii) during comprehension, the auditory cortex tracks the syllabic timescale in the theta range, while neural activity in the alpha-beta range phase locks to the phonemic timescale; (iii) these neural dynamics occur simultaneously and share a joint spatial location; (iv) the spectral flux embeds two timescales-in the theta and low-beta ranges-across 17 natural languages. These findings help us understand how the human brain extracts acoustic information from the continuous speech signal at multiple timescales simultaneously, a prerequisite for subsequent linguistic processing.

Neural populations in the language network differ in the size of their temporal receptive windows

Despite long knowing what brain areas support language comprehension, our knowledge of the neural computations that these frontal and temporal regions implement remains limited. One important unresolved question concerns functional differences among the neural populations that comprise the language network. Here we leveraged the high spatiotemporal resolution of human intracranial recordings (n = 22) to examine responses to sentences and linguistically degraded conditions. We discovered three response profiles that differ in their temporal dynamics. These profiles appear to reflect different temporal receptive windows, with average windows of about 1, 4 and 6 words, respectively. Neural populations exhibiting these profiles are interleaved across the language network, which suggests that all language regions have direct access to distinct, multiscale representations of linguistic input-a property that may be critical for the efficiency and robustness of language processing.

Pronouns reactivate conceptual representations in human hippocampal neurons

During discourse comprehension, every new word adds to an evolving representation of meaning that accumulates over consecutive sentences and constrains the next words. To minimize repetition and utterance length, languages use pronouns, like the word "she," to refer to nouns and phrases that were previously introduced. It has been suggested that language comprehension requires that pronouns activate the same neuronal representations as the nouns themselves. We recorded from individual neurons in the human hippocampus during a reading task. Cells that were selective to a particular noun were later reactivated by pronouns that refer to the cells' preferred noun. These results imply that concept cells contribute to a rapid and dynamic semantic memory network that is recruited during language comprehension.

Lesion-symptom Mapping of Acceptability Judgments in Chronic Poststroke Aphasia Reveals the Neurobiological Underpinnings of Receptive Syntax

Disagreements persist regarding the neural basis of syntactic processing, which has been linked both to inferior frontal and posterior temporal regions of the brain. One focal point of the debate concerns the role of inferior frontal areas in receptive syntactic ability, which is mostly assessed using sentence comprehension involving complex syntactic structures, a task that is potentially confounded with working memory. Syntactic acceptability judgments may provide a better measure of receptive syntax by reducing the need to use high working memory load and complex sentences and by enabling assessment of various types of syntactic violations. We therefore tested the perception of grammatical violations by people with poststroke aphasia (n = 25), along with matched controls (n = 16), using English sentences involving errors in word order, agreement, or subcategorization. Lesion data were also collected. Control participants performed near ceiling in accuracy with higher discriminability of agreement and subcategorization violations than word order; aphasia participants were less able to discriminate violations, but, on average, paralleled control participants discriminability of types of violations. Lesion-symptom mapping showed a correlation between discriminability and posterior temporal regions, but not inferior frontal regions. We argue that these results diverge from models holding that frontal areas are amodal core regions in syntactic structure building and favor models that posit a core hierarchical system in posterior temporal regions.

Language is primarily a tool for communication rather than thought

Language is a defining characteristic of our species, but the function, or functions, that it serves has been debated for centuries. Here we bring recent evidence from neuroscience and allied disciplines to argue that in modern humans, language is a tool for communication, contrary to a prominent view that we use language for thinking. We begin by introducing the brain network that supports linguistic ability in humans. We then review evidence for a double dissociation between language and thought, and discuss several properties of language that suggest that it is optimized for communication. We conclude that although the emergence of language has unquestionably transformed human culture, language does not appear to be a prerequisite for complex thought, including symbolic thought. Instead, language is a powerful tool for the transmission of cultural knowledge; it plausibly co-evolved with our thinking and reasoning capacities, and only reflects, rather than gives rise to, the signature sophistication of human cognition.

The language network as a natural kind within the broader landscape of the human brain

Language behaviour is complex, but neuroscientific evidence disentangles it into distinct components supported by dedicated brain areas or networks. In this Review, we describe the 'core' language network, which includes left-hemisphere frontal and temporal areas, and show that it is strongly interconnected, independent of input and output modalities, causally important for language and language-selective. We discuss evidence that this language network plausibly stores language knowledge and supports core linguistic computations related to accessing words and constructions from memory and combining them to interpret (decode) or generate (encode) linguistic messages. We emphasize that the language network works closely with, but is distinct from, both lower-level - perceptual and motor - mechanisms and higher-level systems of knowledge and reasoning. The perceptual and motor mechanisms process linguistic signals, but, in contrast to the language network, are sensitive only to these signals' surface properties, not their meanings; the systems of knowledge and reasoning (such as the system that supports social reasoning) are sometimes engaged during language use but are not language-selective. This Review lays a foundation both for in-depth investigations of these different components of the language processing pipeline and for probing inter-component interactions.

Tracking the neural codes for words and phrases during semantic composition, working-memory storage, and retrieval

The ability to compose successive words into a meaningful phrase is a characteristic feature of human cognition, yet its neural mechanisms remain incompletely understood. Here, we analyze the cortical mechanisms of semantic composition using magnetoencephalography (MEG) while participants read one-word, two-word, and five-word noun phrases and compared them with a subsequent image. Decoding of MEG signals revealed three processing stages. During phrase comprehension, the representation of individual words was sustained for a variable duration depending on phrasal context. During the delay period, the word code was replaced by a working-memory code whose activation increased with semantic complexity. Finally, the speed and accuracy of retrieval depended on semantic complexity and was faster for surface than for deep semantic properties. In conclusion, we propose that the brain initially encodes phrases using factorized dimensions for successive words but later compresses them in working memory and requires a period of decompression to access them.

ROSE: A Neurocomputational Architecture for Syntax

A comprehensive model of natural language processing in the brain must accommodate four components: representations, operations, structures and encoding. It further requires a principled account of how these different components mechanistically, and causally, relate to each another. While previous models have isolated regions of interest for structure-building and lexical access, and have utilized specific neural recording measures to expose possible signatures of syntax, many gaps remain with respect to bridging distinct scales of analysis that map onto these four components. By expanding existing accounts of how neural oscillations can index various linguistic processes, this article proposes a neurocomputational architecture for syntax, termed the ROSE model (Representation, Operation, Structure, Encoding). Under ROSE, the basic data structures of syntax are atomic features, types of mental representations (R), and are coded at the single-unit and ensemble level. Elementary computations (O) that transform these units into manipulable objects accessible to subsequent structure-building levels are coded via high frequency broadband γ activity. Low frequency synchronization and cross-frequency coupling code for recursive categorial inferences (S). Distinct forms of low frequency coupling and phase-amplitude coupling (δ-θ coupling via pSTS-IFG; θ-γ coupling via IFG to conceptual hubs in lateral and ventral temporal cortex) then encode these structures onto distinct workspaces (E). Causally connecting R to O is spike-phase/LFP coupling; connecting O to S is phase-amplitude coupling; connecting S to E is a system of frontotemporal traveling oscillations; connecting E back to lower levels is low-frequency phase resetting of spike-LFP coupling. This compositional neural code has important implications for algorithmic accounts, since it makes concrete predictions for the appropriate level of study for psycholinguistic parsing models. ROSE is reliant on neurophysiologically plausible mechanisms, is supported at all four levels by a range of recent empirical research, and provides an anatomically precise and falsifiable grounding for the basic property of natural language syntax: hierarchical, recursive structure-building.