Incremental Language Comprehension Difficulty Predicts Activity in the Language Network but Not the Multiple Demand Network

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Current large language models (LLMs) rely on word prediction as their backbone pretraining task. Although word prediction is an important mechanism underlying language processing, human language comprehension occurs at multiple levels, involving the integration of words and sentences to achieve a full understanding of discourse. This study models language comprehension by using the next sentence prediction (NSP) task to investigate mechanisms of discourse-level comprehension. We show that NSP pretraining enhanced a model's alignment with brain data especially in the right hemisphere and in the multiple demand network, highlighting the contributions of nonclassical language regions to high-level language understanding. Our results also suggest that NSP can enable the model to better capture human comprehension performance and to better encode contextual information. Our study demonstrates that the inclusion of diverse learning objectives in a model leads to more human-like representations, and investigating the neurocognitive plausibility of pretraining tasks in LLMs can shed light on outstanding questions in language neuroscience.

The language network as a natural kind within the broader landscape of the human brain

Language behaviour is complex, but neuroscientific evidence disentangles it into distinct components supported by dedicated brain areas or networks. In this Review, we describe the 'core' language network, which includes left-hemisphere frontal and temporal areas, and show that it is strongly interconnected, independent of input and output modalities, causally important for language and language-selective. We discuss evidence that this language network plausibly stores language knowledge and supports core linguistic computations related to accessing words and constructions from memory and combining them to interpret (decode) or generate (encode) linguistic messages. We emphasize that the language network works closely with, but is distinct from, both lower-level - perceptual and motor - mechanisms and higher-level systems of knowledge and reasoning. The perceptual and motor mechanisms process linguistic signals, but, in contrast to the language network, are sensitive only to these signals' surface properties, not their meanings; the systems of knowledge and reasoning (such as the system that supports social reasoning) are sometimes engaged during language use but are not language-selective. This Review lays a foundation both for in-depth investigations of these different components of the language processing pipeline and for probing inter-component interactions.

Disentangling neuroplasticity mechanisms in post-stroke language recovery

A major objective in post-stroke aphasia research is to gain a deeper understanding of neuroplastic mechanisms that drive language recovery, with the ultimate goal of enhancing treatment outcomes. Subsequent to recent advances in neuroimaging techniques, we now have the ability to examine more closely how neural activity patterns change after a stroke. However, the way these neural activity changes relate to language impairments and language recovery is still debated. The aim of this review is to provide a theoretical framework to better investigate and interpret neuroplasticity mechanisms underlying language recovery in post-stroke aphasia. We detail two sets of neuroplasticity mechanisms observed at the synaptic level that may explain functional neuroimaging findings in post-stroke aphasia recovery at the network level: feedback-based homeostatic plasticity and associative Hebbian plasticity. In conjunction with these plasticity mechanisms, higher-order cognitive control processes dynamically modulate neural activity in other regions to meet communication demands, despite reduced neural resources. This work provides a network-level neurobiological framework for understanding neural changes observed in post-stroke aphasia and can be used to define guidelines for personalized treatment development.

Driving and suppressing the human language network using large language models

Transformer models such as GPT generate human-like language and are predictive of human brain responses to language. Here, using functional-MRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of the brain response associated with each sentence. We then use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress the activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also non-invasively control neural activity in higher-level cortical areas, such as the language network.

Functional characterization of the language network of polyglots and hyperpolyglots with precision fMRI

How do polyglots-individuals who speak five or more languages-process their languages, and what can this population tell us about the language system? Using fMRI, we identified the language network in each of 34 polyglots (including 16 hyperpolyglots with knowledge of 10+ languages) and examined its response to the native language, non-native languages of varying proficiency, and unfamiliar languages. All language conditions engaged all areas of the language network relative to a control condition. Languages that participants rated as higher proficiency elicited stronger responses, except for the native language, which elicited a similar or lower response than a non-native language of similar proficiency. Furthermore, unfamiliar languages that were typologically related to the participants' high-to-moderate-proficiency languages elicited a stronger response than unfamiliar unrelated languages. The results suggest that the language network's response magnitude scales with the degree of engagement of linguistic computations (e.g. related to lexical access and syntactic-structure building). We also replicated a prior finding of weaker responses to native language in polyglots than non-polyglot bilinguals. These results contribute to our understanding of how multiple languages coexist within a single brain and provide new evidence that the language network responds more strongly to stimuli that more fully engage linguistic computations.

Conversational production and comprehension: fMRI-evidence reminiscent of but deviant from the classical Broca-Wernicke model

A key question in research on the neurobiology of language is to which extent the language production and comprehension systems share neural infrastructure, but this question has not been addressed in the context of conversation. We utilized a public fMRI dataset where 24 participants engaged in unscripted conversations with a confederate outside the scanner, via an audio-video link. We provide evidence indicating that the two systems share neural infrastructure in the left-lateralized perisylvian language network, but diverge regarding the level of activation in regions within the network. Activity in the left inferior frontal gyrus was stronger in production compared to comprehension, while comprehension showed stronger recruitment of the left anterior middle temporal gyrus and superior temporal sulcus, compared to production. Although our results are reminiscent of the classical Broca-Wernicke model, the anterior (rather than posterior) temporal activation is a notable difference from that model. This is one of the findings that may be a consequence of the conversational setting, another being that conversational production activated what we interpret as higher-level socio-pragmatic processes. In conclusion, we present evidence for partial overlap and functional asymmetry of the neural infrastructure of production and comprehension, in the above-mentioned frontal vs temporal regions during conversation.

Will you read how I will read? Naturalistic fMRI predictors of emergent reading

Despite reading being an essential and almost universal skill in the developed world, reading proficiency varies substantially from person to person. To study why, the fMRI field is beginning to turn from single-word or nonword reading tasks to naturalistic stimuli like connected text and listening to stories. To study reading development in children just beginning to read, listening to stories is an appropriate paradigm because speech perception and phonological processing are important for, and are predictors of, reading proficiency. Our study examined the relationship between behavioral reading-related skills and the neural response to listening to stories in the fMRI environment. Functional MRI were gathered in a 3T TIM-Trio scanner. During the fMRI scan, children aged approximately 7 years listened to professionally narrated common short stories and answered comprehension questions following the narration. Analyses of the data used inter-subject correlation (ISC), and representational similarity analysis (RSA). Our primary finding is that ISC reveals areas of increased synchrony in both high- and low-performing emergent readers previously implicated in reading ability/disability. Of particular interest are that several previously identified brain regions (medial temporal gyrus (MTG), inferior frontal gyrus (IFG), inferior temporal gyrus (ITG)) were found to "synchronize" across higher reading ability participants, while lower reading ability participants had idiosyncratic activation patterns in these regions. Additionally, two regions (superior frontal gyrus (SFG) and another portion of ITG) were recruited by all participants, but their specific timecourse of activation depended on reading performance. These analyses support the idea that different brain regions involved in reading follow different developmental trajectories that correlate with reading proficiency on a spectrum rather than the usual dichotomy of poor readers versus strong readers.

Domain-general and language-specific contributions to speech production in a second language: an fMRI study using functional localizers

For bilinguals, speaking in a second language (L2) compared to the native language (L1) is usually more difficult. In this study we asked whether the difficulty in L2 production reflects increased demands imposed on domain-general or core language mechanisms. We compared the brain response to speech production in L1 and L2 within two functionally-defined networks in the brain: the Multiple Demand (MD) network and the language network. We found that speech production in L2 was linked to a widespread increase of brain activity in the domain-general MD network. The language network did not show a similarly robust differences in processing speech in the two languages, however, we found increased response to L2 production in the language-specific portion of the left inferior frontal gyrus (IFG). To further explore our results, we have looked at domain-general and language-specific response within the brain structures postulated to form a Bilingual Language Control (BLC) network. Within this network, we found a robust increase in response to L2 in the domain-general, but also in some language-specific voxels including in the left IFG. Our findings show that L2 production strongly engages domain-general mechanisms, but only affects language sensitive portions of the left IFG. These results put constraints on the current model of bilingual language control by precisely disentangling the domain-general and language-specific contributions to the difficulty in speech production in L2.

Left-Hemisphere Cortical Language Regions Respond Equally to Observed Dialogue and Monologue

Much of the language we encounter in our everyday lives comes in the form of conversation, yet the majority of research on the neural basis of language comprehension has used input from only one speaker at a time. Twenty adults were scanned while passively observing audiovisual conversations using functional magnetic resonance imaging. In a block-design task, participants watched 20 s videos of puppets speaking either to another puppet (the dialogue condition) or directly to the viewer (the monologue condition), while the audio was either comprehensible (played forward) or incomprehensible (played backward). Individually functionally localized left-hemisphere language regions responded more to comprehensible than incomprehensible speech but did not respond differently to dialogue than monologue. In a second task, participants watched videos (1-3 min each) of two puppets conversing with each other, in which one puppet was comprehensible while the other's speech was reversed. All participants saw the same visual input but were randomly assigned which character's speech was comprehensible. In left-hemisphere cortical language regions, the time course of activity was correlated only among participants who heard the same character speaking comprehensibly, despite identical visual input across all participants. For comparison, some individually localized theory of mind regions and right-hemisphere homologues of language regions responded more to dialogue than monologue in the first task, and in the second task, activity in some regions was correlated across all participants regardless of which character was speaking comprehensibly. Together, these results suggest that canonical left-hemisphere cortical language regions are not sensitive to differences between observed dialogue and monologue.

Semantic-specific and domain-general mechanisms for integration and update of contextual information

Recent research has highlighted the importance of domain-general processes and brain regions for language and semantic cognition. Yet, this has been mainly observed in executively demanding tasks, leaving open the question of the contribution of domain-general processes to natural language and semantic cognition. Using fMRI, we investigated whether neural processes reflecting context integration and context update-two key aspects of naturalistic language and semantic processing-are domain-specific versus domain-general. Thus, we compared neural responses during the integration of contextual information across semantic and non-semantic tasks. Whole-brain results revealed both shared (left posterior-dorsal inferior frontal gyrus, left posterior inferior temporal gyrus, and left dorsal angular gyrus/intraparietal sulcus) and distinct (left anterior-ventral inferior frontal gyrus, left anterior ventral angular gyrus, left posterior middle temporal gyrus for semantic control only) regions involved in context integration and update. Furthermore, data-driven functional connectivity analysis clustered domain-specific versus domain-general brain regions into distinct but interacting functional neural networks. These results provide a first characterisation of the neural processes required for context-dependent integration during language processing along the domain-specificity dimension, and at the same time, they bring new insights into the role of left posterior lateral temporal cortex and left angular gyrus for semantic cognition.

Prediction during language comprehension: what is next?

Prediction is often regarded as an integral aspect of incremental language comprehension, but little is known about the cognitive architectures and mechanisms that support it. We review studies showing that listeners and readers use all manner of contextual information to generate multifaceted predictions about upcoming input. The nature of these predictions may vary between individuals owing to differences in language experience, among other factors. We then turn to unresolved questions which may guide the search for the underlying mechanisms. (i) Is prediction essential to language processing or an optional strategy? (ii) Are predictions generated from within the language system or by domain-general processes? (iii) What is the relationship between prediction and memory? (iv) Does prediction in comprehension require simulation via the production system? We discuss promising directions for making progress in answering these questions and for developing a mechanistic understanding of prediction in language.

Driving and suppressing the human language network using large language models

Transformer models such as GPT generate human-like language and are highly predictive of human brain responses to language. Here, using fMRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict the magnitude of brain response associated with each sentence. Then, we use the model to identify new sentences that are predicted to drive or suppress responses in the human language network. We show that these model-selected novel sentences indeed strongly drive and suppress activity of human language areas in new individuals. A systematic analysis of the model-selected sentences reveals that surprisal and well-formedness of linguistic input are key determinants of response strength in the language network. These results establish the ability of neural network models to not only mimic human language but also noninvasively control neural activity in higher-level cortical areas, like the language network.

Modality-Specificity of the Neural Correlates of Linguistic and Non-Linguistic Demand

Imaging studies of language processing in clinical populations can be complicated to interpret for several reasons, one being the difficulty of matching the effortfulness of processing across individuals or tasks. To better understand how effortful linguistic processing is reflected in functional activity, we investigated the neural correlates of task difficulty in linguistic and non-linguistic contexts in the auditory modality and then compared our findings to a recent analogous experiment in the visual modality in a different cohort. Nineteen neurologically normal individuals were scanned with fMRI as they performed a linguistic task (semantic matching) and a non-linguistic task (melodic matching), each with two levels of difficulty. We found that left hemisphere frontal and temporal language regions, as well as the right inferior frontal gyrus, were modulated by linguistic demand and not by non-linguistic demand. This was broadly similar to what was previously observed in the visual modality. In contrast, the multiple demand (MD) network, a set of brain regions thought to support cognitive flexibility in many contexts, was modulated neither by linguistic demand nor by non-linguistic demand in the auditory modality. This finding was in striking contradistinction to what was previously observed in the visual modality, where the MD network was robustly modulated by both linguistic and non-linguistic demand. Our findings suggest that while the language network is modulated by linguistic demand irrespective of modality, modulation of the MD network by linguistic demand is not inherent to linguistic processing, but rather depends on specific task factors.

Neural substrate underlying the learning of a passage with unfamiliar vocabulary and syntax

Speech comprehension is a complex process involving multiple stages, such as decoding of phonetic units, recognizing words, and understanding sentences and passages. In this study, we identify cortical networks beyond basic phonetic processing using a novel passage learning paradigm. Participants learn to comprehend a story composed of syllables of their native language, but containing unfamiliar vocabulary and syntax. Three learning methods are employed, each resulting in some degree of learning within a 12-min learning session. Functional magnetic resonance imaging results reveal that, when listening to the same story, the classic temporal-frontal language network is significantly enhanced by learning. Critically, activation of the left anterior and posterior temporal lobe correlates with the learning outcome that is assessed behaviorally through, e.g. word recognition and passage comprehension tests. This study demonstrates that a brief learning session is sufficient to induce neural plasticity in the left temporal lobe, which underlies the transformation from phonetic units to the units of meaning, such as words and sentences.

Multivariate fMRI responses in superior temporal cortex predict visual contributions to, and individual differences in, the intelligibility of noisy speech

Humans have the unique ability to decode the rapid stream of language elements that constitute speech, even when it is contaminated by noise. Two reliable observations about noisy speech perception are that seeing the face of the talker improves intelligibility and the existence of individual differences in the ability to perceive noisy speech. We introduce a multivariate BOLD fMRI measure that explains both observations. In two independent fMRI studies, clear and noisy speech was presented in visual, auditory and audiovisual formats to thirty-seven participants who rated intelligibility. An event-related design was used to sort noisy speech trials by their intelligibility. Individual-differences multidimensional scaling was applied to fMRI response patterns in superior temporal cortex and the dissimilarity between responses to clear speech and noisy (but intelligible) speech was measured. Neural dissimilarity was less for audiovisual speech than auditory-only speech, corresponding to the greater intelligibility of noisy audiovisual speech. Dissimilarity was less in participants with better noisy speech perception, corresponding to individual differences. These relationships held for both single word and entire sentence stimuli, suggesting that they were driven by intelligibility rather than the specific stimuli tested. A neural measure of perceptual intelligibility may aid in the development of strategies for helping those with impaired speech perception.

The words that little by little revealed everything: Neural response to lexical-semantic content during narrative comprehension

The ease with which narratives are understood belies the complexity of the information being conveyed and the cognitive processes that support comprehension. The meanings of the words must be rapidly accessed and integrated with the reader's mental representation of the overarching, unfolding scenario. A broad, bilateral brain network is engaged by this process, but it is not clear how words that vary on specific semantic dimensions, such as ambiguity, emotion, or socialness, engage the semantic, semantic control, or social cognition systems. In the present study, data from 48 participants who listened to The Little Prince audiobook during MRI scanning were selected from the Le Petit Prince dataset. The lexical and semantic content within the narrative was quantified from the transcript words with factor scores capturing Word Length, Semantic Flexibility, Emotional Strength, and Social Impact. These scores, along with word quantity variables, were used to investigate where these predictors co-vary with activation across the brain. In contrast to studies of isolated word processing, large networks were found to co-vary with the lexical and semantic content within the narrative. An increase in semantic content engaged the ventral portion of ventrolateral ATL, consistent with its role as a semantic hub. Decreased semantic content engaged temporal pole and inferior parietal lobule, which may reflect semantic integration. The semantic control network was engaged by words with low Semantic Flexibility, perhaps due to the demand required to process infrequent, less semantically diverse language. Activation in ATL co-varied with an increase in Social Impact, which is consistent with the claim that social knowledge is housed within the neural architecture of the semantic system. These results suggest that current models of language processing may present an impoverished estimate of the neural systems that coordinate to support narrative comprehension, and, by extension, real-world language processing.

Resting State Network Segregation Modulates Age-Related Differences in Language Production

Older adults typically exhibit decline in language production. However, how the brain supports or fails to support these processes is unclear. Moreover, there are competing hypotheses about the nature of age-related neural changes and whether age-related increases in neural activity reflect compensation or a decline in neural efficiency. In the current study, we investigated the neural bases of language production focusing on resting state functional connectivity. We hypothesized that language production performance, functional connectivity, and their relationship would differ as a function of age. Consistent with prior work, older age was associated with worse language production performance. Functional connectivity analyses showed that network segregation within the left hemisphere language network was maintained across adulthood. However, increased age was associated with lower whole brain network segregation. Moreover, network segregation was related to language production ability. In both network analyses, there were significant interactions with age-higher network segregation was associated with better language production abilities for younger and middle-aged adults, but not for older adults. Interestingly, there was a stronger relationship between language production and the whole brain network segregation than between production and the language network. These results highlight the utility of network segregation measures as an index of brain function, with higher network segregation associated with better language production ability. Moreover, these results are consistent with stability in the left hemisphere language network across adulthood and suggest that dedifferentiation among brain networks, outside of the language network, is a hallmark of aging and may contribute to age-related language production difficulties.

The human language system, including its inferior frontal component in "Broca's area," does not support music perception

Language and music are two human-unique capacities whose relationship remains debated. Some have argued for overlap in processing mechanisms, especially for structure processing. Such claims often concern the inferior frontal component of the language system located within "Broca's area." However, others have failed to find overlap. Using a robust individual-subject fMRI approach, we examined the responses of language brain regions to music stimuli, and probed the musical abilities of individuals with severe aphasia. Across 4 experiments, we obtained a clear answer: music perception does not engage the language system, and judgments about music structure are possible even in the presence of severe damage to the language network. In particular, the language regions' responses to music are generally low, often below the fixation baseline, and never exceed responses elicited by nonmusic auditory conditions, like animal sounds. Furthermore, the language regions are not sensitive to music structure: they show low responses to both intact and structure-scrambled music, and to melodies with vs. without structural violations. Finally, in line with past patient investigations, individuals with aphasia, who cannot judge sentence grammaticality, perform well on melody well-formedness judgments. Thus, the mechanisms that process structure in language do not appear to process music, including music syntax.

Non-literal language processing is jointly supported by the language and theory of mind networks: Evidence from a novel meta-analytic fMRI approach

Going beyond the literal meaning of language is key to communicative success. However, the mechanisms that support non-literal inferences remain debated. Using a novel meta-analytic approach, we evaluate the contribution of linguistic, social-cognitive, and executive mechanisms to non-literal interpretation. We identified 74 fMRI experiments (n = 1,430 participants) from 2001 to 2021 that contrasted non-literal language comprehension with a literal control condition, spanning ten phenomena (e.g., metaphor, irony, indirect speech). Applying the activation likelihood estimation approach to the 825 activation peaks yielded six left-lateralized clusters. We then evaluated the locations of both the individual-study peaks and the clusters against probabilistic functional atlases (cf. anatomical locations, as is typically done) for three candidate brain networks-the language-selective network (Fedorenko, Behr, & Kanwisher, 2011), which supports language processing, the Theory of Mind (ToM) network (Saxe & Kanwisher, 2003), which supports social inferences, and the domain-general Multiple-Demand (MD) network (Duncan, 2010), which supports executive control. These atlases were created by overlaying individual activation maps of participants who performed robust and extensively validated 'localizer' tasks that selectively target each network in question (n = 806 for language; n = 198 for ToM; n = 691 for MD). We found that both the individual-study peaks and the ALE clusters fell primarily within the language network and the ToM network. These results suggest that non-literal processing is supported by both i) mechanisms that process literal linguistic meaning, and ii) mechanisms that support general social inference. They thus undermine a strong divide between literal and non-literal aspects of language and challenge the claim that non-literal processing requires additional executive resources.

Situation updating during discourse comprehension recruits right posterior portion of the multiple-demand network

Discourse comprehension involves the construction of a mental representation of the situation model as well as a continuous update of this representation. This mental update is cognitively demanding and likely engages the multiple-demand network. However, there is little evidence for the involvement of the multiple-demand network during situation updating. In this study, we used fMRI to test whether situation updating based on the change of spatial location activated the multiple-demand network. In a discourse comprehension task, readers read two-sentence discourses in which the second sentence either continues or introduces a shift of the spatial location information presented in the first sentence. Compared to situation continuation, situation updating reliably activated the right superior parietal lobule. This area is a part of the multiple-demand network as defined by a digit N-back localizer task and locates within the dorsal attention network as defined in the previous study by Yeo et al. in 2011. Our results provide evidence for the reliable involvement of a specific area of the multiple-demand network in situation updating during high-level discourse processing.

Timing in Conversation

Turn-taking in everyday conversation is fast, with median latencies in corpora of conversational speech often reported to be under 300 ms. This seems like magic, given that experimental research on speech planning has shown that speakers need much more time to plan and produce even the shortest of utterances. This paper reviews how language scientists have combined linguistic analyses of conversations and experimental work to understand the skill of swift turn-taking and proposes a tentative solution to the riddle of fast turn-taking.

Causal Contributions of the Domain-General (Multiple Demand) and the Language-Selective Brain Networks to Perceptual and Semantic Challenges in Speech Comprehension

Listening to spoken language engages domain-general multiple demand (MD; frontoparietal) regions of the human brain, in addition to domain-selective (frontotemporal) language regions, particularly when comprehension is challenging. However, there is limited evidence that the MD network makes a functional contribution to core aspects of understanding language. In a behavioural study of volunteers (n = 19) with chronic brain lesions, but without aphasia, we assessed the causal role of these networks in perceiving, comprehending, and adapting to spoken sentences made more challenging by acoustic-degradation or lexico-semantic ambiguity. We measured perception of and adaptation to acoustically degraded (noise-vocoded) sentences with a word report task before and after training. Participants with greater damage to MD but not language regions required more vocoder channels to achieve 50% word report, indicating impaired perception. Perception improved following training, reflecting adaptation to acoustic degradation, but adaptation was unrelated to lesion location or extent. Comprehension of spoken sentences with semantically ambiguous words was measured with a sentence coherence judgement task. Accuracy was high and unaffected by lesion location or extent. Adaptation to semantic ambiguity was measured in a subsequent word association task, which showed that availability of lower-frequency meanings of ambiguous words increased following their comprehension (word-meaning priming). Word-meaning priming was reduced for participants with greater damage to language but not MD regions. Language and MD networks make dissociable contributions to challenging speech comprehension: Using recent experience to update word meaning preferences depends on language-selective regions, whereas the domain-general MD network plays a causal role in reporting words from degraded speech.

Explaining neural activity in human listeners with deep learning via natural language processing of narrative text

Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction.

A middle ground where executive control meets semantics: the neural substrates of semantic control are topographically sandwiched between the multiple-demand and default-mode systems

Semantic control is the capability to operate on meaningful representations, selectively focusing on certain aspects of meaning while purposefully ignoring other aspects based on one's behavioral aim. This ability is especially vital for comprehending figurative/ambiguous language. It remains unclear why and how regions involved in semantic control seem reliably juxtaposed alongside other functionally specialized regions in the association cortex, prompting speculation about the relationship between topography and function. We investigated this issue by characterizing how semantic control regions topographically relate to the default-mode network (associated with memory and abstract cognition) and multiple-demand network (associated with executive control). Topographically, we established that semantic control areas were sandwiched by the default-mode and multi-demand networks, forming an orderly arrangement observed both at the individual and group level. Functionally, semantic control regions exhibited "hybrid" responses, fusing generic preferences for cognitively demanding operation (multiple-demand) and for meaningful representations (default-mode) into a domain-specific preference for difficult operation on meaningful representations. When projected onto the principal gradient of human connectome, the neural activity of semantic control showed a robustly dissociable trajectory from visuospatial control, implying different roles in the functional transition from sensation to cognition. We discuss why the hybrid functional profile of semantic control regions might result from their intermediate topographical positions on the cortex.

Probabilistic atlas for the language network based on precision fMRI data from >800 individuals

Two analytic traditions characterize fMRI language research. One relies on averaging activations across individuals. This approach has limitations: because of inter-individual variability in the locations of language areas, any given voxel/vertex in a common brain space is part of the language network in some individuals but in others, may belong to a distinct network. An alternative approach relies on identifying language areas in each individual using a functional 'localizer'. Because of its greater sensitivity, functional resolution, and interpretability, functional localization is gaining popularity, but it is not always feasible, and cannot be applied retroactively to past studies. To bridge these disjoint approaches, we created a probabilistic functional atlas using fMRI data for an extensively validated language localizer in 806 individuals. This atlas enables estimating the probability that any given location in a common space belongs to the language network, and thus can help interpret group-level activation peaks and lesion locations, or select voxels/electrodes for analysis. More meaningful comparisons of findings across studies should increase robustness and replicability in language research.

Dissociable language and executive control deficits and recovery in post-stroke aphasia: An exploratory observational and case series study

A growing body of evidence indicates many, but not all, individuals with post-stroke aphasia experience executive dysfunction. Relationships between language and executive function skills are often reported in the literature, but the degree of interdependence between these abilities remains largely unanswered. Therefore, in this study, we investigated the extent to which language and executive control deficits dissociated in 1) acute stroke and 2) longitudinal aphasia recovery. Twenty-three individuals admitted to Johns Hopkins Hospital with a new left hemisphere stroke completed the Western Aphasia Battery-Revised (WAB-R), several additional language measures (of naming, semantics, spontaneous speech, and oral reading), and three non-linguistic cognitive tasks from the NIH Toolbox (i.e., Pattern Comparison Processing Speed Test, Flanker Inhibitory Control and Attention Test, and Dimensional Change Card Sort Test). Two participants with aphasia (PWA) with temporoparietal lesions, one of whom (PWA1) had greater temporal but less frontal and superior parietal damage than the other (PWA2), also completed testing at subacute (three months post-onset) and early chronic (six months post-onset) time points. In aim 1, principal component analysis on the acute test data (excluding the WAB-R) revealed language and non-linguistic executive control tasks largely loaded onto separate components. Both components were significant predictors of acute aphasia severity per the WAB-R Aphasia Quotient (AQ). Crucially, executive dysfunction explained an additional 17% of the variance in AQ beyond the explanatory power of language impairments alone. In aim 2, both case patients exhibited language and executive control deficits at the acute post-stroke stage. A dissociation was observed in longitudinal recovery of these patients. By the early chronic time point, PWA1 exhibited improved (but persistent) deficits in several language domains and recovered executive control. In contrast, PWA2 demonstrated mostly recovered language but persistent executive dysfunction. Greater damage to language and attention networks in these respective patients may explain the observed behavioral patterns. These results demonstrate that language and executive control can dissociate (at least to a degree), but both contribute to early post-stroke presentation of aphasia and likely influence longitudinal aphasia recovery.

Differential Tracking of Linguistic vs. Mental State Content in Naturalistic Stimuli by Language and Theory of Mind (ToM) Brain Networks

Language and social cognition, especially the ability to reason about mental states, known as theory of mind (ToM), are deeply related in development and everyday use. However, whether these cognitive faculties rely on distinct, overlapping, or the same mechanisms remains debated. Some evidence suggests that, by adulthood, language and ToM draw on largely distinct-though plausibly interacting-cortical networks. However, the broad topography of these networks is similar, and some have emphasized the importance of social content / communicative intent in the linguistic signal for eliciting responses in the language areas. Here, we combine the power of individual-subject functional localization with the naturalistic-cognition inter-subject correlation approach to illuminate the language-ToM relationship. Using functional magnetic resonance imaging (fMRI), we recorded neural activity as participants (n = 43) listened to stories and dialogues with mental state content (+linguistic, +ToM), viewed silent animations and live action films with mental state content but no language (-linguistic, +ToM), or listened to an expository text (+linguistic, -ToM). The ToM network robustly tracked stimuli rich in mental state information regardless of whether mental states were conveyed linguistically or non-linguistically, while tracking a +linguistic / -ToM stimulus only weakly. In contrast, the language network tracked linguistic stimuli more strongly than (a) non-linguistic stimuli, and than (b) the ToM network, and showed reliable tracking even for the linguistic condition devoid of mental state content. These findings suggest that in spite of their indisputably close links, language and ToM dissociate robustly in their neural substrates-and thus plausibly cognitive mechanisms-including during the processing of rich naturalistic materials.

Task-specific network interactions across key cognitive domains

Human cognition is organized in distributed networks in the brain. Although distinct specialized networks have been identified for different cognitive functions, previous work also emphasizes the overlap of key cognitive domains in higher level association areas. The majority of previous studies focused on network overlap and dissociation during resting states whereas task-related network interactions across cognitive domains remain largely unexplored. A better understanding of network overlap and dissociation during different cognitive tasks may elucidate flexible (re-)distribution of resources during human cognition. The present study addresses this issue by providing a broad characterization of large-scale network dynamics in three key cognitive domains. Combining prototypical tasks of the larger domains of attention, language, and social cognition with whole-brain multivariate activity and connectivity approaches, we provide a spatiotemporal characterization of multiple large-scale, overlapping networks that differentially interact across cognitive domains. We show that network activity and interactions increase with increased cognitive complexity across domains. Interaction patterns reveal a common core structure across domains as well as dissociable domain-specific network activity. The observed patterns of activation and deactivation of overlapping and strongly coupled networks provide insight beyond region-specific activity within a particular cognitive domain toward a network perspective approach across diverse key cognitive functions.

Similarity of computations across domains does not imply shared implementation: The case of language comprehension

Understanding language requires applying cognitive operations (e.g., memory retrieval, prediction, structure building)-relevant across many cognitive domains-to specialized knowledge structures (a particular language's phonology, lexicon, and syntax). Are these computations carried out by domain-general circuits or by circuits that store domain-specific representations? Recent work has characterized the roles in language comprehension of the language-selective network and the multiple demand (MD) network, which has been implicated in executive functions and linked to fluid intelligence, making it a prime candidate for implementing computations that support information processing across domains. The language network responds robustly to diverse aspects of comprehension, but the MD network shows no sensitivity to linguistic variables. We therefore argue that the MD network does not play a core role in language comprehension, and that past claims to the contrary are likely due to methodological artifacts. Although future studies may discover some aspects of language that require the MD network, evidence to date suggests that those will not be related to core linguistic processes like lexical access or composition. The finding that the circuits that store linguistic knowledge carry out computations on those representations aligns with general arguments against the separation between memory and computation in the mind and brain.

The neural architecture of language: Integrative modeling converges on predictive processing

The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural mechanisms in the target domain. We here present a systematic study taking this approach to higher-level cognition: human language processing, our species' signature cognitive skill. We find that the most powerful "transformer" models predict nearly 100% of explainable variance in neural responses to sentences and generalize across different datasets and imaging modalities (functional MRI and electrocorticography). Models' neural fits ("brain score") and fits to behavioral responses are both strongly correlated with model accuracy on the next-word prediction task (but not other language tasks). Model architecture appears to substantially contribute to neural fit. These results provide computationally explicit evidence that predictive processing fundamentally shapes the language comprehension mechanisms in the human brain.

Engagement of Language and Domain General Networks during Word Monitoring in a Native and Unknown Language

Functional neuroimaging studies have highlighted the roles of three networks in processing language, all of which are typically left-lateralized: a ventral stream involved in semantics, a dorsal stream involved in phonology and speech production, and a more dorsal "multiple demand" network involved in many effortful tasks. As lateralization in all networks may be affected by life factors such as age, literacy, education, and brain pathology, we sought to develop a task paradigm with which to investigate the engagement of these networks, including manipulations to selectively emphasize semantic and phonological processing within a single task performable by almost anyone regardless of literacy status. In young healthy participants, we administered an auditory word monitoring task, in which participants had to note the occurrence of a target word within a continuous story presented in either their native language, Portuguese, or the unknown language, Japanese. Native language task performance activated ventral stream language networks, left lateralized but bilateral in the anterior temporal lobe. Unfamiliar language performance, being more difficult, activated left hemisphere dorsal stream structures and the multiple demand network bilaterally, but predominantly in the right hemisphere. These findings suggest that increased demands on phonological processing to accomplish word monitoring in the absence of semantic support may result in the bilateral recruitment of networks involved in speech perception under more challenging conditions.

Continuous-time deconvolutional regression for psycholinguistic modeling

The influence of stimuli in psycholinguistic experiments diffuses across time because the human response to language is not instantaneous. The linear models typically used to analyze psycholinguistic data are unable to account for this phenomenon due to strong temporal independence assumptions, while existing deconvolutional methods for estimating diffuse temporal structure model time discretely and therefore cannot be directly applied to natural language stimuli where events (words) have variable duration. In light of evidence that continuous-time deconvolutional regression (CDR) can address these issues (Shain & Schuler, 2018), this article motivates the use of CDR for many experimental settings, exposits some of its mathematical properties, and empirically evaluates the influence of various experimental confounds (noise, multicollinearity, and impulse response misspecification), hyperparameter settings, and response types (behavioral and fMRI). Results show that CDR (1) yields highly consistent estimates across a variety of hyperparameter configurations, (2) faithfully recovers the data-generating model on synthetic data, even under adverse training conditions, and (3) outperforms widely-used statistical approaches when applied to naturalistic reading and fMRI data. In addition, procedures for testing scientific hypotheses using CDR are defined and demonstrated, and empirically-motivated best-practices for CDR modeling are proposed. Results support the use of CDR for analyzing psycholinguistic time series, especially in a naturalistic experimental paradigm.