1
|
Nourski KV, Steinschneider M, Rhone AE, Dappen ER, Kawasaki H, Howard MA. Processing of auditory novelty in human cortex during a semantic categorization task. Hear Res 2024; 444:108972. [PMID: 38359485 PMCID: PMC10984345 DOI: 10.1016/j.heares.2024.108972] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/18/2024] [Revised: 02/05/2024] [Accepted: 02/10/2024] [Indexed: 02/17/2024]
Abstract
Auditory semantic novelty - a new meaningful sound in the context of a predictable acoustical environment - can probe neural circuits involved in language processing. Aberrant novelty detection is a feature of many neuropsychiatric disorders. This large-scale human intracranial electrophysiology study examined the spatial distribution of gamma and alpha power and auditory evoked potentials (AEP) associated with responses to unexpected words during performance of semantic categorization tasks. Participants were neurosurgical patients undergoing monitoring for medically intractable epilepsy. Each task included repeatedly presented monosyllabic words from different talkers ("common") and ten words presented only once ("novel"). Targets were words belonging to a specific semantic category. Novelty effects were defined as differences between neural responses to novel and common words. Novelty increased task difficulty and was associated with augmented gamma, suppressed alpha power, and AEP differences broadly distributed across the cortex. Gamma novelty effect had the highest prevalence in planum temporale, posterior superior temporal gyrus (STG) and pars triangularis of the inferior frontal gyrus; alpha in anterolateral Heschl's gyrus (HG), anterior STG and middle anterior cingulate cortex; AEP in posteromedial HG, lower bank of the superior temporal sulcus, and planum polare. Gamma novelty effect had a higher prevalence in dorsal than ventral auditory-related areas. Novelty effects were more pronounced in the left hemisphere. Better novel target detection was associated with reduced gamma novelty effect within auditory cortex and enhanced gamma effect within prefrontal and sensorimotor cortex. Alpha and AEP novelty effects were generally more prevalent in better performing participants. Multiple areas, including auditory cortex on the superior temporal plane, featured AEP novelty effect within the time frame of P3a and N400 scalp-recorded novelty-related potentials. This work provides a detailed account of auditory novelty in a paradigm that directly examined brain regions associated with semantic processing. Future studies may aid in the development of objective measures to assess the integrity of semantic novelty processing in clinical populations.
Collapse
Affiliation(s)
- Kirill V Nourski
- Department of Neurosurgery, The University of Iowa, Iowa City, IA 52242, United States; Iowa Neuroscience Institute, The University of Iowa, Iowa City, IA 52242, United States.
| | - Mitchell Steinschneider
- Department of Neurosurgery, The University of Iowa, Iowa City, IA 52242, United States; Departments of Neurology, Neuroscience, and Pediatrics, Albert Einstein College of Medicine, Bronx, NY 10461, United States
| | - Ariane E Rhone
- Department of Neurosurgery, The University of Iowa, Iowa City, IA 52242, United States
| | - Emily R Dappen
- Department of Neurosurgery, The University of Iowa, Iowa City, IA 52242, United States; Iowa Neuroscience Institute, The University of Iowa, Iowa City, IA 52242, United States
| | - Hiroto Kawasaki
- Department of Neurosurgery, The University of Iowa, Iowa City, IA 52242, United States
| | - Matthew A Howard
- Department of Neurosurgery, The University of Iowa, Iowa City, IA 52242, United States; Iowa Neuroscience Institute, The University of Iowa, Iowa City, IA 52242, United States; Pappajohn Biomedical Institute, The University of Iowa, Iowa City, IA 52242, United States
| |
Collapse
|
2
|
Anderson EJ, Midgley KJ, Holcomb PJ, Riès SK. Taxonomic and thematic semantic relationships in picture naming as revealed by Laplacian-transformed event-related potentials. Psychophysiology 2022; 59:e14091. [PMID: 35554943 PMCID: PMC9788343 DOI: 10.1111/psyp.14091] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2021] [Revised: 03/30/2022] [Accepted: 04/20/2022] [Indexed: 12/31/2022]
Abstract
Semantically related concepts co-activate when we speak. Prior research reported both behavioral interference and facilitation due to co-activation during picture naming. Different word relationships may account for some of this discrepancy. Taxonomically related words (e.g., WOLF-DOG) have been associated with semantic interference; thematically related words (e.g., BONE-DOG) have been associated with facilitation. Although these different semantic relationships have been associated with opposite behavioral outcomes, electrophysiological studies have found inconsistent effects on event-related potentials. We conducted a picture-word interference electroencephalography experiment to examine word retrieval dynamics in these different semantic relationships. Importantly, we used traditional monopolar analysis as well as Laplacian transformation allowing us to examine spatially deblurred event-related components. Both analyses revealed greater negativity (150-250 ms) for unrelated than related taxonomic pairs, though more restricted in space for thematic pairs. Critically, Laplacian analyses revealed a larger negative-going component in the 300 to 500 ms time window in taxonomically related versus unrelated pairs which were restricted to a left frontal recording site. In parallel, an opposite effect was found in the same time window but localized to a left parietal site. Finding these opposite effects in the same time window was feasible thanks to the use of the Laplacian transformation and suggests that frontal control processes are concurrently engaged with cascading effects of the spread of activation through semantically related representations.
Collapse
Affiliation(s)
- Elizabeth J. Anderson
- Joint Doctoral Program in Language and Communicative DisordersSan Diego State UniversitySan DiegoCaliforniaUSA,Joint Doctoral Program in Language and Communicative DisordersUniversity of California San DiegoLa JollaCaliforniaUSA
| | | | - Phillip J. Holcomb
- Department of PsychologySan Diego State UniversitySan DiegoCaliforniaUSA
| | - Stephanie K. Riès
- School of Speech, Language, and Hearing SciencesSan Diego State UniversitySan DiegoCaliforniaUSA
| |
Collapse
|
3
|
Schmitt LM, Erb J, Tune S, Rysop AU, Hartwigsen G, Obleser J. Predicting speech from a cortical hierarchy of event-based time scales. SCIENCE ADVANCES 2021. [PMID: 34860554 DOI: 10.1101/2020.12.19.423616] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/05/2023]
Abstract
How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse.
Collapse
Affiliation(s)
- Lea-Maria Schmitt
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Julia Erb
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Sarah Tune
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Anna U Rysop
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Stephanstraße 1 A, 04103 Leipzig, Germany
| | - Gesa Hartwigsen
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Stephanstraße 1 A, 04103 Leipzig, Germany
| | - Jonas Obleser
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| |
Collapse
|
4
|
Schmitt LM, Erb J, Tune S, Rysop AU, Hartwigsen G, Obleser J. Predicting speech from a cortical hierarchy of event-based time scales. SCIENCE ADVANCES 2021; 7:eabi6070. [PMID: 34860554 PMCID: PMC8641937 DOI: 10.1126/sciadv.abi6070] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/19/2021] [Accepted: 10/15/2021] [Indexed: 05/30/2023]
Abstract
How do predictions in the brain incorporate the temporal unfolding of context in our natural environment? We here provide evidence for a neural coding scheme that sparsely updates contextual representations at the boundary of events. This yields a hierarchical, multilayered organization of predictive language comprehension. Training artificial neural networks to predict the next word in a story at five stacked time scales and then using model-based functional magnetic resonance imaging, we observe an event-based “surprisal hierarchy” evolving along a temporoparietal pathway. Along this hierarchy, surprisal at any given time scale gated bottom-up and top-down connectivity to neighboring time scales. In contrast, surprisal derived from continuously updated context influenced temporoparietal activity only at short time scales. Representing context in the form of increasingly coarse events constitutes a network architecture for making predictions that is both computationally efficient and contextually diverse.
Collapse
Affiliation(s)
- Lea-Maria Schmitt
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Julia Erb
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Sarah Tune
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Anna U. Rysop
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Stephanstraße 1 A, 04103 Leipzig, Germany
| | - Gesa Hartwigsen
- Lise Meitner Research Group Cognition and Plasticity, Max Planck Institute for Human Cognitive and Brain Sciences, Stephanstraße 1 A, 04103 Leipzig, Germany
| | - Jonas Obleser
- Department of Psychology, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
- Center of Brain, Behavior and Metabolism, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| |
Collapse
|
5
|
Pawełczyk A, Łojek E, Żurner N, Gawłowska-Sawosz M, Gębski P, Pawełczyk T. The correlation between white matter integrity and pragmatic language processing in first episode schizophrenia. Brain Imaging Behav 2021; 15:1068-1084. [PMID: 32710335 PMCID: PMC8032571 DOI: 10.1007/s11682-020-00314-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/13/2023]
Abstract
Objective: Higher-order language disturbances could be the result of white matter tract abnormalities. The study explores the relationship between white matter and pragmatic skills in first-episode schizophrenia. Methods: Thirty-four first-episode patients with schizophrenia and 32 healthy subjects participated in a pragmatic language and Diffusion Tensor Imaging study, where fractional anisotropy of the arcuate fasciculus, corpus callosum and cingulum was correlated with the Polish version of the Right Hemisphere Language Battery. Results: The patients showed reduced fractional anisotropy in the right arcuate fasciculus, left anterior cingulum bundle and left forceps minor. Among the first episode patients, reduced understanding of written metaphors correlated with reduced fractional anisotropy of left forceps minor, and greater explanation of written and picture metaphors correlated with reduced fractional anisotropy of the left anterior cingulum. Conclusions: The white matter dysfunctions may underlie the pragmatic language impairment in schizophrenia. Our results shed further light on the functional neuroanatomical basis of pragmatic language use by patients with schizophrenia.
Collapse
Affiliation(s)
- Agnieszka Pawełczyk
- Department of Affective and Psychotic Disorders, Medical University of Łódź, Łódź, Poland.
| | | | - Natalia Żurner
- Adolescent Ward, Central Clinical Hospital of Medical University of Łódź, Łódź, Poland
| | | | - Piotr Gębski
- Scanlab Diagnostyka Medyczna Księży Młyn, Medical Examination Centre, Medical University of Łódź, Łódź, Poland
| | - Tomasz Pawełczyk
- Department of Affective and Psychotic Disorders, Medical University of Łódź, Łódź, Poland
| |
Collapse
|
6
|
Lee CS, Aly M, Baldassano C. Anticipation of temporally structured events in the brain. eLife 2021; 10:e64972. [PMID: 33884953 PMCID: PMC8169103 DOI: 10.7554/elife.64972] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2020] [Accepted: 04/21/2021] [Indexed: 01/17/2023] Open
Abstract
Learning about temporal structure is adaptive because it enables the generation of expectations. We examined how the brain uses experience in structured environments to anticipate upcoming events. During fMRI (functional magnetic resonance imaging), individuals watched a 90 s movie clip six times. Using a hidden Markov model applied to searchlights across the whole brain, we identified temporal shifts between activity patterns evoked by the first vs. repeated viewings of the movie clip. In many regions throughout the cortex, neural activity patterns for repeated viewings shifted to precede those of initial viewing by up to 15 s. This anticipation varied hierarchically in a posterior (less anticipation) to anterior (more anticipation) fashion. We also identified specific regions in which the timing of the brain's event boundaries was related to those of human-labeled event boundaries, with the timing of this relationship shifting on repeated viewings. With repeated viewing, the brain's event boundaries came to precede human-annotated boundaries by 1-4 s on average. Together, these results demonstrate a hierarchy of anticipatory signals in the human brain and link them to subjective experiences of events.
Collapse
Affiliation(s)
- Caroline S Lee
- Columbia University, Department of PsychologyNew YorkUnited States
- Dartmouth College, Department of Psychological and Brain SciencesHanoverUnited States
| | - Mariam Aly
- Columbia University, Department of PsychologyNew YorkUnited States
| | | |
Collapse
|
7
|
Movies and narratives as naturalistic stimuli in neuroimaging. Neuroimage 2020; 224:117445. [PMID: 33059053 PMCID: PMC7805386 DOI: 10.1016/j.neuroimage.2020.117445] [Citation(s) in RCA: 54] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2020] [Revised: 10/06/2020] [Accepted: 10/09/2020] [Indexed: 01/06/2023] Open
Abstract
Using movies and narratives as naturalistic stimuli in human neuroimaging studies has yielded significant advances in understanding of cognitive and emotional functions. The relevant literature was reviewed, with emphasis on how the use of naturalistic stimuli has helped advance scientific understanding of human memory, attention, language, emotions, and social cognition in ways that would have been difficult otherwise. These advances include discovering a cortical hierarchy of temporal receptive windows, which supports processing of dynamic information that accumulates over several time scales, such as immediate reactions vs. slowly emerging patterns in social interactions. Naturalistic stimuli have also helped elucidate how the hippocampus supports segmentation and memorization of events in day-to-day life and have afforded insights into attentional brain mechanisms underlying our ability to adopt specific perspectives during natural viewing. Further, neuroimaging studies with naturalistic stimuli have revealed the role of the default-mode network in narrative-processing and in social cognition. Finally, by robustly eliciting genuine emotions, these stimuli have helped elucidate the brain basis of both basic and social emotions apparently manifested as highly overlapping yet distinguishable patterns of brain activity.
Collapse
|
8
|
Blank IA, Fedorenko E. No evidence for differences among language regions in their temporal receptive windows. Neuroimage 2020; 219:116925. [PMID: 32407994 DOI: 10.1016/j.neuroimage.2020.116925] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2019] [Revised: 03/20/2020] [Accepted: 05/06/2020] [Indexed: 10/24/2022] Open
Abstract
The "core language network" consists of left frontal and temporal regions that are selectively engaged in linguistic processing. Whereas functional differences among these regions have long been debated, many accounts propose distinctions in terms of representational grain-size-e.g., words vs. phrases/sentences-or processing time-scale, i.e., operating on local linguistic features vs. larger spans of input. Indeed, the topography of language regions appears to overlap with a cortical hierarchy reported by Lerner et al. (2011) wherein mid-posterior temporal regions are sensitive to low-level features of speech, surrounding areas-to word-level information, and inferior frontal areas-to sentence-level information and beyond. However, the correspondence between the language network and this hierarchy of "temporal receptive windows" (TRWs) is difficult to establish because the precise anatomical locations of language regions vary across individuals. To directly test this correspondence, we first identified language regions in each participant with a well-validated task-based localizer, which confers high functional resolution to the study of TRWs (traditionally based on stereotactic coordinates); then, we characterized regional TRWs with the naturalistic story listening paradigm of Lerner et al. (2011), which augments task-based characterizations of the language network by more closely resembling comprehension "in the wild". We find no region-by-TRW interactions across temporal and inferior frontal regions, which are all sensitive to both word-level and sentence-level information. Therefore, the language network as a whole constitutes a unique stage of information integration within a broader cortical hierarchy.
Collapse
Affiliation(s)
- Idan A Blank
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA.
| | - Evelina Fedorenko
- Department of Brain and Cognitive Sciences and McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| |
Collapse
|
9
|
Honbolygó F, Kóbor A, Hermann P, Kettinger ÁO, Vidnyánszky Z, Kovács G, Csépe V. Expectations about word stress modulate neural activity in speech-sensitive cortical areas. Neuropsychologia 2020; 143:107467. [PMID: 32305299 DOI: 10.1016/j.neuropsychologia.2020.107467] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2019] [Revised: 03/06/2020] [Accepted: 04/12/2020] [Indexed: 10/24/2022]
Abstract
A recent dual-stream model of language processing proposed that the postero-dorsal stream performs predictive sequential processing of linguistic information via hierarchically organized internal models. However, it remains unexplored whether the prosodic segmentation of linguistic information involves predictive processes. Here, we addressed this question by investigating the processing of word stress, a major component of speech segmentation, using probabilistic repetition suppression (RS) modulation as a marker of predictive processing. In an event-related acoustic fMRI RS paradigm, we presented pairs of pseudowords having the same (Rep) or different (Alt) stress patterns, in blocks with varying Rep and Alt trial probabilities. We found that the BOLD signal was significantly lower for Rep than for Alt trials, indicating RS in the posterior and middle superior temporal gyrus (STG) bilaterally, and in the anterior STG in the left hemisphere. Importantly, the magnitude of RS was modulated by repetition probability in the posterior and middle STG. These results reveal the predictive processing of word stress in the STG areas and raise the possibility that words stress processing is related to the dorsal "where" auditory stream.
Collapse
Affiliation(s)
- Ferenc Honbolygó
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Institute of Psychology, Eötvös Loránd University, Budapest, Hungary.
| | - Andrea Kóbor
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary
| | - Petra Hermann
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary
| | - Ádám Ottó Kettinger
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Department of Nuclear Techniques, Budapest University of Technology and Economics, Budapest, Hungary
| | - Zoltán Vidnyánszky
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary
| | - Gyula Kovács
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Department of Biological Psychology and Cognitive Neuroscience, Institute of Psychology, Friedrich Schiller University Jena, Jena, Germany
| | - Valéria Csépe
- Brain Imaging Centre, Research Centre for Natural Sciences, Budapest, Hungary; Faculty of Modern Philology and Social Sciences, University of Pannonia, Veszprém, Hungary
| |
Collapse
|
10
|
Bornkessel-Schlesewsky I, Schlesewsky M. Toward a Neurobiologically Plausible Model of Language-Related, Negative Event-Related Potentials. Front Psychol 2019; 10:298. [PMID: 30846950 PMCID: PMC6393377 DOI: 10.3389/fpsyg.2019.00298] [Citation(s) in RCA: 64] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2018] [Accepted: 01/30/2019] [Indexed: 11/13/2022] Open
Abstract
Language-related event-related potential (ERP) components such as the N400 have traditionally been associated with linguistic or cognitive functional interpretations. By contrast, it has been considerably more difficult to relate these components to neurobiologically grounded accounts of language. Here, we propose a theoretical framework based on a predictive coding architecture, within which negative language-related ERP components such as the N400 can be accounted for in a neurobiologically plausible manner. Specifically, we posit that the amplitude of negative language-related ERP components reflects precision-weighted prediction error signals, i.e., prediction errors weighted by the relevance of the information source leading to the error. From this perspective, precision has a direct link to cue validity in a particular language and, thereby, to relevance of individual linguistic features for internal model updating. We view components such as the N400 and LAN as members of a family with similar functional characteristics and suggest that latency and topography differences between these components reflect the locus of prediction errors and model updating within a hierarchically organized cortical predictive coding architecture. This account has the potential to unify findings from the full range of the N400 literature, including word-level, sentence-, and discourse-level results as well as cross-linguistic differences.
Collapse
Affiliation(s)
- Ina Bornkessel-Schlesewsky
- Centre for Cognitive and Systems Neuroscience, University of South Australia, Adelaide, SA, Australia
- School of Psychology, Social Work and Social Policy, University of South Australia, Adelaide, SA, Australia
| | - Matthias Schlesewsky
- Centre for Cognitive and Systems Neuroscience, University of South Australia, Adelaide, SA, Australia
- School of Psychology, Social Work and Social Policy, University of South Australia, Adelaide, SA, Australia
| |
Collapse
|
11
|
Hamilton LS, Huth AG. The revolution will not be controlled: natural stimuli in speech neuroscience. LANGUAGE, COGNITION AND NEUROSCIENCE 2018; 35:573-582. [PMID: 32656294 PMCID: PMC7324135 DOI: 10.1080/23273798.2018.1499946] [Citation(s) in RCA: 93] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Accepted: 07/03/2018] [Indexed: 05/22/2023]
Abstract
Humans have a unique ability to produce and consume rich, complex, and varied language in order to communicate ideas to one another. Still, outside of natural reading, the most common methods for studying how our brains process speech or understand language use only isolated words or simple sentences. Recent studies have upset this status quo by employing complex natural stimuli and measuring how the brain responds to language as it is used. In this article we argue that natural stimuli offer many advantages over simplified, controlled stimuli for studying how language is processed by the brain. Furthermore, the downsides of using natural language stimuli can be mitigated using modern statistical and computational techniques.
Collapse
Affiliation(s)
- Liberty S. Hamilton
- Communication Sciences & Disorders, Moody College of Communication, The University of Texas at Austin, Austin, USA
- Department of Neurology, Dell Medical School, The University of Texas at Austin, Austin, USA
| | - Alexander G. Huth
- Department of Neuroscience, The University of Texas at Austin, Austin, USA
- Department of Computer Science, The University of Texas at Austin, Austin, USA
| |
Collapse
|
12
|
Giordano BL, Ince RAA, Gross J, Schyns PG, Panzeri S, Kayser C. Contributions of local speech encoding and functional connectivity to audio-visual speech perception. eLife 2017; 6. [PMID: 28590903 PMCID: PMC5462535 DOI: 10.7554/elife.24763] [Citation(s) in RCA: 43] [Impact Index Per Article: 6.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2016] [Accepted: 05/07/2017] [Indexed: 11/13/2022] Open
Abstract
Seeing a speaker’s face enhances speech intelligibility in adverse environments. We investigated the underlying network mechanisms by quantifying local speech representations and directed connectivity in MEG data obtained while human participants listened to speech of varying acoustic SNR and visual context. During high acoustic SNR speech encoding by temporally entrained brain activity was strong in temporal and inferior frontal cortex, while during low SNR strong entrainment emerged in premotor and superior frontal cortex. These changes in local encoding were accompanied by changes in directed connectivity along the ventral stream and the auditory-premotor axis. Importantly, the behavioral benefit arising from seeing the speaker’s face was not predicted by changes in local encoding but rather by enhanced functional connectivity between temporal and inferior frontal cortex. Our results demonstrate a role of auditory-frontal interactions in visual speech representations and suggest that functional connectivity along the ventral pathway facilitates speech comprehension in multisensory environments. DOI:http://dx.doi.org/10.7554/eLife.24763.001 When listening to someone in a noisy environment, such as a cocktail party, we can understand the speaker more easily if we can also see his or her face. Movements of the lips and tongue convey additional information that helps the listener’s brain separate out syllables, words and sentences. However, exactly where in the brain this effect occurs and how it works remain unclear. To find out, Giordano et al. scanned the brains of healthy volunteers as they watched clips of people speaking. The clarity of the speech varied between clips. Furthermore, in some of the clips the lip movements of the speaker corresponded to the speech in question, whereas in others the lip movements were nonsense babble. As expected, the volunteers performed better on a word recognition task when the speech was clear and when the lips movements agreed with the spoken dialogue. Watching the video clips stimulated rhythmic activity in multiple regions of the volunteers’ brains, including areas that process sound and areas that plan movements. Speech is itself rhythmic, and the volunteers’ brain activity synchronized with the rhythms of the speech they were listening to. Seeing the speaker’s face increased this degree of synchrony. However, it also made it easier for sound-processing regions within the listeners’ brains to transfer information to one other. Notably, only the latter effect predicted improved performance on the word recognition task. This suggests that seeing a person’s face makes it easier to understand his or her speech by boosting communication between brain regions, rather than through effects on individual areas. Further work is required to determine where and how the brain encodes lip movements and speech sounds. The next challenge will be to identify where these two sets of information interact, and how the brain merges them together to generate the impression of specific words. DOI:http://dx.doi.org/10.7554/eLife.24763.002
Collapse
Affiliation(s)
- Bruno L Giordano
- Institut de Neurosciences de la Timone UMR 7289, Aix Marseille Université - Centre National de la Recherche Scientifique, Marseille, France.,Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Robin A A Ince
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Joachim Gross
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Philippe G Schyns
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Stefano Panzeri
- Neural Computation Laboratory, Center for Neuroscience and Cognitive Systems, Istituto Italiano di Tecnologia, Rovereto, Italy
| | - Christoph Kayser
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| |
Collapse
|
13
|
Kandylaki KD, Henrich K, Nagels A, Kircher T, Domahs U, Schlesewsky M, Bornkessel-Schlesewsky I, Wiese R. Where Is the Beat? The Neural Correlates of Lexical Stress and Rhythmical Well-formedness in Auditory Story Comprehension. J Cogn Neurosci 2017; 29:1119-1131. [PMID: 28294714 DOI: 10.1162/jocn_a_01122] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
While listening to continuous speech, humans process beat information to correctly identify word boundaries. The beats of language are stress patterns that are created by combining lexical (word-specific) stress patterns and the rhythm of a specific language. Sometimes, the lexical stress pattern needs to be altered to obey the rhythm of the language. This study investigated the interplay of lexical stress patterns and rhythmical well-formedness in natural speech with fMRI. Previous electrophysiological studies on cases in which a regular lexical stress pattern may be altered to obtain rhythmical well-formedness showed that even subtle rhythmic deviations are detected by the brain if attention is directed toward prosody. Here, we present a new approach to this phenomenon by having participants listen to contextually rich stories in the absence of a task targeting the manipulation. For the interaction of lexical stress and rhythmical well-formedness, we found one suprathreshold cluster localized between the cerebellum and the brain stem. For the main effect of lexical stress, we found higher BOLD responses to the retained lexical stress pattern in the bilateral SMA, bilateral postcentral gyrus, bilateral middle fontal gyrus, bilateral inferior and right superior parietal lobule, and right precuneus. These results support the view that lexical stress is processed as part of a sensorimotor network of speech comprehension. Moreover, our results connect beat processing in language to domain-independent timing perception.
Collapse
Affiliation(s)
| | - Karen Henrich
- 2 Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany
| | | | | | - Ulrike Domahs
- 4 Free University of Bozen-Bolzano, Brixen-Bressanone, Italy
| | | | | | | |
Collapse
|