1
|
Zhao Y, Chen Y, Cheng K, Huang W. Artificial intelligence based multimodal language decoding from brain activity: A review. Brain Res Bull 2023; 201:110713. [PMID: 37487829 DOI: 10.1016/j.brainresbull.2023.110713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 06/26/2023] [Accepted: 07/20/2023] [Indexed: 07/26/2023]
Abstract
Decoding brain activity is conducive to the breakthrough of brain-computer interface (BCI) technology. The development of artificial intelligence (AI) continually promotes the progress of brain language decoding technology. Existent research has mainly focused on a single modality and paid insufficient attention to AI methods. Therefore, our objective is to provide an overview of relevant decoding research from the perspective of different modalities and methodologies. The modalities involve text, speech, image, and video, whereas the core method is using AI-built decoders to translate brain signals induced by multimodal stimuli into text or vocal language. The semantic information of brain activity can be successfully decoded into a language at various levels, ranging from words through sentences to discourses. However, the decoding effect is affected by various factors, such as the decoding model, vector representation model, and brain regions. Challenges and future directions are also discussed. The advances in brain language decoding and BCI technology will potentially assist patients with clinical aphasia in regaining the ability to communicate.
Collapse
Affiliation(s)
- Yuhao Zhao
- College of Language Intelligence, Sichuan International Studies University, Chongqing 400031, PR China
| | - Yu Chen
- Technical College for the Deaf, Tianjin University of Technology, Tianjin 300384, PR China
| | - Kaiwen Cheng
- College of Language Intelligence, Sichuan International Studies University, Chongqing 400031, PR China.
| | - Wei Huang
- Sichuan Provincial Key Laboratory for Human Disease Gene Study, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu 611731, PR China.
| |
Collapse
|
2
|
Zhang J, Li H, Zhang M, Wang Z, Ao X, Jian J, Wei N, Liu H, Ding G, Meng X. Functional preference of the left inferior parietal lobule to second language reading. Neuroimage 2023; 270:119989. [PMID: 36858331 DOI: 10.1016/j.neuroimage.2023.119989] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2022] [Revised: 02/20/2023] [Accepted: 02/25/2023] [Indexed: 03/02/2023] Open
Abstract
Additional neural substance for reading in a second language has been reported by prior studies. However, to date, there has been little investigation into whether and how the brain's adaptation to a second language is induced by specific linguistic tasks or is a general effect during reading in a new language. To address this issue, our study investigated Chinese children learning English as a second language by combining cross-sectional and longitudinal Functional Magnetic Resonance Imaging (fMRI) studies. We compared brain activation across four reading tasks, orthographic tasks and phonological tasks in Chinese (the first language, L1) and English (the second language, L2). By comparing the activation pattern across languages, we observed greater activation in the left inferior parietal lobule (LIPL) in English compared to Chinese, suggesting a functional preference of the LIPL to L2. In addition, greater correlation between LIPL-related FC and L2 was mainly observed in the phonological task, indicating that LIPL could be associated with phonological processing. Moreover, a proportion of the children were enrolled in an 8-week phonological-based reading-training program. We observed significant functional plasticity of the LIPL elicited by this training program only in the English phonological task and not in the orthographic task, further substantiating that the additional requirements of the LIPL in L2 are mainly associated with phonological processing. The findings provide new insights into understanding the functional contribution of the LIPL to reading in a second language.
Collapse
Affiliation(s)
- Jia Zhang
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, China
| | - Hehui Li
- Center for Brain Disorders and Cognitive Sciences, School of Psychology, Shenzhen University, Shenzhen 518060, China
| | - Manli Zhang
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavioral and Mental Health, Peking University, Beijing 100871, China; Human Communication, Development, and Information Sciences, Faculty of Education, The University of Hong Kong, Hong Kong, China
| | - Zhengke Wang
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavioral and Mental Health, Peking University, Beijing 100871, China
| | - Xiya Ao
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavioral and Mental Health, Peking University, Beijing 100871, China
| | - Jie Jian
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavioral and Mental Health, Peking University, Beijing 100871, China
| | - Na Wei
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, China
| | - Haiyi Liu
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, China
| | - Guosheng Ding
- State Key Laboratory of Cognitive Neuroscience and Learning and IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing 100875, China
| | - Xiangzhi Meng
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavioral and Mental Health, Peking University, Beijing 100871, China; PekingU-PolyU Center for Child Development and Learning, Peking University, Beijing 100871, China.
| |
Collapse
|
3
|
Gibbs SK, Fulton S, Mudigoudar B, Boop FA, Narayana S. Presurgical language mapping in bilingual children using transcranial magnetic stimulation: illustrative case. JOURNAL OF NEUROSURGERY: CASE LESSONS 2021; 2:CASE21391. [PMID: 36131569 PMCID: PMC9563954 DOI: 10.3171/case21391] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Accepted: 07/26/2021] [Indexed: 12/02/2022]
Abstract
BACKGROUND Presurgical mapping of eloquent cortex in young patients undergoing neurosurgery is critical but presents challenges unique to the pediatric population, including motion artifact, noncompliance, and sedation requirements. Furthermore, as bilingualism in children increases, functional mapping of more than one language is becoming increasingly critical. Transcranial magnetic stimulation (TMS), a noninvasive brain stimulation technique, is well suited to evaluate language areas in children since it does not require the patient to remain still during mapping. OBSERVATIONS A 13-year-old bilingual male with glioblastoma multiforme involving the left parietal lobe and deep occipital white matter underwent preoperative language mapping using magnetic resonance imaging-guided TMS. Language-specific cortices were successfully identified in both hemispheres. TMS findings aided in discussing with the family the risks of postsurgical deficits of tumor resection; postoperatively, the patient had intact bilingual speech and was referred for chemotherapy and radiation. LESSONS The authors’ findings add to the evolving case for preoperative dual language mapping in bilingual neurosurgical candidates. The authors illustrate the feasibility and utility of TMS as a noninvasive functional mapping tool in this child. TMS is safe, effective, and can be used for preoperative mapping of language cortex in bilingual children to aid in surgical planning and discussion with families.
Collapse
Affiliation(s)
- Savannah K. Gibbs
- Neuroscience Institute, Le Bonheur Children’s Hospital, Memphis, Tennessee
| | - Stephen Fulton
- Neuroscience Institute, Le Bonheur Children’s Hospital, Memphis, Tennessee
- Departments of Pediatrics
| | - Basanagoud Mudigoudar
- Neuroscience Institute, Le Bonheur Children’s Hospital, Memphis, Tennessee
- Departments of Pediatrics
| | - Frederick A. Boop
- Neuroscience Institute, Le Bonheur Children’s Hospital, Memphis, Tennessee
- Neurosurgery, and
- Anatomy and Neurobiology, University of Tennessee Health Science Center, Memphis, Tennessee; and
| | - Shalini Narayana
- Neuroscience Institute, Le Bonheur Children’s Hospital, Memphis, Tennessee
- Departments of Pediatrics
- Semmes Murphey Neurologic and Spine Institute, Memphis, Tennessee
| |
Collapse
|
4
|
Połczyńska MM, Bookheimer SY. General principles governing the amount of neuroanatomical overlap between languages in bilinguals. Neurosci Biobehav Rev 2021; 130:1-14. [PMID: 34400175 PMCID: PMC8958881 DOI: 10.1016/j.neubiorev.2021.08.005] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2020] [Revised: 07/30/2021] [Accepted: 08/08/2021] [Indexed: 11/28/2022]
Abstract
The literature has identified many important factors affecting the extent to which languages in bilinguals rely on the same neural populations in the specific brain region. The factors include the age of acquisition of the second language (L2), proficiency level of the first language (L1) and L2, and the amount of language exposure, among others. What is lacking is a set of global principles that explain how the many factors relate to the degree to which languages overlap neuroanatomically in bilinguals. We are offering a set of such principles that together account for the numerous sources of data that have been examined individually but not collectively: (1) the principle of acquisition similarity between L1 and L2, (2) the principle of linguistic similarity between L1 and L2, and (3) the principle of cognitive control and effort. Referencing the broad characteristics of language organization in bilinguals, as presented by the principles, can provide a roadmap for future clinical and basic science research.
Collapse
Affiliation(s)
- Monika M Połczyńska
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine at UCLA, University of California, Los Angeles, CA, USA.
| | - Susan Y Bookheimer
- Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine at UCLA, University of California, Los Angeles, CA, USA.
| |
Collapse
|
5
|
Mariën P, van Dun K, Van Dormael J, Vandenborre D, Keulen S, Manto M, Verhoeven J, Abutalebi J. Cerebellar induced differential polyglot aphasia: A neurolinguistic and fMRI study. BRAIN AND LANGUAGE 2017; 175:18-28. [PMID: 28917165 DOI: 10.1016/j.bandl.2017.09.001] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2017] [Revised: 06/27/2017] [Accepted: 09/03/2017] [Indexed: 06/07/2023]
Abstract
Research has shown that linguistic functions in the bilingual brain are subserved by similar neural circuits as in monolinguals, but with extra-activity associated with cognitive and attentional control. Although a role for the right cerebellum in multilingual language processing has recently been acknowledged, a potential role of the left cerebellum remains largely unexplored. This paper reports the clinical and fMRI findings in a strongly right-handed (late) multilingual patient who developed differential polyglot aphasia, ataxic dysarthria and a selective decrease in executive function due to an ischemic stroke in the left cerebellum. fMRI revealed that lexical-semantic retrieval in the unaffected L1 was predominantly associated with activations in the left cortical areas (left prefrontal area and left postcentral gyrus), while naming in two affected non-native languages recruited a significantly larger bilateral functional network, including the cerebellum. It is hypothesized that the left cerebellar insult resulted in decreased right prefrontal hemisphere functioning due to a loss of cerebellar impulses through the cerebello-cerebral pathways.
Collapse
Affiliation(s)
- Peter Mariën
- Clinical and Experimental Neurolinguistics, CLIN, Vrije Universiteit Brussel, Brussels, Belgium; Department of Neurology and Memory Clinic, ZNA Middelheim Hospital, Antwerp, Belgium.
| | - Kim van Dun
- Clinical and Experimental Neurolinguistics, CLIN, Vrije Universiteit Brussel, Brussels, Belgium
| | - Johanna Van Dormael
- Department of Neurological Rehabilitation, University Hospital Brussels, Brussels, Belgium
| | - Dorien Vandenborre
- Clinical and Experimental Neurolinguistics, CLIN, Vrije Universiteit Brussel, Brussels, Belgium; Speech therapy and audiology, Thomas More, Antwerp, Belgium
| | - Stefanie Keulen
- Clinical and Experimental Neurolinguistics, CLIN, Vrije Universiteit Brussel, Brussels, Belgium; Center for Language and Cognition, Rijksuniversiteit Groningen, Groningen, The Netherlands
| | - Mario Manto
- Unité d'Étude du Mouvement, FNRS Neurologie, ULB Erasme, Brussels, Belgium
| | - Jo Verhoeven
- Department of Language and Communication Science, City University, London, UK
| | - Jubin Abutalebi
- Centre for Neurolinguistics and Psycholinguistics, University Vita-Salute San Raffaele, Milan, Italy
| |
Collapse
|
6
|
García AM. Neurocognitive determinants of performance variability among world-language users. ACTA ACUST UNITED AC 2014. [DOI: 10.1080/21698252.2014.893671] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Abstract
Abstract
Although the notion of world language has been variously defined, most accounts acknowledge inter-user performance variability as a key aspect of the construct. The sociocultural aspects of such a phenomenon have been extensively treated in the literature. However, comparatively little attention has been paid to its neurocognitive underpinnings. This paper addresses the biopsychological bases of performance variability among word-language users, focusing on bilingual speakers of English. Available evidence reveals four neurocognitive determinants of variability, namely manner of appropriation, age of acquisition, level of proficiency, and degree of formal similarity between the native and the non-native language. In its concluding section, the paper highlights the benefits of incorporating neurocognitive evidence into the study and conceptualization of world languages.
Collapse
Affiliation(s)
- Adolfo M. García
- Facultad de Lenguas, Universidad Nacional de Córdoba (UNC) , Argentina
- Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Ministry of Science, Technology and Productive Innovation , Buenos Aires , Argentina
- Laboratorio de Psicología Experimental y Neurociencias (LPEN), Instituto de Neurología Cognitiva (INECO) , Buenos Aires , Argentina
| |
Collapse
|
7
|
Costa A, Calabria M, Marne P, Hernández M, Juncadella M, Gascón-Bayarri J, Lleó A, Ortiz-Gil J, Ugas L, Blesa R, Reñé R. On the parallel deterioration of lexico-semantic processes in the bilinguals’ two languages: Evidence from Alzheimer's disease. Neuropsychologia 2012; 50:740-53. [DOI: 10.1016/j.neuropsychologia.2012.01.008] [Citation(s) in RCA: 30] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2011] [Revised: 12/19/2011] [Accepted: 01/03/2012] [Indexed: 11/26/2022]
|
8
|
|
9
|
Green DW, Grogan A, Crinion J, Ali N, Sutton C, Price CJ. Language control and parallel recovery of language in individuals with aphasia. APHASIOLOGY 2010; 24:188-209. [PMID: 20186261 PMCID: PMC2826155 DOI: 10.1080/02687030902958316] [Citation(s) in RCA: 22] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
Background: The causal basis of the different patterns of language recovery following stroke in bilingual speakers is not well understood. Our approach distinguishes the representation of language from the mechanisms involved in its control. Previous studies have suggested that difficulties in language control can explain selective aphasia in one language as well as pathological switching between languages. Here we test the hypothesis that difficulties in managing and resolving competition will also be observed in those who are equally impaired in both their languages even in the absence of pathological switching.Aims: To examine difficulties in language control in bilingual individuals with parallel recovery in aphasia and to compare their performance on different types of conflict task.Methods & Procedures: Two right-handed, non-native English-speaking participants who showed parallel recovery of two languages after stroke and a group of non-native English-speaking, bilingual controls described a scene in English and in their first language and completed three explicit conflict tasks. Two of these were verbal conflict tasks: a lexical decision task in English, in which individuals distinguished English words from non-words, and a Stroop task, in English and in their first language. The third conflict task was a non-verbal flanker task.Outcomes & Results: Both participants with aphasia were impaired in the picture description task in English and in their first language but showed different patterns of impairment on the conflict tasks. For the participant with left subcortical damage, conflict was abnormally high during the verbal tasks (lexical decision and Stroop) but not during the non-verbal flanker task. In contrast, for the participant with extensive left parietal damage, conflict was less abnormal during the Stroop task than the flanker or lexical decision task.Conclusions: Our data reveal two distinct control impairments associated with parallel recovery. We stress the need to explore the precise nature of control problems and how control is implemented in order to develop fuller causal accounts of language recovery patterns in bilingual aphasia.
Collapse
Affiliation(s)
- David W. Green
- Address correspondence to: David W. Green, Division of Psychology and Language Sciences, 26 Bedford Way, London WC1H 0AP, UK. E-mail:
| | | | | | | | | | | |
Collapse
|
10
|
Bilingualism and the Brain: Myth and Reality. NEUROLOGÍA (ENGLISH EDITION) 2010. [DOI: 10.1016/s2173-5808(10)70082-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022] Open
|
11
|
Roux FE, Trémoulet M. Organization of language areas in bilingual patients: a cortical stimulation study. J Neurosurg 2002; 97:857-64. [PMID: 12405374 DOI: 10.3171/jns.2002.97.4.0857] [Citation(s) in RCA: 91] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
OBJECT In an attempt to gain a better understanding of how multiple languages are represented in the human brain, the authors studied bilingual patients who underwent surgery for brain tumors, during which the authors mapped cortical language sites by using electrostimulation. METHODS Reading, counting, and word retrieval tasks were studied in 12 right-handed bilingual patients with no language deficit. All bilingual patients were native to France. One patient spoke four languages. The patients constituted a nonhomogeneous group in terms of language proficiency or age of acquisition. Languages were evaluated and classified into three major groups, depending on proficiency and date of acquisition. Strict conditions of language site validation were applied, separating typical anomia sites from speech arrest or other language sites (such as hesitation sites). A total of 30 speech arrest sites, 16 anomia sites, and three sites of language difficulties (not typically classified as speech arrest) were found throughout the 26 language studies performed. Strict overlapping of language areas (for all language tasks) was found in five patients, whereas the remaining seven had at least one area that was language-specific and sometimes task-specific. Specific areas for a particular language were found for word retrieval tasks (anomia) in eight sites (50%) but also in six (20%) of the reading or counting sites (speech arrest), either in frontal (three patients) or in temporoparietal (four patients) regions. Among the four early bilingual patients tested (languages acquired before the age of 7 years), three had language-specific cortical areas. Interestingly, six patients in this series who had a discrepancy between two languages did not have more cortical areas devoted to the less proficient language (with acknowledgment of the limit in cortical exposure available for testing by the craniotomy). CONCLUSIONS In this series, the authors found that bilingual patients could have common but also different cortical areas for both languages in temporoparietal areas and in frontal areas. In some cases, the authors found that language tasks such as counting, reading, or word retrieval in different languages can be sustained by language- and task-specific cortical areas. In bilingual patients, cortical mapping should ideally be performed using different language tasks in all languages in which the patient is fluent.
Collapse
Affiliation(s)
- Franck-Emmanuel Roux
- Institut National de la Santé et de la Recherche Médicale, Unité 455, Hĵpital Purpan, Toulouse, France.
| | | |
Collapse
|
12
|
Abstract
We report a pediatric patient with auditory agnosia as a sequel of herpes encephalitis. His early development was completely normal. He uttered three words at 12 months old. Disease onset was 1 year and 2 months of age. He was discharged from the hospital seemingly with no sequel; however, he could not recover his intelligible words even at age 2 years. He was diagnosed as having auditory agnosia caused by bilateral temporal lobe injury. We began to train him at once, individually and intensively. Adult patients with pure auditory agnosia followed by two episodes of temporal lobe infarction have impairment in central hearing but not inner language. Therefore, they can communicate by reading and writing. Moreover, impairment in hearing is not always severe and is often transient. However, despite long-term (more than 15 years) energetic education and almost normal intellectual ability (Performance IQ of Wechsler Intelligence Scale for Children-Revised was 91), our patient's language ability was extremely poor. Cerebral plasticity could not work fully on our patient, whose bilateral temporal lobe was severely injured in early childhood. The establishment of a systematic training method in such patients is an urgent objective in this field.
Collapse
Affiliation(s)
- M Kaga
- National Institute of Mental Health, National Center of Neurology and Psychiatry, Chiba, Japan
| | | | | |
Collapse
|