1
|
Pearl JE, Matsumoto N, Hayashi K, Matsuda K, Miura K, Nagai Y, Miyakawa N, Minanimoto T, Saunders RC, Sugase-Miyamoto Y, Richmond BJ, Eldridge MAG. Neural correlates of category learning in monkey inferior temporal cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.12.05.568765. [PMID: 38168336 PMCID: PMC10760203 DOI: 10.1101/2023.12.05.568765] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/05/2024]
Abstract
We trained two monkeys implanted with multi-electrode arrays to categorize natural images of cats and dogs, in order to observe changes in neural activity related to category learning. We recorded neural activity from area TE, which is required for normal learning of visual categories based on perceptual similarity. Neural activity during a passive viewing task was compared pre- and post-training. After the category training, the accuracy of abstract category decoding improved. Specifically, the proportion of single units with category selectivity increased, and units sustained their category-specific responses for longer. Visual category learning thus appears to enhance category separability in area TE by driving changes in the stimulus selectivity of individual neurons and by recruiting more units to the active network.
Collapse
Affiliation(s)
- Jonah E Pearl
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
- Current affiliation: Department of Neurobiology, Harvard Medical School, Boston, MA 02115, United States
| | - Narihisa Matsumoto
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan
| | - Kazuko Hayashi
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan
| | - Keiji Matsuda
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan
| | - Kenichiro Miura
- Department of Pathology of Mental Diseases, National Institute of Mental Health, National Center of Neurology and Psychiatry, Kodaira, Tokyo, Japan
| | - Yuji Nagai
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, (QST), Inage, Chiba 263-8555, Japan
| | - Naohisa Miyakawa
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, (QST), Inage, Chiba 263-8555, Japan
| | - Takafumi Minanimoto
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, (QST), Inage, Chiba 263-8555, Japan
| | - Richard C Saunders
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| | - Yasuko Sugase-Miyamoto
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| | - Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| |
Collapse
|
2
|
Eldridge MAG, Pearl JE, Fomani GP, Masseau EC, Fredericks JM, Chen G, Richmond BJ. Visual recognition in rhesus monkeys requires area TE but not TEO. Cereb Cortex 2023; 33:3098-3106. [PMID: 35770336 PMCID: PMC10016064 DOI: 10.1093/cercor/bhac263] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2021] [Revised: 06/05/2022] [Accepted: 06/06/2022] [Indexed: 11/13/2022] Open
Abstract
The primate visual system is often described as a hierarchical feature-conjunction pathway, whereby each level represents an increasingly complex combination of image elements, culminating in the representation of whole coherent images in anterior inferior temporal cortex. Although many models of the ventral visual stream emphasize serial feedforward processing ((Poggio T, Mutch J, Leibo J, Rosasco L, Tacchetti A. The computationalmagic of the ventral stream: sketch of a theory (and why some deep architectures work). TechRep MIT-CSAIL-TR-2012-035. MIT CSAIL, Cambridge, MA. 2012); (Yamins DLK, DiCarlo JJ. Eight open questions in the computational modeling of higher sensory cortex. Curr Opin Neurobiol. 2016:37:114-120.)), anatomical studies show connections that bypass intermediate areas and that feedback to preceding areas ((Distler C, Boussaoud D, Desimone R, Ungerleider LG. Cortical connections of inferior temporal area TEO in macaque monkeys. J Comp Neurol. 1993:334(1):125-150.); (Kravitz DJ, Saleem KS, Baker CI, Mishkin M. A new neural framework for visuospatial processing. Nat Rev Neurosci. 2011:12(4):217-230.)). Prior studies on visual discrimination and object transforms also provide evidence against a strictly feed-forward serial transfer of information between adjacent areas ((Kikuchi R, Iwai E. The locus of the posterior subdivision of the inferotemporal visual learning area in the monkey. Brain Res. 1980:198(2):347-360.); (Weiskrantz L, Saunders RC. Impairments of visual object transforms in monkeys. Brain. 1984:107(4):1033-1072.); (Kar K, DiCarlo JJ. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust Core visual object recognition. Neuron. 2021:109(1):164-176.e5.)). Thus, we sought to investigate whether behaviorally relevant propagation of visual information is as strictly sequential as sometimes supposed. We compared the accuracy of visual recognition after selective removal of specific subregions of inferior temporal cortex-area TEO, area TE, or both areas combined. Removal of TEO alone had no detectable effect on recognition memory, whereas removal of TE alone produced a large and significant impairment. Combined removal of both areas created no additional deficit relative to removal of TE alone. Thus, area TE is critical for rapid visual object recognition, and detailed image-level visual information can reach area TE via a route other than through TEO.
Collapse
Affiliation(s)
- Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| | - Jonah E Pearl
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
- Department of Neurobiology, Harvard Medical School, Boston, MA 02115, United States
| | - Grace P Fomani
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| | - Evan C Masseau
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| | - J Megan Fredericks
- Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, One Gustave L. Levy Place, New York, NY 10014, United States
| | - Gang Chen
- Scientific and Statistical Computing Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, United States
| |
Collapse
|
3
|
Local features drive identity responses in macaque anterior face patches. Nat Commun 2022; 13:5592. [PMID: 36151142 PMCID: PMC9508131 DOI: 10.1038/s41467-022-33240-w] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2021] [Accepted: 09/08/2022] [Indexed: 12/03/2022] Open
Abstract
Humans and other primates recognize one another in part based on unique structural details of the face, including both local features and their spatial configuration within the head and body. Visual analysis of the face is supported by specialized regions of the primate cerebral cortex, which in macaques are commonly known as face patches. Here we ask whether the responses of neurons in anterior face patches, thought to encode face identity, are more strongly driven by local or holistic facial structure. We created stimuli consisting of recombinant photorealistic images of macaques, where we interchanged the eyes, mouth, head, and body between individuals. Unexpectedly, neurons in the anterior medial (AM) and anterior fundus (AF) face patches were predominantly tuned to local facial features, with minimal neural selectivity for feature combinations. These findings indicate that the high-level structural encoding of face identity rests upon populations of neurons specialized for local features. Anterior face patches in the macaque have been assumed to represent face identity in a holistic manner. Here the authors show that the neural encoding of face identity in the anterior medial and anterior fundus face patches are instead driven principally by local features.
Collapse
|
4
|
Matsumoto N, Eldridge MAG, Fredericks JM, Lowe KA, Richmond BJ. Comparing performance between a deep neural network and monkeys with bilateral removals of visual area TE in categorizing feature-ambiguous stimuli. J Comput Neurosci 2022; 51:381-387. [PMID: 37195295 DOI: 10.1007/s10827-023-00854-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2021] [Revised: 03/24/2023] [Accepted: 05/03/2023] [Indexed: 05/18/2023]
Abstract
In the canonical view of visual processing the neural representation of complex objects emerges as visual information is integrated through a set of convergent, hierarchically organized processing stages, ending in the primate inferior temporal lobe. It seems reasonable to infer that visual perceptual categorization requires the integrity of anterior inferior temporal cortex (area TE). Many deep neural networks (DNNs) are structured to simulate the canonical view of hierarchical processing within the visual system. However, there are some discrepancies between DNNs and the primate brain. Here we evaluated the performance of a simulated hierarchical model of vision in discriminating the same categorization problems presented to monkeys with TE removals. The model was able to simulate the performance of monkeys with TE removals in the categorization task but performed poorly when challenged with visually degraded stimuli. We conclude that further development of the model is required to match the level of visual flexibility present in the monkey visual system.
Collapse
Affiliation(s)
- Narihisa Matsumoto
- Human Informatics and Interaction Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Ibaraki, Japan.
| | - Mark A G Eldridge
- National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA
| | - J Megan Fredericks
- National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA
| | - Kaleb A Lowe
- National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA
| | - Barry J Richmond
- National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA.
| |
Collapse
|
5
|
Setogawa T, Eldridge MAG, Fomani GP, Saunders RC, Richmond BJ. Contributions of the Monkey Inferior Temporal Areas TE and TEO to Visual Categorization. Cereb Cortex 2021; 31:4891-4900. [PMID: 33987672 PMCID: PMC8491680 DOI: 10.1093/cercor/bhab129] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2021] [Revised: 03/22/2021] [Accepted: 04/14/2021] [Indexed: 11/14/2022] Open
Abstract
The ability to categorize images is thought to depend on neural processing within the ventral visual stream. Recently, we reported that after removal of architectonic area TE, the terminal region of the ventral stream, monkeys were still able to categorize images as cats or dogs moderately well. Here, we investigate the contribution of TEO, the architectonically defined region located one step earlier than area TE in the ventral stream. Bilateral removal of TEO caused only a mild impairment in categorization. However, combined TE + TEO removal was followed by a severe, long-lasting impairment in categorization. All of the monkeys tested, including those with combined TE + TEO removals, had normal low-level visual functions, such as visual acuity. These results support the conclusion that categorization based on visual similarity is processed in parallel in TE and TEO.
Collapse
Affiliation(s)
- Tsuyoshi Setogawa
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, USA
- System Emotional Science, Faculty of Medicine, University of Toyama, Toyama 930-0194, Japan
| | - Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, USA
| | - Grace P Fomani
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, USA
| | - Richard C Saunders
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, USA
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, MD 20892, USA
| |
Collapse
|
6
|
Eldridge MAG, Hines BE, Murray EA. The visual prefrontal cortex of anthropoids: interaction with temporal cortex in decision making and its role in the making of "visual animals". Curr Opin Behav Sci 2021; 41:22-29. [PMID: 33796638 PMCID: PMC8009333 DOI: 10.1016/j.cobeha.2021.02.012] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
The ventral prefrontal cortex (PFC) of primates-a region strongly implicated in decision making-receives highly processed visual sensory inputs from the inferior temporal cortex (ITC) and perirhinal cortex (PRC) and can therefore be considered visual PFC. Usually, the functions of temporal cortex and visual PFC have been discussed in separate literatures. By considering them together, we aim to clarify the ways in which fronto-temporal networks guide decision making. After discussing the ways in which visual PFC interacts with temporal cortex to promote decision making, we offer specific predictions about the selective roles of the ITC- and PRC-based fronto-temporal networks. Finally, we suggest that an increased reliance on visual PFC in anthropoid primates led to our emergence as 'visual' animals.
Collapse
Affiliation(s)
- Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, Bethesda, MD 20892
| | - Brendan E Hines
- Laboratory of Neuropsychology, National Institute of Mental Health, Bethesda, MD 20892
| | - Elisabeth A Murray
- Laboratory of Neuropsychology, National Institute of Mental Health, Bethesda, MD 20892
| |
Collapse
|
7
|
|
8
|
Rajalingham R, DiCarlo JJ. Reversible Inactivation of Different Millimeter-Scale Regions of Primate IT Results in Different Patterns of Core Object Recognition Deficits. Neuron 2019; 102:493-505.e5. [PMID: 30878289 DOI: 10.1016/j.neuron.2019.02.001] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2018] [Revised: 12/06/2018] [Accepted: 01/31/2019] [Indexed: 11/26/2022]
Abstract
Extensive research suggests that the inferior temporal (IT) population supports visual object recognition behavior. However, causal evidence for this hypothesis has been equivocal, particularly beyond the specific case of face-selective subregions of IT. Here, we directly tested this hypothesis by pharmacologically inactivating individual, millimeter-scale subregions of IT while monkeys performed several core object recognition subtasks, interleaved trial-by trial. First, we observed that IT inactivation resulted in reliable contralateral-biased subtask-selective behavioral deficits. Moreover, inactivating different IT subregions resulted in different patterns of subtask deficits, predicted by each subregion's neuronal object discriminability. Finally, the similarity between different inactivation effects was tightly related to the anatomical distance between corresponding inactivation sites. Taken together, these results provide direct evidence that the IT cortex causally supports general core object recognition and that the underlying IT coding dimensions are topographically organized.
Collapse
Affiliation(s)
- Rishi Rajalingham
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
| | - James J DiCarlo
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA.
| |
Collapse
|
9
|
Eldridge MAG, Matsumoto N, Wittig JH, Masseau EC, Saunders RC, Richmond BJ. Perceptual processing in the ventral visual stream requires area TE but not rhinal cortex. eLife 2018; 7:e36310. [PMID: 30311907 PMCID: PMC6207425 DOI: 10.7554/elife.36310] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2018] [Accepted: 10/11/2018] [Indexed: 11/13/2022] Open
Abstract
There is an on-going debate over whether area TE, or the anatomically adjacent rhinal cortex, is the final stage of visual object processing. Both regions have been implicated in visual perception, but their involvement in non-perceptual functions, such as short-term memory, hinders clear-cut interpretation. Here, using a two-interval forced choice task without a short-term memory demand, we find that after bilateral removal of area TE, monkeys trained to categorize images based on perceptual similarity (morphs between dogs and cats), are, on the initial viewing, badly impaired when given a new set of images. They improve markedly with a small amount of practice but nonetheless remain moderately impaired indefinitely. The monkeys with bilateral removal of rhinal cortex are, under all conditions, indistinguishable from unoperated controls. We conclude that the final stage of the integration of visual perceptual information into object percepts in the ventral visual stream occurs in area TE.
Collapse
Affiliation(s)
- Mark AG Eldridge
- Laboratory of NeuropsychologyNational Institute of Mental Health, National Institutes of HealthBethesdaUnited States
| | - Narihisa Matsumoto
- Human Informatics Research InstituteNational Institute of Advanced Industrial Science and TechnologyTsukubaJapan
| | - John H Wittig
- Surgical Neurology BranchNational Institute of Neurological Disorders and Stroke, National Institutes of HealthBethesdaUnited States
| | - Evan C Masseau
- Laboratory of NeuropsychologyNational Institute of Mental Health, National Institutes of HealthBethesdaUnited States
| | - Richard C Saunders
- Laboratory of NeuropsychologyNational Institute of Mental Health, National Institutes of HealthBethesdaUnited States
| | - Barry J Richmond
- Laboratory of NeuropsychologyNational Institute of Mental Health, National Institutes of HealthBethesdaUnited States
| |
Collapse
|
10
|
Abstract
Inferior temporal cortex (IT) is a key part of the ventral visual pathway implicated in object, face, and scene perception. But how does IT work? Here, I describe an organizational scheme that marries form and function and provides a framework for future research. The scheme consists of a series of stages arranged along the posterior-anterior axis of IT, defined by anatomical connections and functional responses. Each stage comprises a complement of subregions that have a systematic spatial relationship. The organization of each stage is governed by an eccentricity template, and corresponding eccentricity representations across stages are interconnected. Foveal representations take on a role in high-acuity object vision (including face recognition); intermediate representations compute other aspects of object vision such as behavioral valence (using color and surface cues); and peripheral representations encode information about scenes. This multistage, parallel-processing model invokes an innately determined organization refined by visual experience that is consistent with principles of cortical development. The model is also consistent with principles of evolution, which suggest that visual cortex expanded through replication of retinotopic areas. Finally, the model predicts that the most extensively studied network within IT-the face patches-is not unique but rather one manifestation of a canonical set of operations that reveal general principles of how IT works.
Collapse
Affiliation(s)
- Bevil R Conway
- Laboratory of Sensorimotor Research, National Eye Institute, National Institutes of Health, Bethesda, Maryland 28092, USA; .,National Institutes of Mental Health, National Institute of Neurological Disease and Stroke, National Institutes of Health, Bethesda, Maryland 28092, USA
| |
Collapse
|