1
|
Thieu MK, Ayzenberg V, Lourenco SF, Kragel PA. Visual looming is a primitive for human emotion. iScience 2024; 27:109886. [PMID: 38799577 PMCID: PMC11126809 DOI: 10.1016/j.isci.2024.109886] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Revised: 03/11/2024] [Accepted: 04/30/2024] [Indexed: 05/29/2024] Open
Abstract
The neural computations for looming detection are strikingly similar across species. In mammals, information about approaching threats is conveyed from the retina to the midbrain superior colliculus, where approach variables are computed to enable defensive behavior. Although neuroscientific theories posit that midbrain representations contribute to emotion through connectivity with distributed brain systems, it remains unknown whether a computational system for looming detection can predict both defensive behavior and phenomenal experience in humans. Here, we show that a shallow convolutional neural network based on the Drosophila visual system predicts defensive blinking to looming objects in infants and superior colliculus responses to optical expansion in adults. Further, the neural network's responses to naturalistic video clips predict self-reported emotion largely by way of subjective arousal. These findings illustrate how a simple neural network architecture optimized for a species-general task relevant for survival explains motor and experiential components of human emotion.
Collapse
Affiliation(s)
| | - Vladislav Ayzenberg
- Emory University, Atlanta, GA, USA
- University of Pennsylvania, Philadelphia, PA, USA
| | | | | |
Collapse
|
2
|
Lützow Holm E, Fernández Slezak D, Tagliazucchi E. Contribution of low-level image statistics to EEG decoding of semantic content in multivariate and univariate models with feature optimization. Neuroimage 2024; 293:120626. [PMID: 38677632 DOI: 10.1016/j.neuroimage.2024.120626] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2024] [Revised: 04/23/2024] [Accepted: 04/24/2024] [Indexed: 04/29/2024] Open
Abstract
Spatio-temporal patterns of evoked brain activity contain information that can be used to decode and categorize the semantic content of visual stimuli. However, this procedure can be biased by low-level image features independently of the semantic content present in the stimuli, prompting the need to understand the robustness of different models regarding these confounding factors. In this study, we trained machine learning models to distinguish between concepts included in the publicly available THINGS-EEG dataset using electroencephalography (EEG) data acquired during a rapid serial visual presentation paradigm. We investigated the contribution of low-level image features to decoding accuracy in a multivariate model, utilizing broadband data from all EEG channels. Additionally, we explored a univariate model obtained through data-driven feature selection applied to the spatial and frequency domains. While the univariate models exhibited better decoding accuracy, their predictions were less robust to the confounding effect of low-level image statistics. Notably, some of the models maintained their accuracy even after random replacement of the training dataset with semantically unrelated samples that presented similar low-level content. In conclusion, our findings suggest that model optimization impacts sensitivity to confounding factors, regardless of the resulting classification performance. Therefore, the choice of EEG features for semantic decoding should ideally be informed by criteria beyond classifier performance, such as the neurobiological mechanisms under study.
Collapse
Affiliation(s)
- Eric Lützow Holm
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Institute of Applied and Interdisciplinary Physics and Department of Physics, University of Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina.
| | - Diego Fernández Slezak
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Departamento de Computación, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina; Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-Universidad de Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina
| | - Enzo Tagliazucchi
- National Scientific and Technical Research Council (CONICET), Godoy Cruz 2290, CABA 1425, Argentina; Institute of Applied and Interdisciplinary Physics and Department of Physics, University of Buenos Aires, Pabellón 1, Ciudad Universitaria, CABA 1425, Argentina; Latin American Brain Health (BrainLat), Universidad Adolfo Ibáñez, Av. Diag. Las Torres 2640, Peñalolén 7941169, Santiago Región Metropolitana, Chile.
| |
Collapse
|
3
|
Ren Y, Bashivan P. How well do models of visual cortex generalize to out of distribution samples? PLoS Comput Biol 2024; 20:e1011145. [PMID: 38820563 DOI: 10.1371/journal.pcbi.1011145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Accepted: 04/29/2024] [Indexed: 06/02/2024] Open
Abstract
Unit activity in particular deep neural networks (DNNs) are remarkably similar to the neuronal population responses to static images along the primate ventral visual cortex. Linear combinations of DNN unit activities are widely used to build predictive models of neuronal activity in the visual cortex. Nevertheless, prediction performance in these models is often investigated on stimulus sets consisting of everyday objects under naturalistic settings. Recent work has revealed a generalization gap in how predicting neuronal responses to synthetically generated out-of-distribution (OOD) stimuli. Here, we investigated how the recent progress in improving DNNs' object recognition generalization, as well as various DNN design choices such as architecture, learning algorithm, and datasets have impacted the generalization gap in neural predictivity. We came to a surprising conclusion that the performance on none of the common computer vision OOD object recognition benchmarks is predictive of OOD neural predictivity performance. Furthermore, we found that adversarially robust models often yield substantially higher generalization in neural predictivity, although the degree of robustness itself was not predictive of neural predictivity score. These results suggest that improving object recognition behavior on current benchmarks alone may not lead to more general models of neurons in the primate ventral visual cortex.
Collapse
Affiliation(s)
- Yifei Ren
- Department of Computer Science, McGill University, Montreal, Canada
| | - Pouya Bashivan
- Department of Computer Science, McGill University, Montreal, Canada
- Department of Computer Physiology, McGill University, Montreal, Canada
- Mila, Université de Montréal, Montreal, Canada
| |
Collapse
|
4
|
Brands AM, Devore S, Devinsky O, Doyle W, Flinker A, Friedman D, Dugan P, Winawer J, Groen IIA. Temporal dynamics of short-term neural adaptation across human visual cortex. PLoS Comput Biol 2024; 20:e1012161. [PMID: 38815000 DOI: 10.1371/journal.pcbi.1012161] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Accepted: 05/12/2024] [Indexed: 06/01/2024] Open
Abstract
Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses between lower and higher visual brain areas and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.
Collapse
Affiliation(s)
| | - Sasha Devore
- New York University Grossman School of Medicine, New York, New York, United States of America
| | - Orrin Devinsky
- New York University Grossman School of Medicine, New York, New York, United States of America
| | - Werner Doyle
- New York University Grossman School of Medicine, New York, New York, United States of America
| | - Adeen Flinker
- New York University Grossman School of Medicine, New York, New York, United States of America
| | - Daniel Friedman
- New York University Grossman School of Medicine, New York, New York, United States of America
| | - Patricia Dugan
- New York University Grossman School of Medicine, New York, New York, United States of America
| | - Jonathan Winawer
- Department of Psychology, New York University, New York, New York, United States of America
| | | |
Collapse
|
5
|
Millière R. Philosophy of cognitive science in the age of deep learning. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2024:e1684. [PMID: 38773731 DOI: 10.1002/wcs.1684] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 04/22/2024] [Accepted: 04/24/2024] [Indexed: 05/24/2024]
Abstract
Deep learning has enabled major advances across most areas of artificial intelligence research. This remarkable progress extends beyond mere engineering achievements and holds significant relevance for the philosophy of cognitive science. Deep neural networks have made significant strides in overcoming the limitations of older connectionist models that once occupied the center stage of philosophical debates about cognition. This development is directly relevant to long-standing theoretical debates in the philosophy of cognitive science. Furthermore, ongoing methodological challenges related to the comparative evaluation of deep neural networks stand to benefit greatly from interdisciplinary collaboration with philosophy and cognitive science. The time is ripe for philosophers to explore foundational issues related to deep learning and cognition; this perspective paper surveys key areas where their contributions can be especially fruitful. This article is categorized under: Philosophy > Artificial Intelligence Computer Science and Robotics > Machine Learning.
Collapse
Affiliation(s)
- Raphaël Millière
- Department of Philosophy, Macquarie University, Sydney, Australia
| |
Collapse
|
6
|
Mascarenhas M, Alencoão I, Carinhas MJ, Martins M, Cardoso P, Mendes F, Fernandes J, Ferreira J, Macedo G, Zulmira Macedo R. Artificial Intelligence and Colposcopy: Automatic Identification of Cervical Squamous Cell Carcinoma Precursors. J Clin Med 2024; 13:3003. [PMID: 38792544 PMCID: PMC11122610 DOI: 10.3390/jcm13103003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2024] [Revised: 04/21/2024] [Accepted: 05/16/2024] [Indexed: 05/26/2024] Open
Abstract
Background/Objectives: Proficient colposcopy is crucial for the adequate management of cervical cancer precursor lesions; nonetheless its limitations may impact its cost-effectiveness. The development of artificial intelligence models is experiencing an exponential growth, particularly in image-based specialties. The aim of this study is to develop and validate a Convolutional Neural Network (CNN) for the automatic differentiation of high-grade (HSIL) from low-grade dysplasia (LSIL) in colposcopy. Methods: A unicentric retrospective study was conducted based on 70 colposcopy exams, comprising a total of 22,693 frames. Among these, 8729 were categorized as HSIL based on histopathology. The total dataset was divided into a training (90%, n = 20,423) and a testing set (10%, n = 2270), the latter being used to evaluate the model's performance. The main outcome measures included sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (NPV), and the area under the receiving operating curve (AUC-ROC). Results: The sensitivity was 99.7% and the specificity was 98.6%. The PPV and NPV were 97.8% and 99.8%, respectively. The overall accuracy was 99.0%. The AUC-ROC was 0.98. The CNN processed 112 frames per second. Conclusions: We developed a CNN capable of differentiating cervical cancer precursors in colposcopy frames. The high levels of accuracy for the differentiation of HSIL from LSIL may improve the diagnostic yield of this exam.
Collapse
Affiliation(s)
- Miguel Mascarenhas
- Precision Medicine Unit, Department of Gastroenterology, São João University Hospital, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal; (M.M.); (P.C.); (G.M.)
- Faculty of Medicine, University of Porto, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal
| | - Inês Alencoão
- Department of Gynecology, Centro Materno-Infantil do Norte Dr. Albino Aroso (CMIN), Santo António University Hospital, Largo da Maternidade Júlio Dinis, 4050-061 Porto, Portugal; (I.A.); (M.J.C.); (R.Z.M.)
| | - Maria João Carinhas
- Department of Gynecology, Centro Materno-Infantil do Norte Dr. Albino Aroso (CMIN), Santo António University Hospital, Largo da Maternidade Júlio Dinis, 4050-061 Porto, Portugal; (I.A.); (M.J.C.); (R.Z.M.)
| | - Miguel Martins
- Precision Medicine Unit, Department of Gastroenterology, São João University Hospital, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal; (M.M.); (P.C.); (G.M.)
| | - Pedro Cardoso
- Precision Medicine Unit, Department of Gastroenterology, São João University Hospital, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal; (M.M.); (P.C.); (G.M.)
| | - Francisco Mendes
- Precision Medicine Unit, Department of Gastroenterology, São João University Hospital, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal; (M.M.); (P.C.); (G.M.)
| | - Joana Fernandes
- Faculty of Engineering, University of Porto, Rua Dr. Roberto Frias, 4200-065 Porto, Portugal; (J.F.); (J.F.)
| | - João Ferreira
- Faculty of Engineering, University of Porto, Rua Dr. Roberto Frias, 4200-065 Porto, Portugal; (J.F.); (J.F.)
| | - Guilherme Macedo
- Precision Medicine Unit, Department of Gastroenterology, São João University Hospital, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal; (M.M.); (P.C.); (G.M.)
- Faculty of Medicine, University of Porto, Alameda Professor Hernâni Monteiro, 4200-427 Porto, Portugal
| | - Rosa Zulmira Macedo
- Department of Gynecology, Centro Materno-Infantil do Norte Dr. Albino Aroso (CMIN), Santo António University Hospital, Largo da Maternidade Júlio Dinis, 4050-061 Porto, Portugal; (I.A.); (M.J.C.); (R.Z.M.)
| |
Collapse
|
7
|
Mattia GM, Villain E, Nemmi F, Le Lann MV, Franceries X, Péran P. Investigating the discrimination ability of 3D convolutional neural networks applied to altered brain MRI parametric maps. Artif Intell Med 2024; 153:102897. [PMID: 38810471 DOI: 10.1016/j.artmed.2024.102897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 03/05/2024] [Accepted: 05/15/2024] [Indexed: 05/31/2024]
Abstract
Convolutional neural networks (CNNs) are gradually being recognized in the neuroimaging community as a powerful tool for image analysis. Despite their outstanding performances, some aspects of CNN functioning are still not fully understood by human operators. We postulated that the interpretability of CNNs applied to neuroimaging data could be improved by investigating their behavior when they are fed data with known characteristics. We analyzed the ability of 3D CNNs to discriminate between original and altered whole-brain parametric maps derived from diffusion-weighted magnetic resonance imaging. The alteration consisted in linearly changing the voxel intensity of either one (monoregion) or two (biregion) anatomical regions in each brain volume, but without mimicking any neuropathology. Performing ten-fold cross-validation and using a hold-out set for testing, we assessed the CNNs' discrimination ability according to the intensity of the altered regions, comparing the latter's size and relative position. Monoregion CNNs showed that the larger the modified region, the smaller the intensity increase needed to achieve good performances. Biregion CNNs systematically outperformed monoregion CNNs, but could only detect one of the two target regions when tested on the corresponding monoregion images. Exploiting prior information on training data allowed for a better understanding of CNN behavior, especially when altered regions were combined. This can inform about the complexity of CNN pattern retrieval and elucidate misclassified examples, particularly relevant for pathological data. The proposed analytical approach may serve to gain insights into CNN behavior and guide the design of enhanced detection systems exploiting our prior knowledge.
Collapse
Affiliation(s)
- Giulia Maria Mattia
- ToNIC, Toulouse NeuroImaging Center, Université de Toulouse, Inserm, UPS, Toulouse, France.
| | - Edouard Villain
- ToNIC, Toulouse NeuroImaging Center, Université de Toulouse, Inserm, UPS, Toulouse, France; LAAS CNRS, Université de Toulouse, CNRS, INSA, UPS, Toulouse, France.
| | - Federico Nemmi
- ToNIC, Toulouse NeuroImaging Center, Université de Toulouse, Inserm, UPS, Toulouse, France.
| | | | - Xavier Franceries
- CRCT, Centre de Recherche en Cancérologie de Toulouse, Inserm, UPS, Toulouse, France.
| | - Patrice Péran
- ToNIC, Toulouse NeuroImaging Center, Université de Toulouse, Inserm, UPS, Toulouse, France.
| |
Collapse
|
8
|
Mastrovito D, Liu YH, Kusmierz L, Shea-Brown E, Koch C, Mihalas S. Transition to chaos separates learning regimes and relates to measure of consciousness in recurrent neural networks. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.15.594236. [PMID: 38798582 PMCID: PMC11118502 DOI: 10.1101/2024.05.15.594236] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
Recurrent neural networks exhibit chaotic dynamics when the variance in their connection strengths exceed a critical value. Recent work indicates connection variance also modulates learning strategies; networks learn "rich" representations when initialized with low coupling and "lazier"solutions with larger variance. Using Watts-Strogatz networks of varying sparsity, structure, and hidden weight variance, we find that the critical coupling strength dividing chaotic from ordered dynamics also differentiates rich and lazy learning strategies. Training moves both stable and chaotic networks closer to the edge of chaos, with networks learning richer representations before the transition to chaos. In contrast, biologically realistic connectivity structures foster stability over a wide range of variances. The transition to chaos is also reflected in a measure that clinically discriminates levels of consciousness, the perturbational complexity index (PCIst). Networks with high values of PCIst exhibit stable dynamics and rich learning, suggesting a consciousness prior may promote rich learning. The results suggest a clear relationship between critical dynamics, learning regimes and complexity-based measures of consciousness.
Collapse
|
9
|
Margalit E, Lee H, Finzi D, DiCarlo JJ, Grill-Spector K, Yamins DLK. A unifying framework for functional organization in early and higher ventral visual cortex. Neuron 2024:S0896-6273(24)00279-4. [PMID: 38733985 DOI: 10.1016/j.neuron.2024.04.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Revised: 12/08/2023] [Accepted: 04/15/2024] [Indexed: 05/13/2024]
Abstract
A key feature of cortical systems is functional organization: the arrangement of functionally distinct neurons in characteristic spatial patterns. However, the principles underlying the emergence of functional organization in the cortex are poorly understood. Here, we develop the topographic deep artificial neural network (TDANN), the first model to predict several aspects of the functional organization of multiple cortical areas in the primate visual system. We analyze the factors driving the TDANN's success and find that it balances two objectives: learning a task-general sensory representation and maximizing the spatial smoothness of responses according to a metric that scales with cortical surface area. In turn, the representations learned by the TDANN are more brain-like than in spatially unconstrained models. Finally, we provide evidence that the TDANN's functional organization balances performance with between-area connection length. Our results offer a unified principle for understanding the functional organization of the primate ventral visual system.
Collapse
Affiliation(s)
- Eshed Margalit
- Neurosciences Graduate Program, Stanford University, Stanford, CA 94305, USA.
| | - Hyodong Lee
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Dawn Finzi
- Department of Psychology, Stanford University, Stanford, CA 94305, USA; Department of Computer Science, Stanford University, Stanford, CA 94305, USA
| | - James J DiCarlo
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139, USA; McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA; Center for Brains Minds and Machines, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Kalanit Grill-Spector
- Department of Psychology, Stanford University, Stanford, CA 94305, USA; Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305, USA
| | - Daniel L K Yamins
- Department of Psychology, Stanford University, Stanford, CA 94305, USA; Department of Computer Science, Stanford University, Stanford, CA 94305, USA; Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305, USA
| |
Collapse
|
10
|
Chen Z, Yadollahpour A. A new era in cognitive neuroscience: the tidal wave of artificial intelligence (AI). BMC Neurosci 2024; 25:23. [PMID: 38711047 DOI: 10.1186/s12868-024-00869-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2023] [Accepted: 04/26/2024] [Indexed: 05/08/2024] Open
Abstract
Translating artificial intelligence techniques into the realm of cognitive neuroscience holds promise for significant breakthroughs in our ability to probe the intrinsic mechanisms of the brain. The recent unprecedented development of robust AI models is changing how and what we understand about the brain. In this Editorial, we invite contributions for a BMC Neuroscience Collection on "AI and Cognitive Neuroscience".
Collapse
Affiliation(s)
- Zhiyi Chen
- Experimental Research Center for Medical and Psychological Science (ERC-MPS), School of Psychology, Third Military Medical University, No.30 Gao Tan-Yan Main Street, Shapingba, Chongqing, 400038, People's Republic of China.
- Faculty of Psychology, Southwest University, Chongqing, People's Republic of China.
| | - Ali Yadollahpour
- Department of Psychology, University of Sheffield, Sheffield, UK.
| |
Collapse
|
11
|
Ratzon A, Derdikman D, Barak O. Representational drift as a result of implicit regularization. eLife 2024; 12:RP90069. [PMID: 38695551 PMCID: PMC11065423 DOI: 10.7554/elife.90069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/04/2024] Open
Abstract
Recent studies show that, even in constant environments, the tuning of single neurons changes over time in a variety of brain regions. This representational drift has been suggested to be a consequence of continuous learning under noise, but its properties are still not fully understood. To investigate the underlying mechanism, we trained an artificial network on a simplified navigational task. The network quickly reached a state of high performance, and many units exhibited spatial tuning. We then continued training the network and noticed that the activity became sparser with time. Initial learning was orders of magnitude faster than ensuing sparsification. This sparsification is consistent with recent results in machine learning, in which networks slowly move within their solution space until they reach a flat area of the loss function. We analyzed four datasets from different labs, all demonstrating that CA1 neurons become sparser and more spatially informative with exposure to the same environment. We conclude that learning is divided into three overlapping phases: (i) Fast familiarity with the environment; (ii) slow implicit regularization; and (iii) a steady state of null drift. The variability in drift dynamics opens the possibility of inferring learning algorithms from observations of drift statistics.
Collapse
Affiliation(s)
- Aviv Ratzon
- Rappaport Faculty of Medicine, Technion - Israel Institute of TechnologyHaifaIsrael
- Network Biology Research Laboratory, Technion - Israel Institute of TechnologyHaifaIsrael
| | - Dori Derdikman
- Rappaport Faculty of Medicine, Technion - Israel Institute of TechnologyHaifaIsrael
| | - Omri Barak
- Rappaport Faculty of Medicine, Technion - Israel Institute of TechnologyHaifaIsrael
- Network Biology Research Laboratory, Technion - Israel Institute of TechnologyHaifaIsrael
| |
Collapse
|
12
|
Ruffle JK, Gray RJ, Mohinta S, Pombo G, Kaul C, Hyare H, Rees G, Nachev P. Computational limits to the legibility of the imaged human brain. Neuroimage 2024; 291:120600. [PMID: 38569979 DOI: 10.1016/j.neuroimage.2024.120600] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 03/08/2024] [Accepted: 03/31/2024] [Indexed: 04/05/2024] Open
Abstract
Our knowledge of the organisation of the human brain at the population-level is yet to translate into power to predict functional differences at the individual-level, limiting clinical applications and casting doubt on the generalisability of inferred mechanisms. It remains unknown whether the difficulty arises from the absence of individuating biological patterns within the brain, or from limited power to access them with the models and compute at our disposal. Here we comprehensively investigate the resolvability of such patterns with data and compute at unprecedented scale. Across 23 810 unique participants from UK Biobank, we systematically evaluate the predictability of 25 individual biological characteristics, from all available combinations of structural and functional neuroimaging data. Over 4526 GPU*hours of computation, we train, optimize, and evaluate out-of-sample 700 individual predictive models, including fully-connected feed-forward neural networks of demographic, psychological, serological, chronic disease, and functional connectivity characteristics, and both uni- and multi-modal 3D convolutional neural network models of macro- and micro-structural brain imaging. We find a marked discrepancy between the high predictability of sex (balanced accuracy 99.7%), age (mean absolute error 2.048 years, R2 0.859), and weight (mean absolute error 2.609Kg, R2 0.625), for which we set new state-of-the-art performance, and the surprisingly low predictability of other characteristics. Neither structural nor functional imaging predicted an individual's psychology better than the coincidence of common chronic disease (p < 0.05). Serology predicted chronic disease (p < 0.05) and was best predicted by it (p < 0.001), followed by structural neuroimaging (p < 0.05). Our findings suggest either more informative imaging or more powerful models will be needed to decipher individual level characteristics from the human brain. We make our models and code openly available.
Collapse
Affiliation(s)
- James K Ruffle
- Queen Square Institute of Neurology, University College London, London, United Kingdom.
| | - Robert J Gray
- Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Samia Mohinta
- Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Guilherme Pombo
- Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Chaitanya Kaul
- School of Computing Science, University of Glasgow, Glasgow, United Kingdom
| | - Harpreet Hyare
- Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Geraint Rees
- Queen Square Institute of Neurology, University College London, London, United Kingdom
| | - Parashkev Nachev
- Queen Square Institute of Neurology, University College London, London, United Kingdom.
| |
Collapse
|
13
|
Cowley BR, Calhoun AJ, Rangarajan N, Ireland E, Turner MH, Pillow JW, Murthy M. Mapping model units to visual neurons reveals population code for social behaviour. Nature 2024; 629:1100-1108. [PMID: 38778103 PMCID: PMC11136655 DOI: 10.1038/s41586-024-07451-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2022] [Accepted: 04/19/2024] [Indexed: 05/25/2024]
Abstract
The rich variety of behaviours observed in animals arises through the interplay between sensory processing and motor control. To understand these sensorimotor transformations, it is useful to build models that predict not only neural responses to sensory input1-5 but also how each neuron causally contributes to behaviour6,7. Here we demonstrate a novel modelling approach to identify a one-to-one mapping between internal units in a deep neural network and real neurons by predicting the behavioural changes that arise from systematic perturbations of more than a dozen neuronal cell types. A key ingredient that we introduce is 'knockout training', which involves perturbing the network during training to match the perturbations of the real neurons during behavioural experiments. We apply this approach to model the sensorimotor transformations of Drosophila melanogaster males during a complex, visually guided social behaviour8-11. The visual projection neurons at the interface between the optic lobe and central brain form a set of discrete channels12, and prior work indicates that each channel encodes a specific visual feature to drive a particular behaviour13,14. Our model reaches a different conclusion: combinations of visual projection neurons, including those involved in non-social behaviours, drive male interactions with the female, forming a rich population code for behaviour. Overall, our framework consolidates behavioural effects elicited from various neural perturbations into a single, unified model, providing a map from stimulus to neuronal cell type to behaviour, and enabling future incorporation of wiring diagrams of the brain15 into the model.
Collapse
Affiliation(s)
- Benjamin R Cowley
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, USA.
| | - Adam J Calhoun
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | | | - Elise Ireland
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Maxwell H Turner
- Department of Neurobiology, Stanford University, Stanford, CA, USA
| | - Jonathan W Pillow
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Mala Murthy
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
| |
Collapse
|
14
|
Henkes A, Eshraghian JK, Wessels H. Spiking neural networks for nonlinear regression. ROYAL SOCIETY OPEN SCIENCE 2024; 11:231606. [PMID: 38699557 PMCID: PMC11062414 DOI: 10.1098/rsos.231606] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 01/25/2024] [Accepted: 02/12/2024] [Indexed: 05/05/2024]
Abstract
Spiking neural networks (SNN), also often referred to as the third generation of neural networks, carry the potential for a massive reduction in memory and energy consumption over traditional, second-generation neural networks. Inspired by the undisputed efficiency of the human brain, they introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. Energy efficiency plays a crucial role in many engineering applications, for instance, in structural health monitoring. Machine learning in engineering contexts, especially in data-driven mechanics, focuses on regression. While regression with SNN has already been discussed in a variety of publications, in this contribution, we provide a novel formulation for its accuracy and energy efficiency. In particular, a network topology for decoding binary spike trains to real numbers is introduced, using the membrane potential of spiking neurons. Several different spiking neural architectures, ranging from simple spiking feed-forward to complex spiking long short-term memory neural networks, are derived. Since the proposed architectures do not contain any dense layers, they exploit the full potential of SNN in terms of energy efficiency. At the same time, the accuracy of the proposed SNN architectures is demonstrated by numerical examples, namely different material models. Linear and nonlinear, as well as history-dependent material models, are examined. While this contribution focuses on mechanical examples, the interested reader may regress any custom function by adapting the published source code.
Collapse
Affiliation(s)
- Alexander Henkes
- Computational Mechanics Group, ETH Zurich, Zurich, Switzerland
- Division Data-Driven Modeling of Mechanical Systems, Technical University Braunschweig, Braunschweig, Germany
| | - Jason K. Eshraghian
- Department of Electrical and Computer Engineering, University of California, Santa Cruz, CA, USA
| | - Henning Wessels
- Division Data-Driven Modeling of Mechanical Systems, Technical University Braunschweig, Braunschweig, Germany
| |
Collapse
|
15
|
Wang Y, Wang Z, Guo X, Cao Y, Xing H, Wang Y, Xing B, Wang Y, Yao Y, Ma W. Artificial neural network identified a 20-gene panel in predicting immunotherapy response and survival benefits after anti-PD1/PD-L1 treatment in glioblastoma patients. Cancer Med 2024; 13:e7218. [PMID: 38733169 PMCID: PMC11087814 DOI: 10.1002/cam4.7218] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2023] [Revised: 04/03/2024] [Accepted: 04/15/2024] [Indexed: 05/13/2024] Open
Abstract
BACKGROUND Immune checkpoint inhibitors (ICIs) are a promising immunotherapy approach, but glioblastoma clinical trials have not yielded satisfactory results. OBJECTIVE To screen glioblastoma patients who may benefit from immunotherapy. METHODS Eighty-one patients receiving anti-PD1/PD-L1 treatment from a large-scale clinical trial and 364 patients without immunotherapy from The Cancer Genome Atlas (TCGA) were included. Patients in the ICI-treated cohort were divided into responders and nonresponders according to overall survival (OS), and the most critical responder-relevant features were screened using random forest (RF). We constructed an artificial neural network (ANN) model and verified its predictive value with immunotherapy response and OS. RESULTS We defined two groups of ICI-treated glioblastoma patients with large differences in survival benefits as nonresponders (OS ≤6 months, n = 18) and responders (OS ≥17 months, n = 8). No differentially mutated genes were observed between responders and nonresponders. We performed RF analysis to select the most critical responder-relevant features and developed an ANN with 20 input variables, five hidden neurons and one output neuron. Receiver operating characteristic analysis and the DeLong test demonstrated that the ANN had the best performance in predicting responders, with an AUC of 0.97. Survival analysis indicated that ANN-predicted responders had significantly better OS rates than nonresponders. CONCLUSION The 20-gene panel developed by the ANN could be a promising biomarker for predicting immunotherapy response and prognostic benefits in ICI-treated GBM patients and may guide oncologists to accurately select potential responders for the preferential use of ICIs.
Collapse
Affiliation(s)
- Yaning Wang
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Zihao Wang
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Xiaopeng Guo
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Yaning Cao
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Hao Xing
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Yuekun Wang
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Bing Xing
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Yu Wang
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Yong Yao
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| | - Wenbin Ma
- Department of Neurosurgery, Center for Malignant Brain Tumors, National Glioma MDT Alliance, Peking Union Medical College HospitalChinese Academy of Medical Sciences and Peking UnionMedical CollegeBeijingChina
| |
Collapse
|
16
|
Galloni AR, Yuan Y, Zhu M, Yu H, Bisht RS, Wu CTM, Grienberger C, Ramanathan S, Milstein AD. Neuromorphic one-shot learning utilizing a phase-transition material. Proc Natl Acad Sci U S A 2024; 121:e2318362121. [PMID: 38630718 PMCID: PMC11047090 DOI: 10.1073/pnas.2318362121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 03/25/2024] [Indexed: 04/19/2024] Open
Abstract
Design of hardware based on biological principles of neuronal computation and plasticity in the brain is a leading approach to realizing energy- and sample-efficient AI and learning machines. An important factor in selection of the hardware building blocks is the identification of candidate materials with physical properties suitable to emulate the large dynamic ranges and varied timescales of neuronal signaling. Previous work has shown that the all-or-none spiking behavior of neurons can be mimicked by threshold switches utilizing material phase transitions. Here, we demonstrate that devices based on a prototypical metal-insulator-transition material, vanadium dioxide (VO2), can be dynamically controlled to access a continuum of intermediate resistance states. Furthermore, the timescale of their intrinsic relaxation can be configured to match a range of biologically relevant timescales from milliseconds to seconds. We exploit these device properties to emulate three aspects of neuronal analog computation: fast (~1 ms) spiking in a neuronal soma compartment, slow (~100 ms) spiking in a dendritic compartment, and ultraslow (~1 s) biochemical signaling involved in temporal credit assignment for a recently discovered biological mechanism of one-shot learning. Simulations show that an artificial neural network using properties of VO2 devices to control an agent navigating a spatial environment can learn an efficient path to a reward in up to fourfold fewer trials than standard methods. The phase relaxations described in our study may be engineered in a variety of materials and can be controlled by thermal, electrical, or optical stimuli, suggesting further opportunities to emulate biological learning in neuromorphic hardware.
Collapse
Affiliation(s)
- Alessandro R. Galloni
- Department of Neuroscience and Cell Biology, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ08854
- Center for Advanced Biotechnology and Medicine, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Yifan Yuan
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Minning Zhu
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Haoming Yu
- School of Materials Engineering, Purdue University, West Lafayette, IN47907
| | - Ravindra S. Bisht
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Chung-Tse Michael Wu
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Christine Grienberger
- Department of Neuroscience, Brandeis University, Waltham, MA02453
- Department of Biology and Volen National Center for Complex Systems, Brandeis University, Waltham, MA02453
| | - Shriram Ramanathan
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Aaron D. Milstein
- Department of Neuroscience and Cell Biology, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ08854
- Center for Advanced Biotechnology and Medicine, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| |
Collapse
|
17
|
Lippl S, Kay K, Jensen G, Ferrera VP, Abbott L. A mathematical theory of relational generalization in transitive inference. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.08.22.554287. [PMID: 37662223 PMCID: PMC10473627 DOI: 10.1101/2023.08.22.554287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
Humans and animals routinely infer relations between different items or events and generalize these relations to novel combinations of items. This allows them to respond appropriately to radically novel circumstances and is fundamental to advanced cognition. However, how learning systems (including the brain) can implement the necessary inductive biases has been unclear. Here we investigated transitive inference (TI), a classic relational task paradigm in which subjects must learn a relation (A > B and B > C) and generalize it to new combinations of items (A > C). Through mathematical analysis, we found that a broad range of biologically relevant learning models (e.g. gradient flow or ridge regression) perform TI successfully and recapitulate signature behavioral patterns long observed in living subjects. First, we found that models with item-wise additive representations automatically encode transitive relations. Second, for more general representations, a single scalar "conjunctivity factor" determines model behavior on TI and, further, the principle of norm minimization (a standard statistical inductive bias) enables models with fixed, partly conjunctive representations to generalize transitively. Finally, neural networks in the "rich regime," which enables representation learning and has been found to improve generalization, unexpectedly show poor generalization and anomalous behavior. We find that such networks implement a form of norm minimization (over hidden weights) that yields a local encoding mechanism lacking transitivity. Our findings show how minimal statistical learning principles give rise to a classical relational inductive bias (transitivity), explain empirically observed behaviors, and establish a formal approach to understanding the neural basis of relational abstraction.
Collapse
Affiliation(s)
- Samuel Lippl
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, NY
- Center for Theoretical Neuroscience, Columbia University, NY
- Department of Neuroscience, Columbia University Medical Center, NY
| | - Kenneth Kay
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, NY
- Center for Theoretical Neuroscience, Columbia University, NY
- Grossman Center for the Statistics of Mind, Columbia University, NY
| | - Greg Jensen
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, NY
- Department of Neuroscience, Columbia University Medical Center, NY
- Department of Psychology at Reed College, OR
| | - Vincent P. Ferrera
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, NY
- Department of Neuroscience, Columbia University Medical Center, NY
- Department of Psychiatry, Columbia University Medical Center, NY
| | - L.F. Abbott
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, NY
- Center for Theoretical Neuroscience, Columbia University, NY
- Department of Neuroscience, Columbia University Medical Center, NY
| |
Collapse
|
18
|
Branchi I. Uncovering the determinants of brain functioning, behavior and their interplay in the light of context. Eur J Neurosci 2024. [PMID: 38558227 DOI: 10.1111/ejn.16331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Accepted: 03/07/2024] [Indexed: 04/04/2024]
Abstract
Notwithstanding the huge progress in molecular and cellular neuroscience, our ability to understand the brain and develop effective treatments promoting mental health is still limited. This can be partially ascribed to the reductionist, deterministic and mechanistic approaches in neuroscience that struggle with the complexity of the central nervous system. Here, I introduce the Context theory of constrained systems proposing a novel role of contextual factors and genetic, molecular and neural substrates in determining brain functioning and behavior. This theory entails key conceptual implications. First, context is the main driver of behavior and mental states. Second, substrates, from genes to brain areas, have no direct causal link to complex behavioral responses as they can be combined in multiple ways to produce the same response and different responses can impinge on the same substrates. Third, context and biological substrates play distinct roles in determining behavior: context drives behavior, substrates constrain the behavioral repertoire that can be implemented. Fourth, since behavior is the interface between the central nervous system and the environment, it is a privileged level of control and orchestration of brain functioning. Such implications are illustrated through the Kitchen metaphor of the brain. This theoretical framework calls for the revision of key concepts in neuroscience and psychiatry, including causality, specificity and individuality. Moreover, at the clinical level, it proposes treatments inducing behavioral changes through contextual interventions as having the highest impact to reorganize the complexity of the human mind and to achieve a long-lasting improvement in mental health.
Collapse
Affiliation(s)
- Igor Branchi
- Center for Behavioral Sciences and Mental Health, Istituto Superiore di Sanità, Rome, Italy
| |
Collapse
|
19
|
Heinen R, Bierbrauer A, Wolf OT, Axmacher N. Representational formats of human memory traces. Brain Struct Funct 2024; 229:513-529. [PMID: 37022435 PMCID: PMC10978732 DOI: 10.1007/s00429-023-02636-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 03/28/2023] [Indexed: 04/07/2023]
Abstract
Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.
Collapse
Affiliation(s)
- Rebekka Heinen
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany.
| | - Anne Bierbrauer
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
- Institute for Systems Neuroscience, Medical Center Hamburg-Eppendorf, Martinistraße 52, 20251, Hamburg, Germany
| | - Oliver T Wolf
- Department of Cognitive Psychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
| | - Nikolai Axmacher
- Department of Neuropsychology, Institute of Cognitive Neuroscience, Faculty of Psychology, Ruhr University Bochum, Universitätsstraße 150, 44801, Bochum, Germany
| |
Collapse
|
20
|
Kay K, Biderman N, Khajeh R, Beiran M, Cueva CJ, Shohamy D, Jensen G, Wei XX, Ferrera VP, Abbott LF. Emergent neural dynamics and geometry for generalization in a transitive inference task. PLoS Comput Biol 2024; 20:e1011954. [PMID: 38662797 PMCID: PMC11125559 DOI: 10.1371/journal.pcbi.1011954] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 05/24/2024] [Accepted: 02/28/2024] [Indexed: 05/25/2024] Open
Abstract
Relational cognition-the ability to infer relationships that generalize to novel combinations of objects-is fundamental to human and animal intelligence. Despite this importance, it remains unclear how relational cognition is implemented in the brain due in part to a lack of hypotheses and predictions at the levels of collective neural activity and behavior. Here we discovered, analyzed, and experimentally tested neural networks (NNs) that perform transitive inference (TI), a classic relational task (if A > B and B > C, then A > C). We found NNs that (i) generalized perfectly, despite lacking overt transitive structure prior to training, (ii) generalized when the task required working memory (WM), a capacity thought to be essential to inference in the brain, (iii) emergently expressed behaviors long observed in living subjects, in addition to a novel order-dependent behavior, and (iv) expressed different task solutions yielding alternative behavioral and neural predictions. Further, in a large-scale experiment, we found that human subjects performing WM-based TI showed behavior inconsistent with a class of NNs that characteristically expressed an intuitive task solution. These findings provide neural insights into a classical relational ability, with wider implications for how the brain realizes relational cognition.
Collapse
Affiliation(s)
- Kenneth Kay
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
- Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
| | - Natalie Biderman
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
| | - Ramin Khajeh
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
| | - Manuel Beiran
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
| | - Christopher J. Cueva
- Department of Brain and Cognitive Sciences, MIT, Cambridge, Massachusetts, United States of America
| | - Daphna Shohamy
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
- The Kavli Institute for Brain Science, Columbia University, New York, New York, United States of America
| | - Greg Jensen
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
- Department of Psychology at Reed College, Portland, Oregon, United States of America
| | - Xue-Xin Wei
- Departments of Neuroscience and Psychology, The University of Texas at Austin, Austin, Texas, United States of America
| | - Vincent P. Ferrera
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
- Department of Psychiatry, Columbia University Medical Center, New York, New York, United States of America
| | - LF Abbott
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
- The Kavli Institute for Brain Science, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
| |
Collapse
|
21
|
Goldstein A, Grinstein-Dabush A, Schain M, Wang H, Hong Z, Aubrey B, Schain M, Nastase SA, Zada Z, Ham E, Feder A, Gazula H, Buchnik E, Doyle W, Devore S, Dugan P, Reichart R, Friedman D, Brenner M, Hassidim A, Devinsky O, Flinker A, Hasson U. Alignment of brain embeddings and artificial contextual embeddings in natural language points to common geometric patterns. Nat Commun 2024; 15:2768. [PMID: 38553456 PMCID: PMC10980748 DOI: 10.1038/s41467-024-46631-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2022] [Accepted: 03/04/2024] [Indexed: 04/02/2024] Open
Abstract
Contextual embeddings, derived from deep language models (DLMs), provide a continuous vectorial representation of language. This embedding space differs fundamentally from the symbolic representations posited by traditional psycholinguistics. We hypothesize that language areas in the human brain, similar to DLMs, rely on a continuous embedding space to represent language. To test this hypothesis, we densely record the neural activity patterns in the inferior frontal gyrus (IFG) of three participants using dense intracranial arrays while they listened to a 30-minute podcast. From these fine-grained spatiotemporal neural recordings, we derive a continuous vectorial representation for each word (i.e., a brain embedding) in each patient. Using stringent zero-shot mapping we demonstrate that brain embeddings in the IFG and the DLM contextual embedding space have common geometric patterns. The common geometric patterns allow us to predict the brain embedding in IFG of a given left-out word based solely on its geometrical relationship to other non-overlapping words in the podcast. Furthermore, we show that contextual embeddings capture the geometry of IFG embeddings better than static word embeddings. The continuous brain embedding space exposes a vector-based neural code for natural language processing in the human brain.
Collapse
Affiliation(s)
- Ariel Goldstein
- Business School, Data Science department and Cognitive Department, Hebrew University, Jerusalem, Israel.
- Google Research, Tel Aviv, Israel.
| | | | | | - Haocheng Wang
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Zhuoqiao Hong
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Bobbi Aubrey
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
- New York University Grossman School of Medicine, New York, NY, USA
| | | | - Samuel A Nastase
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Zaid Zada
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | - Eric Ham
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | | | - Harshvardhan Gazula
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| | | | - Werner Doyle
- New York University Grossman School of Medicine, New York, NY, USA
| | - Sasha Devore
- New York University Grossman School of Medicine, New York, NY, USA
| | - Patricia Dugan
- New York University Grossman School of Medicine, New York, NY, USA
| | - Roi Reichart
- Faculty of Industrial Engineering and Management, Technion, Israel Institute of Technology, Haifa, Israel
| | - Daniel Friedman
- New York University Grossman School of Medicine, New York, NY, USA
| | - Michael Brenner
- Google Research, Tel Aviv, Israel
- School of Engineering and Applied Science, Harvard University, Cambridge, MA, USA
| | | | - Orrin Devinsky
- New York University Grossman School of Medicine, New York, NY, USA
| | - Adeen Flinker
- New York University Grossman School of Medicine, New York, NY, USA
- New York University Tandon School of Engineering, Brooklyn, NY, USA
| | - Uri Hasson
- Google Research, Tel Aviv, Israel
- Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ, USA
| |
Collapse
|
22
|
Marin Vargas A, Bisi A, Chiappa AS, Versteeg C, Miller LE, Mathis A. Task-driven neural network models predict neural dynamics of proprioception. Cell 2024; 187:1745-1761.e19. [PMID: 38518772 DOI: 10.1016/j.cell.2024.02.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2023] [Revised: 12/06/2023] [Accepted: 02/27/2024] [Indexed: 03/24/2024]
Abstract
Proprioception tells the brain the state of the body based on distributed sensory neurons. Yet, the principles that govern proprioceptive processing are poorly understood. Here, we employ a task-driven modeling approach to investigate the neural code of proprioceptive neurons in cuneate nucleus (CN) and somatosensory cortex area 2 (S1). We simulated muscle spindle signals through musculoskeletal modeling and generated a large-scale movement repertoire to train neural networks based on 16 hypotheses, each representing different computational goals. We found that the emerging, task-optimized internal representations generalize from synthetic data to predict neural dynamics in CN and S1 of primates. Computational tasks that aim to predict the limb position and velocity were the best at predicting the neural activity in both areas. Since task optimization develops representations that better predict neural activity during active than passive movements, we postulate that neural activity in the CN and S1 is top-down modulated during goal-directed movements.
Collapse
Affiliation(s)
- Alessandro Marin Vargas
- Brain Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland; NeuroX Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland
| | - Axel Bisi
- Brain Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland; NeuroX Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland
| | - Alberto S Chiappa
- Brain Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland; NeuroX Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland
| | - Chris Versteeg
- Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA; Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA; Department of Biomedical Engineering, McCormick School of Engineering, Northwestern University, Evanston, IL 60208, USA; Shirley Ryan AbilityLab, Chicago, IL 60611, USA
| | - Lee E Miller
- Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA; Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA; Department of Biomedical Engineering, McCormick School of Engineering, Northwestern University, Evanston, IL 60208, USA; Shirley Ryan AbilityLab, Chicago, IL 60611, USA
| | - Alexander Mathis
- Brain Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland; NeuroX Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland.
| |
Collapse
|
23
|
Juliani A, Safron A, Kanai R. Deep CANALs: a deep learning approach to refining the canalization theory of psychopathology. Neurosci Conscious 2024; 2024:niae005. [PMID: 38533457 PMCID: PMC10965250 DOI: 10.1093/nc/niae005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2023] [Revised: 01/15/2024] [Accepted: 01/19/2024] [Indexed: 03/28/2024] Open
Abstract
Psychedelic therapy has seen a resurgence of interest in the last decade, with promising clinical outcomes for the treatment of a variety of psychopathologies. In response to this success, several theoretical models have been proposed to account for the positive therapeutic effects of psychedelics. One of the more prominent models is "RElaxed Beliefs Under pSychedelics," which proposes that psychedelics act therapeutically by relaxing the strength of maladaptive high-level beliefs encoded in the brain. The more recent "CANAL" model of psychopathology builds on the explanatory framework of RElaxed Beliefs Under pSychedelics by proposing that canalization (the development of overly rigid belief landscapes) may be a primary factor in psychopathology. Here, we make use of learning theory in deep neural networks to develop a series of refinements to the original CANAL model. Our primary theoretical contribution is to disambiguate two separate optimization landscapes underlying belief representation in the brain and describe the unique pathologies which can arise from the canalization of each. Along each dimension, we identify pathologies of either too much or too little canalization, implying that the construct of canalization does not have a simple linear correlation with the presentation of psychopathology. In this expanded paradigm, we demonstrate the ability to make novel predictions regarding what aspects of psychopathology may be amenable to psychedelic therapy, as well as what forms of psychedelic therapy may ultimately be most beneficial for a given individual.
Collapse
Affiliation(s)
- Arthur Juliani
- Microsoft Research , Microsoft, 300 Lafayette St, New York, NY 10012, USA
| | - Adam Safron
- Department of Psychiatry and Behavioral Sciences, Johns Hopkins University, 600 N Wolfe St, Baltimore, MD 21205, USA
| | - Ryota Kanai
- Neurotechnology R & D Unit, Araya Inc, 6F Sanpo Sakuma Building, 1-11 Kandasakumacho, Chiyoda-ku, Tokyo 101-0025, Japan
| |
Collapse
|
24
|
McMullin MA, Kumar R, Higgins NC, Gygi B, Elhilali M, Snyder JS. Preliminary Evidence for Global Properties in Human Listeners During Natural Auditory Scene Perception. Open Mind (Camb) 2024; 8:333-365. [PMID: 38571530 PMCID: PMC10990578 DOI: 10.1162/opmi_a_00131] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Accepted: 02/10/2024] [Indexed: 04/05/2024] Open
Abstract
Theories of auditory and visual scene analysis suggest the perception of scenes relies on the identification and segregation of objects within it, resembling a detail-oriented processing style. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. It is our understanding that a similar line of research has not been explored in the auditory domain; therefore, we evaluated the contributions of high-level global and low-level acoustic information to auditory scene perception. An additional aim was to increase the field's ecological validity by using and making available a new collection of high-quality auditory scenes. Participants rated scenes on 8 global properties (e.g., open vs. enclosed) and an acoustic analysis evaluated which low-level features predicted the ratings. We submitted the acoustic measures and average ratings of the global properties to separate exploratory factor analyses (EFAs). The EFA of the acoustic measures revealed a seven-factor structure explaining 57% of the variance in the data, while the EFA of the global property measures revealed a two-factor structure explaining 64% of the variance in the data. Regression analyses revealed each global property was predicted by at least one acoustic variable (R2 = 0.33-0.87). These findings were extended using deep neural network models where we examined correlations between human ratings of global properties and deep embeddings of two computational models: an object-based model and a scene-based model. The results support that participants' ratings are more strongly explained by a global analysis of the scene setting, though the relationship between scene perception and auditory perception is multifaceted, with differing correlation patterns evident between the two models. Taken together, our results provide evidence for the ability to perceive auditory scenes from a global perspective. Some of the acoustic measures predicted ratings of global scene perception, suggesting representations of auditory objects may be transformed through many stages of processing in the ventral auditory stream, similar to what has been proposed in the ventral visual stream. These findings and the open availability of our scene collection will make future studies on perception, attention, and memory for natural auditory scenes possible.
Collapse
Affiliation(s)
| | - Rohit Kumar
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
| | - Nathan C. Higgins
- Department of Communication Sciences & Disorders, University of South Florida, Tampa, FL, USA
| | - Brian Gygi
- East Bay Institute for Research and Education, Martinez, CA, USA
| | - Mounya Elhilali
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
| | - Joel S. Snyder
- Department of Psychology, University of Nevada, Las Vegas, Las Vegas, NV, USA
| |
Collapse
|
25
|
Wolff M, Halassa MM. The mediodorsal thalamus in executive control. Neuron 2024; 112:893-908. [PMID: 38295791 DOI: 10.1016/j.neuron.2024.01.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Revised: 11/15/2023] [Accepted: 01/03/2024] [Indexed: 03/23/2024]
Abstract
Executive control, the ability to organize thoughts and action plans in real time, is a defining feature of higher cognition. Classical theories have emphasized cortical contributions to this process, but recent studies have reinvigorated interest in the role of the thalamus. Although it is well established that local thalamic damage diminishes cognitive capacity, such observations have been difficult to inform functional models. Recent progress in experimental techniques is beginning to enrich our understanding of the anatomical, physiological, and computational substrates underlying thalamic engagement in executive control. In this review, we discuss this progress and particularly focus on the mediodorsal thalamus, which regulates the activity within and across frontal cortical areas. We end with a synthesis that highlights frontal thalamocortical interactions in cognitive computations and discusses its functional implications in normal and pathological conditions.
Collapse
Affiliation(s)
- Mathieu Wolff
- University of Bordeaux, CNRS, INCIA, UMR 5287, 33000 Bordeaux, France.
| | - Michael M Halassa
- Department of Neuroscience, Tufts University School of Medicine, Boston, MA, USA; Department of Psychiatry, Tufts University School of Medicine, Boston, MA, USA.
| |
Collapse
|
26
|
Feldman J. Visual perception: On the trail of high-level shape aftereffects. Curr Biol 2024; 34:R195-R197. [PMID: 38471446 DOI: 10.1016/j.cub.2024.01.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/14/2024]
Abstract
The representation of visual shape is a critical component of our perception of the objects around us. A new study exploited shape aftereffects to reveal the high-dimensional space of geometric features our brains use to represent shape.
Collapse
Affiliation(s)
- Jacob Feldman
- Department of Psychology, Center for Cognitive Science, Rutgers University, New Brunswick, NJ 08904, USA.
| |
Collapse
|
27
|
Meir Y, Tzach Y, Hodassman S, Tevet O, Kanter I. Towards a universal mechanism for successful deep learning. Sci Rep 2024; 14:5881. [PMID: 38467786 PMCID: PMC10928127 DOI: 10.1038/s41598-024-56609-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Accepted: 03/08/2024] [Indexed: 03/13/2024] Open
Abstract
Recently, the underlying mechanism for successful deep learning (DL) was presented based on a quantitative method that measures the quality of a single filter in each layer of a DL model, particularly VGG-16 trained on CIFAR-10. This method exemplifies that each filter identifies small clusters of possible output labels, with additional noise selected as labels outside the clusters. This feature is progressively sharpened with each layer, resulting in an enhanced signal-to-noise ratio (SNR), which leads to an increase in the accuracy of the DL network. In this study, this mechanism is verified for VGG-16 and EfficientNet-B0 trained on the CIFAR-100 and ImageNet datasets, and the main results are as follows. First, the accuracy and SNR progressively increase with the layers. Second, for a given deep architecture, the maximal error rate increases approximately linearly with the number of output labels. Third, similar trends were obtained for dataset labels in the range [3, 1000], thus supporting the universality of this mechanism. Understanding the performance of a single filter and its dominating features paves the way to highly dilute the deep architecture without affecting its overall accuracy, and this can be achieved by applying the filter's cluster connections (AFCC).
Collapse
Affiliation(s)
- Yuval Meir
- Department of Physics, Bar-Ilan University, 52900, Ramat-Gan, Israel
| | - Yarden Tzach
- Department of Physics, Bar-Ilan University, 52900, Ramat-Gan, Israel
| | - Shiri Hodassman
- Department of Physics, Bar-Ilan University, 52900, Ramat-Gan, Israel
| | - Ofek Tevet
- Department of Physics, Bar-Ilan University, 52900, Ramat-Gan, Israel
| | - Ido Kanter
- Department of Physics, Bar-Ilan University, 52900, Ramat-Gan, Israel.
- Gonda Interdisciplinary Brain Research Center, Bar-Ilan University, 52900, Ramat-Gan, Israel.
| |
Collapse
|
28
|
Liu P, Bo K, Ding M, Fang R. Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.04.16.537079. [PMID: 37163104 PMCID: PMC10168209 DOI: 10.1101/2023.04.16.537079] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that (1) in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and (2) lesioning these neurons by setting their output to 0 or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.
Collapse
Affiliation(s)
- Peng Liu
- J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Ke Bo
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Mingzhou Ding
- J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA
| | - Ruogu Fang
- J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA
- Center for Cognitive Aging and Memory, McKnight Brain Institute, University of Florida, Gainesville, FL, USA
| |
Collapse
|
29
|
Wheatley T, Thornton MA, Stolk A, Chang LJ. The Emerging Science of Interacting Minds. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2024; 19:355-373. [PMID: 38096443 PMCID: PMC10932833 DOI: 10.1177/17456916231200177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/13/2024]
Abstract
For over a century, psychology has focused on uncovering mental processes of a single individual. However, humans rarely navigate the world in isolation. The most important determinants of successful development, mental health, and our individual traits and preferences arise from interacting with other individuals. Social interaction underpins who we are, how we think, and how we behave. Here we discuss the key methodological challenges that have limited progress in establishing a robust science of how minds interact and the new tools that are beginning to overcome these challenges. A deep understanding of the human mind requires studying the context within which it originates and exists: social interaction.
Collapse
Affiliation(s)
- Thalia Wheatley
- Consortium for Interacting Minds, Psychological and Brain Sciences, Dartmouth, Hanover, NH USA
- Santa Fe Institute
| | - Mark A. Thornton
- Consortium for Interacting Minds, Psychological and Brain Sciences, Dartmouth, Hanover, NH USA
| | - Arjen Stolk
- Consortium for Interacting Minds, Psychological and Brain Sciences, Dartmouth, Hanover, NH USA
| | - Luke J. Chang
- Consortium for Interacting Minds, Psychological and Brain Sciences, Dartmouth, Hanover, NH USA
| |
Collapse
|
30
|
Liu P, Bo K, Ding M, Fang R. Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects. PLoS Comput Biol 2024; 20:e1011943. [PMID: 38547053 PMCID: PMC10977720 DOI: 10.1371/journal.pcbi.1011943] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Accepted: 02/24/2024] [Indexed: 04/02/2024] Open
Abstract
Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and lesioning these neurons by setting their output to zero or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.
Collapse
Affiliation(s)
- Peng Liu
- J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, Florida, United States of America
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, United States of America
| | - Ke Bo
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, United States of America
| | - Mingzhou Ding
- J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, Florida, United States of America
| | - Ruogu Fang
- J. Crayton Pruitt Family Department of Biomedical Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, Florida, United States of America
- Center for Cognitive Aging and Memory, McKnight Brain Institute, University of Florida, Gainesville, Florida, United States of America
| |
Collapse
|
31
|
Ichikawa K, Kaneko K. Bayesian inference is facilitated by modular neural networks with different time scales. PLoS Comput Biol 2024; 20:e1011897. [PMID: 38478575 PMCID: PMC10962854 DOI: 10.1371/journal.pcbi.1011897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 03/25/2024] [Accepted: 02/06/2024] [Indexed: 03/26/2024] Open
Abstract
Various animals, including humans, have been suggested to perform Bayesian inferences to handle noisy, time-varying external information. In performing Bayesian inference by the brain, the prior distribution must be acquired and represented by sampling noisy external inputs. However, the mechanism by which neural activities represent such distributions has not yet been elucidated. Our findings reveal that networks with modular structures, composed of fast and slow modules, are adept at representing this prior distribution, enabling more accurate Bayesian inferences. Specifically, the modular network that consists of a main module connected with input and output layers and a sub-module with slower neural activity connected only with the main module outperformed networks with uniform time scales. Prior information was represented specifically by the slow sub-module, which could integrate observed signals over an appropriate period and represent input means and variances. Accordingly, the neural network could effectively predict the time-varying inputs. Furthermore, by training the time scales of neurons starting from networks with uniform time scales and without modular structure, the above slow-fast modular network structure and the division of roles in which prior knowledge is selectively represented in the slow sub-modules spontaneously emerged. These results explain how the prior distribution for Bayesian inference is represented in the brain, provide insight into the relevance of modular structure with time scale hierarchy to information processing, and elucidate the significance of brain areas with slower time scales.
Collapse
Affiliation(s)
- Kohei Ichikawa
- Department of Basic Science, Graduate School of Arts and Sciences, University of Tokyo, Meguro-ku, Tokyo, Japan
| | - Kunihiko Kaneko
- Research Center for Complex Systems Biology, University of Tokyo, Bunkyo-ku, Tokyo, Japan
- The Niels Bohr Institute, University of Copenhagen, Blegdamsvej, Copenhagen, Denmark
| |
Collapse
|
32
|
Brands AM, Devore S, Devinsky O, Doyle W, Flinker A, Friedman D, Dugan P, Winawer J, Groen IIA. Temporal dynamics of short-term neural adaptation across human visual cortex. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.09.13.557378. [PMID: 37745548 PMCID: PMC10515883 DOI: 10.1101/2023.09.13.557378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/26/2023]
Abstract
Neural responses in visual cortex adapt to prolonged and repeated stimuli. While adaptation occurs across the visual cortex, it is unclear how adaptation patterns and computational mechanisms differ across the visual hierarchy. Here we characterize two signatures of short-term neural adaptation in time-varying intracranial electroencephalography (iEEG) data collected while participants viewed naturalistic image categories varying in duration and repetition interval. Ventral- and lateral-occipitotemporal cortex exhibit slower and prolonged adaptation to single stimuli and slower recovery from adaptation to repeated stimuli compared to V1-V3. For category-selective electrodes, recovery from adaptation is slower for preferred than non-preferred stimuli. To model neural adaptation we augment our delayed divisive normalization (DN) model by scaling the input strength as a function of stimulus category, enabling the model to accurately predict neural responses across multiple image categories. The model fits suggest that differences in adaptation patterns arise from slower normalization dynamics in higher visual areas interacting with differences in input strength resulting from category selectivity. Our results reveal systematic differences in temporal adaptation of neural population responses across the human visual hierarchy and show that a single computational model of history-dependent normalization dynamics, fit with area-specific parameters, accounts for these differences.
Collapse
|
33
|
Sievers B, Thornton MA. Deep social neuroscience: the promise and peril of using artificial neural networks to study the social brain. Soc Cogn Affect Neurosci 2024; 19:nsae014. [PMID: 38334747 PMCID: PMC10880882 DOI: 10.1093/scan/nsae014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 12/20/2023] [Accepted: 02/04/2024] [Indexed: 02/10/2024] Open
Abstract
This review offers an accessible primer to social neuroscientists interested in neural networks. It begins by providing an overview of key concepts in deep learning. It then discusses three ways neural networks can be useful to social neuroscientists: (i) building statistical models to predict behavior from brain activity; (ii) quantifying naturalistic stimuli and social interactions; and (iii) generating cognitive models of social brain function. These applications have the potential to enhance the clinical value of neuroimaging and improve the generalizability of social neuroscience research. We also discuss the significant practical challenges, theoretical limitations and ethical issues faced by deep learning. If the field can successfully navigate these hazards, we believe that artificial neural networks may prove indispensable for the next stage of the field's development: deep social neuroscience.
Collapse
Affiliation(s)
- Beau Sievers
- Department of Psychology, Stanford University, 420 Jane Stanford Way, Stanford, CA 94305, USA
- Department of Psychology, Harvard University, 33 Kirkland St., Cambridge, MA 02138, USA
| | - Mark A Thornton
- Department of Psychological and Brain Sciences, Dartmouth College, 6207 Moore Hall, Hanover, NH 03755, USA
| |
Collapse
|
34
|
Liu YH, Baratin A, Cornford J, Mihalas S, Shea-Brown E, Lajoie G. How connectivity structure shapes rich and lazy learning in neural circuits. ARXIV 2024:arXiv:2310.08513v2. [PMID: 37873007 PMCID: PMC10593070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 10/25/2023]
Abstract
In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity could exhibit a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights -- in particular their effective rank -- influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.
Collapse
|
35
|
Ratzon A, Derdikman D, Barak O. Representational drift as a result of implicit regularization. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.05.04.539512. [PMID: 38370656 PMCID: PMC10871206 DOI: 10.1101/2023.05.04.539512] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Recent studies show that, even in constant environments, the tuning of single neurons changes over time in a variety of brain regions. This representational drift has been suggested to be a consequence of continuous learning under noise, but its properties are still not fully understood. To investigate the underlying mechanism, we trained an artificial network on a simplified navigational task. The network quickly reached a state of high performance, and many units exhibited spatial tuning. We then continued training the network and noticed that the activity became sparser with time. Initial learning was orders of magnitude faster than ensuing sparsification. This sparsification is consistent with recent results in machine learning, in which networks slowly move within their solution space until they reach a flat area of the loss function. We analyzed four datasets from different labs, all demonstrating that CA1 neurons become sparser and more spatially informative with exposure to the same environment. We conclude that learning is divided into three overlapping phases: (i) Fast familiarity with the environment; (ii) slow implicit regularization; (iii) a steady state of null drift. The variability in drift dynamics opens the possibility of inferring learning algorithms from observations of drift statistics.
Collapse
Affiliation(s)
- Aviv Ratzon
- Rappaport Faculty of Medicine, Technion - Israel Institute of Technology, Haifa 31096, Israel
- Network Biology Research Laboratory, Technion - Israel Institute of Technology, Haifa 32000, Israel
| | - Dori Derdikman
- Rappaport Faculty of Medicine, Technion - Israel Institute of Technology, Haifa 31096, Israel
| | - Omri Barak
- Rappaport Faculty of Medicine, Technion - Israel Institute of Technology, Haifa 31096, Israel
- Network Biology Research Laboratory, Technion - Israel Institute of Technology, Haifa 32000, Israel
| |
Collapse
|
36
|
Shekhar M, Rahnev D. Human-like dissociations between confidence and accuracy in convolutional neural networks. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.01.578187. [PMID: 38352596 PMCID: PMC10862905 DOI: 10.1101/2024.02.01.578187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/25/2024]
Abstract
Prior research has shown that manipulating stimulus energy by changing both stimulus contrast and variability results in confidence-accuracy dissociations in humans. Specifically, even when performance is matched, higher stimulus energy leads to higher confidence. The most common explanation for this effect is the positive evidence heuristic where confidence neglects evidence that disconfirms the choice. However, an alternative explanation is the signal-and-variance-increase hypothesis, according to which these dissociations arise from low-level changes in the separation and variance of perceptual representations. Because artificial neural networks lack built-in confidence heuristics, they can serve as a test for the necessity of confidence heuristics in explaining confidence-accuracy dissociations. Therefore, we tested whether confidence-accuracy dissociations induced by stimulus energy manipulations emerge naturally in convolutional neural networks (CNNs). We found that, across three different energy manipulations, CNNs produced confidence-accuracy dissociations similar to those found in humans. This effect was present for a range of CNN architectures from shallow 4-layer networks to very deep ones, such as VGG-19 and ResNet -50 pretrained on ImageNet. Further, we traced back the reason for the confidence-accuracy dissociations in all CNNs to the same signal-and-variance increase that has been proposed for humans: higher stimulus energy increased the separation and variance of the CNNs' internal representations leading to higher confidence even for matched accuracy. These findings cast doubt on the necessity of the positive evidence heuristic to explain human confidence and establish CNNs as promising models for adjudicating between low-level, stimulus-driven and high-level, cognitive explanations of human behavior.
Collapse
Affiliation(s)
- Medha Shekhar
- School of Psychology, Georgia Institute of Technology, Atlanta, GA
| | - Dobromir Rahnev
- School of Psychology, Georgia Institute of Technology, Atlanta, GA
| |
Collapse
|
37
|
Chen Z, Han Y, Ma Z, Wang X, Xu S, Tang Y, Vyssotski AL, Si B, Zhan Y. A prefrontal-thalamic circuit encodes social information for social recognition. Nat Commun 2024; 15:1036. [PMID: 38310109 PMCID: PMC10838311 DOI: 10.1038/s41467-024-45376-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 01/19/2024] [Indexed: 02/05/2024] Open
Abstract
Social recognition encompasses encoding social information and distinguishing unfamiliar from familiar individuals to form social relationships. Although the medial prefrontal cortex (mPFC) is known to play a role in social behavior, how identity information is processed and by which route it is communicated in the brain remains unclear. Here we report that a ventral midline thalamic area, nucleus reuniens (Re) that has reciprocal connections with the mPFC, is critical for social recognition in male mice. In vivo single-unit recordings and decoding analysis reveal that neural populations in both mPFC and Re represent different social stimuli, however, mPFC coding capacity is stronger. We demonstrate that chemogenetic inhibitions of Re impair the mPFC-Re neural synchronization and the mPFC social coding. Projection pathway-specific inhibitions by optogenetics reveal that the reciprocal connectivity between the mPFC and the Re is necessary for social recognition. These results reveal an mPFC-thalamic circuit for social information processing.
Collapse
Affiliation(s)
- Zihao Chen
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Yechao Han
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Zheng Ma
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Xinnian Wang
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Surui Xu
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Yong Tang
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Alexei L Vyssotski
- Institute of Neuroinformatics, University of Zurich and Swiss Federal Institute of Technology (ETH), Zurich, Switzerland
| | - Bailu Si
- School of Systems Science, Beijing Normal University, Beijing, China
- Chinese Institute for Brain Research, Beijing, China
| | - Yang Zhan
- The Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
- CAS Key Laboratory of Brain Connectome and Manipulation, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
- Shenzhen-Hong Kong Institute of Brain Science, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
| |
Collapse
|
38
|
Matteucci G, Piasini E, Zoccolan D. Unsupervised learning of mid-level visual representations. Curr Opin Neurobiol 2024; 84:102834. [PMID: 38154417 DOI: 10.1016/j.conb.2023.102834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2023] [Revised: 12/03/2023] [Accepted: 12/05/2023] [Indexed: 12/30/2023]
Abstract
Recently, a confluence between trends in neuroscience and machine learning has brought a renewed focus on unsupervised learning, where sensory processing systems learn to exploit the statistical structure of their inputs in the absence of explicit training targets or rewards. Sophisticated experimental approaches have enabled the investigation of the influence of sensory experience on neural self-organization and its synaptic bases. Meanwhile, novel algorithms for unsupervised and self-supervised learning have become increasingly popular both as inspiration for theories of the brain, particularly for the function of intermediate visual cortical areas, and as building blocks of real-world learning machines. Here we review some of these recent developments, placing them in historical context and highlighting some research lines that promise exciting breakthroughs in the near future.
Collapse
Affiliation(s)
- Giulio Matteucci
- Department of Basic Neurosciences, University of Geneva, Geneva, 1206, Switzerland. https://twitter.com/giulio_matt
| | - Eugenio Piasini
- International School for Advanced Studies (SISSA), Trieste, 34136, Italy
| | - Davide Zoccolan
- International School for Advanced Studies (SISSA), Trieste, 34136, Italy.
| |
Collapse
|
39
|
Naud R, Longtin A. Connecting levels of analysis in the computational era. J Physiol 2024; 602:417-420. [PMID: 38071740 DOI: 10.1113/jp286013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 11/30/2023] [Indexed: 02/02/2024] Open
Affiliation(s)
- Richard Naud
- Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Physics, University of Ottawa, Ottawa, ON, Canada
- Center for Neural Dynamics, University of Ottawa, Ottawa, ON, Canada
- Brain and Mind Research Institute, University of Ottawa, Ottawa, ON, Canada
| | - André Longtin
- Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, ON, Canada
- Department of Physics, University of Ottawa, Ottawa, ON, Canada
- Center for Neural Dynamics, University of Ottawa, Ottawa, ON, Canada
- Brain and Mind Research Institute, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
40
|
Liu Q, Tian Y, Zhou T, Lyu K, Wang Z, Zheng Y, Liu Y, Ren J, Li J. An Explainable and Personalized Cognitive Reasoning Model Based on Knowledge Graph: Toward Decision Making for General Practice. IEEE J Biomed Health Inform 2024; 28:707-718. [PMID: 37669206 DOI: 10.1109/jbhi.2023.3312154] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/07/2023]
Abstract
General practice plays a prominent role in primary health care (PHC). However, evidence has shown that the quality of PHC is still unsatisfactory, and the accuracy of clinical diagnosis and treatment must be improved in China. Decision making tools based on artificial intelligence can help general practitioners diagnose diseases, but most existing research is not sufficiently scalable and explainable. An explainable and personalized cognitive reasoning model based on knowledge graph (CRKG) proposed in this article can provide personalized diagnosis, perform decision making in general practice, and simulate the mode of thinking of human beings utilizing patients' electronic health records (EHRs) and knowledge graph. Taking abdominal diseases as the application point, an abdominal disease knowledge graph is first constructed in a semiautomated manner. Then, the CRKG designed referring to dual process theory in cognitive science involves the update strategy of global graph representations and reasoning on a personal cognitive graph by adopting the idea of graph neural networks and attention mechanisms. For the diagnosis of diseases in general practice, the CRKG outperforms all the baselines with a precision@1 of 0.7873, recall@10 of 0.9020 and hits@10 of 0.9340. Additionally, the visualization of the reasoning process for each visit of a patient based on the knowledge graph enhances clinicians' comprehension and contributes to explainability. This study is of great importance for the exploration and application of decision making based on EHRs and knowledge graph.
Collapse
|
41
|
Barradas VR, Koike Y, Schweighofer N. Theoretical limits on the speed of learning inverse models explain the rate of adaptation in arm reaching tasks. Neural Netw 2024; 170:376-389. [PMID: 38029719 DOI: 10.1016/j.neunet.2023.10.049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 09/08/2023] [Accepted: 10/30/2023] [Indexed: 12/01/2023]
Abstract
An essential aspect of human motor learning is the formation of inverse models, which map desired actions to motor commands. Inverse models can be learned by adjusting parameters in neural circuits to minimize errors in the performance of motor tasks through gradient descent. However, the theory of gradient descent establishes limits on the learning speed. Specifically, the eigenvalues of the Hessian of the error surface around a minimum determine the maximum speed of learning in a task. Here, we use this theoretical framework to analyze the speed of learning in different inverse model learning architectures in a set of isometric arm-reaching tasks. We show theoretically that, in these tasks, the error surface and, thus the speed of learning, are determined by the shapes of the force manipulability ellipsoid of the arm and the distribution of targets in the task. In particular, rounder manipulability ellipsoids generate a rounder error surface, allowing for faster learning of the inverse model. Rounder target distributions have a similar effect. We tested these predictions experimentally in a quasi-isometric reaching task with a visuomotor transformation. The experimental results were consistent with our theoretical predictions. Furthermore, our analysis accounts for the speed of learning in previous experiments with incompatible and compatible virtual surgery tasks, and with visuomotor rotation tasks with different numbers of targets. By identifying aspects of a task that influence the speed of learning, our results provide theoretical principles for the design of motor tasks that allow for faster learning.
Collapse
Affiliation(s)
- Victor R Barradas
- Institute of Innovative Research, Tokyo Institute of Technology, 4259 R2-16 Nagatsuta-cho, Midori-ku, Yokohama, Kanagawa 226-8503, Japan.
| | - Yasuharu Koike
- Institute of Innovative Research, Tokyo Institute of Technology, 4259 R2-16 Nagatsuta-cho, Midori-ku, Yokohama, Kanagawa 226-8503, Japan
| | - Nicolas Schweighofer
- Biokinesiology and Physical Therapy, University of Southern California, 1540 Alcazar Street, CHP 155, Los Angeles, CA 90089-9006, USA
| |
Collapse
|
42
|
Stern M, Liu AJ, Balasubramanian V. Physical effects of learning. Phys Rev E 2024; 109:024311. [PMID: 38491658 DOI: 10.1103/physreve.109.024311] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Accepted: 01/31/2024] [Indexed: 03/18/2024]
Abstract
Interacting many-body physical systems ranging from neural networks in the brain to folding proteins to self-modifying electrical circuits can learn to perform diverse tasks. This learning, both in nature and in engineered systems, can occur through evolutionary selection or through dynamical rules that drive active learning from experience. Here, we show that learning in linear physical networks with weak input signals leaves architectural imprints on the Hessian of a physical system. Compared to a generic organization of the system components, (a) the effective physical dimension of the response to inputs decreases, (b) the response of physical degrees of freedom to random perturbations (or system "susceptibility") increases, and (c) the low-eigenvalue eigenvectors of the Hessian align with the task. Overall, these effects embody the typical scenario for learning processes in physical systems in the weak input regime, suggesting ways of discovering whether a physical network may have been trained.
Collapse
Affiliation(s)
- Menachem Stern
- Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Andrea J Liu
- Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Computational Biology, Flatiron Institute, Simons Foundation, New York, New York 10010, USA
| | - Vijay Balasubramanian
- Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, New Mexico 87501, USA
- Theoretische Natuurkunde, Vrije Universiteit Brussel, Pleinlaan 2, B-1050 Brussels, Belgium
| |
Collapse
|
43
|
Deperrois N, Petrovici MA, Senn W, Jordan J. Learning beyond sensations: How dreams organize neuronal representations. Neurosci Biobehav Rev 2024; 157:105508. [PMID: 38097096 DOI: 10.1016/j.neubiorev.2023.105508] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2023] [Revised: 12/05/2023] [Accepted: 12/09/2023] [Indexed: 12/25/2023]
Abstract
Semantic representations in higher sensory cortices form the basis for robust, yet flexible behavior. These representations are acquired over the course of development in an unsupervised fashion and continuously maintained over an organism's lifespan. Predictive processing theories propose that these representations emerge from predicting or reconstructing sensory inputs. However, brains are known to generate virtual experiences, such as during imagination and dreaming, that go beyond previously experienced inputs. Here, we suggest that virtual experiences may be just as relevant as actual sensory inputs in shaping cortical representations. In particular, we discuss two complementary learning principles that organize representations through the generation of virtual experiences. First, "adversarial dreaming" proposes that creative dreams support a cortical implementation of adversarial learning in which feedback and feedforward pathways engage in a productive game of trying to fool each other. Second, "contrastive dreaming" proposes that the invariance of neuronal representations to irrelevant factors of variation is acquired by trying to map similar virtual experiences together via a contrastive learning process. These principles are compatible with known cortical structure and dynamics and the phenomenology of sleep thus providing promising directions to explain cortical learning beyond the classical predictive processing paradigm.
Collapse
Affiliation(s)
| | | | - Walter Senn
- Department of Physiology, University of Bern, Bern, Switzerland
| | - Jakob Jordan
- Department of Physiology, University of Bern, Bern, Switzerland; Electrical Engineering, Yale University, New Haven, CT, United States
| |
Collapse
|
44
|
Song Y, Millidge B, Salvatori T, Lukasiewicz T, Xu Z, Bogacz R. Inferring neural activity before plasticity as a foundation for learning beyond backpropagation. Nat Neurosci 2024; 27:348-358. [PMID: 38172438 PMCID: PMC7615830 DOI: 10.1038/s41593-023-01514-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2022] [Accepted: 11/02/2023] [Indexed: 01/05/2024]
Abstract
For both humans and machines, the essence of learning is to pinpoint which components in its information processing pipeline are responsible for an error in its output, a challenge that is known as 'credit assignment'. It has long been assumed that credit assignment is best solved by backpropagation, which is also the foundation of modern machine learning. Here, we set out a fundamentally different principle on credit assignment called 'prospective configuration'. In prospective configuration, the network first infers the pattern of neural activity that should result from learning, and then the synaptic weights are modified to consolidate the change in neural activity. We demonstrate that this distinct mechanism, in contrast to backpropagation, (1) underlies learning in a well-established family of models of cortical circuits, (2) enables learning that is more efficient and effective in many contexts faced by biological organisms and (3) reproduces surprising patterns of neural activity and behavior observed in diverse human and rat learning experiments.
Collapse
Affiliation(s)
- Yuhang Song
- Department of Computer Science, University of Oxford, Oxford, UK.
- Medical Research Council Brain Network Dynamics Unit, University of Oxford, Oxford, UK.
- Fractile, Ltd., London, UK.
| | - Beren Millidge
- Medical Research Council Brain Network Dynamics Unit, University of Oxford, Oxford, UK
| | - Tommaso Salvatori
- Department of Computer Science, University of Oxford, Oxford, UK
- Institute of Logic and Computation, Vienna University of Technology, Vienna, Austria
- VERSES AI Research Lab, Los Angeles, CA, USA
| | - Thomas Lukasiewicz
- Department of Computer Science, University of Oxford, Oxford, UK.
- Institute of Logic and Computation, Vienna University of Technology, Vienna, Austria.
| | - Zhenghua Xu
- Department of Computer Science, University of Oxford, Oxford, UK.
- State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, China.
| | - Rafal Bogacz
- Medical Research Council Brain Network Dynamics Unit, University of Oxford, Oxford, UK.
| |
Collapse
|
45
|
Weng G, Clark K, Akbarian A, Noudoost B, Nategh N. Time-varying generalized linear models: characterizing and decoding neuronal dynamics in higher visual areas. Front Comput Neurosci 2024; 18:1273053. [PMID: 38348287 PMCID: PMC10859875 DOI: 10.3389/fncom.2024.1273053] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2023] [Accepted: 01/09/2024] [Indexed: 02/15/2024] Open
Abstract
To create a behaviorally relevant representation of the visual world, neurons in higher visual areas exhibit dynamic response changes to account for the time-varying interactions between external (e.g., visual input) and internal (e.g., reward value) factors. The resulting high-dimensional representational space poses challenges for precisely quantifying individual factors' contributions to the representation and readout of sensory information during a behavior. The widely used point process generalized linear model (GLM) approach provides a powerful framework for a quantitative description of neuronal processing as a function of various sensory and non-sensory inputs (encoding) as well as linking particular response components to particular behaviors (decoding), at the level of single trials and individual neurons. However, most existing variations of GLMs assume the neural systems to be time-invariant, making them inadequate for modeling nonstationary characteristics of neuronal sensitivity in higher visual areas. In this review, we summarize some of the existing GLM variations, with a focus on time-varying extensions. We highlight their applications to understanding neural representations in higher visual areas and decoding transient neuronal sensitivity as well as linking physiology to behavior through manipulation of model components. This time-varying class of statistical models provide valuable insights into the neural basis of various visual behaviors in higher visual areas and hold significant potential for uncovering the fundamental computational principles that govern neuronal processing underlying various behaviors in different regions of the brain.
Collapse
Affiliation(s)
- Geyu Weng
- Department of Biomedical Engineering, University of Utah, Salt Lake City, UT, United States
- Department of Ophthalmology and Visual Sciences, University of Utah, Salt Lake City, UT, United States
| | - Kelsey Clark
- Department of Ophthalmology and Visual Sciences, University of Utah, Salt Lake City, UT, United States
| | - Amir Akbarian
- Department of Ophthalmology and Visual Sciences, University of Utah, Salt Lake City, UT, United States
| | - Behrad Noudoost
- Department of Ophthalmology and Visual Sciences, University of Utah, Salt Lake City, UT, United States
| | - Neda Nategh
- Department of Ophthalmology and Visual Sciences, University of Utah, Salt Lake City, UT, United States
- Department of Electrical and Computer Engineering, University of Utah, Salt Lake City, UT, United States
| |
Collapse
|
46
|
Dyballa L, Rudzite AM, Hoseini MS, Thapa M, Stryker MP, Field GD, Zucker SW. Population encoding of stimulus features along the visual hierarchy. Proc Natl Acad Sci U S A 2024; 121:e2317773121. [PMID: 38227668 PMCID: PMC10823231 DOI: 10.1073/pnas.2317773121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 12/13/2023] [Indexed: 01/18/2024] Open
Abstract
The retina and primary visual cortex (V1) both exhibit diverse neural populations sensitive to diverse visual features. Yet it remains unclear how neural populations in each area partition stimulus space to span these features. One possibility is that neural populations are organized into discrete groups of neurons, with each group signaling a particular constellation of features. Alternatively, neurons could be continuously distributed across feature-encoding space. To distinguish these possibilities, we presented a battery of visual stimuli to the mouse retina and V1 while measuring neural responses with multi-electrode arrays. Using machine learning approaches, we developed a manifold embedding technique that captures how neural populations partition feature space and how visual responses correlate with physiological and anatomical properties of individual neurons. We show that retinal populations discretely encode features, while V1 populations provide a more continuous representation. Applying the same analysis approach to convolutional neural networks that model visual processing, we demonstrate that they partition features much more similarly to the retina, indicating they are more like big retinas than little brains.
Collapse
Affiliation(s)
- Luciano Dyballa
- Department of Computer Science, Yale University, New Haven, CT06511
| | | | - Mahmood S. Hoseini
- Department of Physiology, University of California, San Francisco, CA94143
| | - Mishek Thapa
- Department of Neurobiology, Duke University, Durham, NC27708
- Department of Ophthalmology, David Geffen School of Medicine, Stein Eye Institute, University of California, Los Angeles, CA90095
| | - Michael P. Stryker
- Department of Physiology, University of California, San Francisco, CA94143
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco, CA94143
| | - Greg D. Field
- Department of Neurobiology, Duke University, Durham, NC27708
- Department of Ophthalmology, David Geffen School of Medicine, Stein Eye Institute, University of California, Los Angeles, CA90095
| | - Steven W. Zucker
- Department of Computer Science, Yale University, New Haven, CT06511
- Department of Biomedical Engineering, Yale University, New Haven, CT06511
| |
Collapse
|
47
|
Suárez LE, Mihalik A, Milisav F, Marshall K, Li M, Vértes PE, Lajoie G, Misic B. Connectome-based reservoir computing with the conn2res toolbox. Nat Commun 2024; 15:656. [PMID: 38253577 PMCID: PMC10803782 DOI: 10.1038/s41467-024-44900-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 01/09/2024] [Indexed: 01/24/2024] Open
Abstract
The connection patterns of neural circuits form a complex network. How signaling in these circuits manifests as complex cognition and adaptive behaviour remains the central question in neuroscience. Concomitant advances in connectomics and artificial intelligence open fundamentally new opportunities to understand how connection patterns shape computational capacity in biological brain networks. Reservoir computing is a versatile paradigm that uses high-dimensional, nonlinear dynamical systems to perform computations and approximate cognitive functions. Here we present conn2res: an open-source Python toolbox for implementing biological neural networks as artificial neural networks. conn2res is modular, allowing arbitrary network architecture and dynamics to be imposed. The toolbox allows researchers to input connectomes reconstructed using multiple techniques, from tract tracing to noninvasive diffusion imaging, and to impose multiple dynamical systems, from spiking neurons to memristive dynamics. The versatility of the conn2res toolbox allows us to ask new questions at the confluence of neuroscience and artificial intelligence. By reconceptualizing function as computation, conn2res sets the stage for a more mechanistic understanding of structure-function relationships in brain networks.
Collapse
Affiliation(s)
- Laura E Suárez
- McConnell Brain Imaging Centre, Montréal Neurological Institute, McGill University, Montréal, QC, Canada
- Mila, Quebec Artificial Intelligence Institute, Montreal, QC, Canada
| | - Agoston Mihalik
- Department of Psychiatry, University of Cambridge, Cambridge, UK
| | - Filip Milisav
- McConnell Brain Imaging Centre, Montréal Neurological Institute, McGill University, Montréal, QC, Canada
| | - Kenji Marshall
- Department of Bioengineering, Stanford University, Stanford, CA, USA
| | - Mingze Li
- McConnell Brain Imaging Centre, Montréal Neurological Institute, McGill University, Montréal, QC, Canada
- Mila, Quebec Artificial Intelligence Institute, Montreal, QC, Canada
| | - Petra E Vértes
- Department of Psychiatry, University of Cambridge, Cambridge, UK
| | - Guillaume Lajoie
- Mila, Quebec Artificial Intelligence Institute, Montreal, QC, Canada
- Department of Mathematics and Statistics, Université de Montréal, Montreal, QC, Canada
| | - Bratislav Misic
- McConnell Brain Imaging Centre, Montréal Neurological Institute, McGill University, Montréal, QC, Canada.
| |
Collapse
|
48
|
Dalla Bella S, Janaqi S, Benoit CE, Farrugia N, Bégel V, Verga L, Harding EE, Kotz SA. Unravelling individual rhythmic abilities using machine learning. Sci Rep 2024; 14:1135. [PMID: 38212632 PMCID: PMC10784578 DOI: 10.1038/s41598-024-51257-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 01/02/2024] [Indexed: 01/13/2024] Open
Abstract
Humans can easily extract the rhythm of a complex sound, like music, and move to its regular beat, like in dance. These abilities are modulated by musical training and vary significantly in untrained individuals. The causes of this variability are multidimensional and typically hard to grasp in single tasks. To date we lack a comprehensive model capturing the rhythmic fingerprints of both musicians and non-musicians. Here we harnessed machine learning to extract a parsimonious model of rhythmic abilities, based on behavioral testing (with perceptual and motor tasks) of individuals with and without formal musical training (n = 79). We demonstrate that variability in rhythmic abilities and their link with formal and informal music experience can be successfully captured by profiles including a minimal set of behavioral measures. These findings highlight that machine learning techniques can be employed successfully to distill profiles of rhythmic abilities, and ultimately shed light on individual variability and its relationship with both formal musical training and informal musical experiences.
Collapse
Affiliation(s)
- Simone Dalla Bella
- International Laboratory for Brain, Music, and Sound Research (BRAMS), Montreal, Canada.
- Department of Psychology, University of Montreal, Pavillon Marie-Victorin, CP 6128 Succursale Centre-Ville, Montréal, QC, H3C 3J7, Canada.
- Centre for Research on Brain, Language and Music (CRBLM), Montreal, Canada.
- University of Economics and Human Sciences in Warsaw, Warsaw, Poland.
| | - Stefan Janaqi
- EuroMov Digital Health in Motion, IMT Mines Ales and University of Montpellier, Ales and Montpellier, France
| | - Charles-Etienne Benoit
- Inter-University Laboratory of Human Movement Biology, EA 7424, University Claude Bernard Lyon 1, 69 622, Villeurbanne, France
| | | | | | - Laura Verga
- Comparative Bioacoustics Group, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Department of Neuropsychology & Psychopharmacology, Faculty of Psychology and Neuroscience, Maastricht University, P.O. 616, Maastricht, 6200 MD, The Netherlands
| | - Eleanor E Harding
- Department of Otorhinolaryngology/Head and Neck Surgery, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands
| | - Sonja A Kotz
- Department of Neuropsychology & Psychopharmacology, Faculty of Psychology and Neuroscience, Maastricht University, P.O. 616, Maastricht, 6200 MD, The Netherlands.
- Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany.
| |
Collapse
|
49
|
Barabási DL, Schuhknecht GFP, Engert F. Functional neuronal circuits emerge in the absence of developmental activity. Nat Commun 2024; 15:364. [PMID: 38191595 PMCID: PMC10774424 DOI: 10.1038/s41467-023-44681-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 12/29/2023] [Indexed: 01/10/2024] Open
Abstract
The complex neuronal circuitry of the brain develops from limited information contained in the genome. After the genetic code instructs the birth of neurons, the emergence of brain regions, and the formation of axon tracts, it is believed that temporally structured spiking activity shapes circuits for behavior. Here, we challenge the learning-dominated assumption that spiking activity is required for circuit formation by quantifying its contribution to the development of visually-guided swimming in the larval zebrafish. We found that visual experience had no effect on the emergence of the optomotor response (OMR) in dark-reared zebrafish. We then raised animals while pharmacologically silencing action potentials with the sodium channel blocker tricaine. After washout of the anesthetic, fish could swim and performed with 75-90% accuracy in the OMR paradigm. Brain-wide imaging confirmed that neuronal circuits came 'online' fully tuned, without requiring activity-dependent plasticity. Thus, complex sensory-guided behaviors can emerge through activity-independent developmental mechanisms.
Collapse
Affiliation(s)
- Dániel L Barabási
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA.
- Biophysics Program, Harvard University, Cambridge, MA, USA.
| | | | - Florian Engert
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| |
Collapse
|
50
|
Thieu MK, Ayzenberg V, Lourenco SF, Kragel PA. Visual looming is a primitive for human emotion. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.08.29.555380. [PMID: 37693448 PMCID: PMC10491236 DOI: 10.1101/2023.08.29.555380] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/12/2023]
Abstract
Looming objects afford threat of collision across the animal kingdom. Defensive responses to looming and neural computations for looming detection are strikingly conserved across species. In mammals, information about rapidly approaching threats is conveyed from the retina to the midbrain superior colliculus, where variables that indicate the position and velocity of approach are computed to enable defensive behavior. Although neuroscientific theories posit that midbrain representations contribute to emotion through connectivity with distributed brain systems, it remains unknown whether a computational system for looming detection can predict both defensive behavior and phenomenal experience in humans. Here, we show that a shallow convolutional neural network based on the Drosophila visual system predicts defensive blinking to looming objects in infants and superior colliculus responses to optical expansion in adults. Further, the responses of the convolutional network to a broad array of naturalistic video clips predict self-reported emotion largely on the basis of subjective arousal. Our findings illustrate how motor and experiential components of human emotion relate to species-general systems for survival in unpredictable environments.
Collapse
|