1
|
Galloni AR, Yuan Y, Zhu M, Yu H, Bisht RS, Wu CTM, Grienberger C, Ramanathan S, Milstein AD. Neuromorphic one-shot learning utilizing a phase-transition material. Proc Natl Acad Sci U S A 2024; 121:e2318362121. [PMID: 38630718 PMCID: PMC11047090 DOI: 10.1073/pnas.2318362121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 03/25/2024] [Indexed: 04/19/2024] Open
Abstract
Design of hardware based on biological principles of neuronal computation and plasticity in the brain is a leading approach to realizing energy- and sample-efficient AI and learning machines. An important factor in selection of the hardware building blocks is the identification of candidate materials with physical properties suitable to emulate the large dynamic ranges and varied timescales of neuronal signaling. Previous work has shown that the all-or-none spiking behavior of neurons can be mimicked by threshold switches utilizing material phase transitions. Here, we demonstrate that devices based on a prototypical metal-insulator-transition material, vanadium dioxide (VO2), can be dynamically controlled to access a continuum of intermediate resistance states. Furthermore, the timescale of their intrinsic relaxation can be configured to match a range of biologically relevant timescales from milliseconds to seconds. We exploit these device properties to emulate three aspects of neuronal analog computation: fast (~1 ms) spiking in a neuronal soma compartment, slow (~100 ms) spiking in a dendritic compartment, and ultraslow (~1 s) biochemical signaling involved in temporal credit assignment for a recently discovered biological mechanism of one-shot learning. Simulations show that an artificial neural network using properties of VO2 devices to control an agent navigating a spatial environment can learn an efficient path to a reward in up to fourfold fewer trials than standard methods. The phase relaxations described in our study may be engineered in a variety of materials and can be controlled by thermal, electrical, or optical stimuli, suggesting further opportunities to emulate biological learning in neuromorphic hardware.
Collapse
Affiliation(s)
- Alessandro R. Galloni
- Department of Neuroscience and Cell Biology, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ08854
- Center for Advanced Biotechnology and Medicine, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Yifan Yuan
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Minning Zhu
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Haoming Yu
- School of Materials Engineering, Purdue University, West Lafayette, IN47907
| | - Ravindra S. Bisht
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Chung-Tse Michael Wu
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Christine Grienberger
- Department of Neuroscience, Brandeis University, Waltham, MA02453
- Department of Biology and Volen National Center for Complex Systems, Brandeis University, Waltham, MA02453
| | - Shriram Ramanathan
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Aaron D. Milstein
- Department of Neuroscience and Cell Biology, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ08854
- Center for Advanced Biotechnology and Medicine, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| |
Collapse
|
2
|
George TM, Rastogi M, de Cothi W, Clopath C, Stachenfeld K, Barry C. RatInABox, a toolkit for modelling locomotion and neuronal activity in continuous environments. eLife 2024; 13:e85274. [PMID: 38334473 PMCID: PMC10857787 DOI: 10.7554/elife.85274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Accepted: 01/03/2024] [Indexed: 02/10/2024] Open
Abstract
Generating synthetic locomotory and neural data is a useful yet cumbersome step commonly required to study theoretical models of the brain's role in spatial navigation. This process can be time consuming and, without a common framework, makes it difficult to reproduce or compare studies which each generate test data in different ways. In response, we present RatInABox, an open-source Python toolkit designed to model realistic rodent locomotion and generate synthetic neural data from spatially modulated cell types. This software provides users with (i) the ability to construct one- or two-dimensional environments with configurable barriers and visual cues, (ii) a physically realistic random motion model fitted to experimental data, (iii) rapid online calculation of neural data for many of the known self-location or velocity selective cell types in the hippocampal formation (including place cells, grid cells, boundary vector cells, head direction cells) and (iv) a framework for constructing custom cell types, multi-layer network models and data- or policy-controlled motion trajectories. The motion and neural models are spatially and temporally continuous as well as topographically sensitive to boundary conditions and walls. We demonstrate that out-of-the-box parameter settings replicate many aspects of rodent foraging behaviour such as velocity statistics and the tendency of rodents to over-explore walls. Numerous tutorial scripts are provided, including examples where RatInABox is used for decoding position from neural data or to solve a navigational reinforcement learning task. We hope this tool will significantly streamline computational research into the brain's role in navigation.
Collapse
Affiliation(s)
- Tom M George
- Sainsbury Wellcome Centre, University College LondonLondonUnited Kingdom
| | - Mehul Rastogi
- Sainsbury Wellcome Centre, University College LondonLondonUnited Kingdom
| | - William de Cothi
- Department of Cell and Developmental Biology, University College LondonLondonUnited Kingdom
| | - Claudia Clopath
- Sainsbury Wellcome Centre, University College LondonLondonUnited Kingdom
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | | | - Caswell Barry
- Department of Cell and Developmental Biology, University College LondonLondonUnited Kingdom
| |
Collapse
|
3
|
Son JY, Bhandari A, FeldmanHall O. Abstract cognitive maps of social network structure aid adaptive inference. Proc Natl Acad Sci U S A 2023; 120:e2310801120. [PMID: 37963254 PMCID: PMC10666027 DOI: 10.1073/pnas.2310801120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Accepted: 10/12/2023] [Indexed: 11/16/2023] Open
Abstract
Social navigation-such as anticipating where gossip may spread, or identifying which acquaintances can help land a job-relies on knowing how people are connected within their larger social communities. Problematically, for most social networks, the space of possible relationships is too vast to observe and memorize. Indeed, people's knowledge of these social relations is well known to be biased and error-prone. Here, we reveal that these biased representations reflect a fundamental computation that abstracts over individual relationships to enable principled inferences about unseen relationships. We propose a theory of network representation that explains how people learn inferential cognitive maps of social relations from direct observation, what kinds of knowledge structures emerge as a consequence, and why it can be beneficial to encode systematic biases into social cognitive maps. Leveraging simulations, laboratory experiments, and "field data" from a real-world network, we find that people abstract observations of direct relations (e.g., friends) into inferences of multistep relations (e.g., friends-of-friends). This multistep abstraction mechanism enables people to discover and represent complex social network structure, affording adaptive inferences across a variety of contexts, including friendship, trust, and advice-giving. Moreover, this multistep abstraction mechanism unifies a variety of otherwise puzzling empirical observations about social behavior. Our proposal generalizes the theory of cognitive maps to the fundamental computational problem of social inference, presenting a powerful framework for understanding the workings of a predictive mind operating within a complex social world.
Collapse
Affiliation(s)
- Jae-Young Son
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
| | - Apoorva Bhandari
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
| | - Oriel FeldmanHall
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
- Carney Institute for Brain Sciences, Brown University, Providence, RI02912
| |
Collapse
|
4
|
Plitt MH, Kaganovsky K, Südhof TC, Giocomo LM. Hippocampal place code plasticity in CA1 requires postsynaptic membrane fusion. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.20.567978. [PMID: 38045362 PMCID: PMC10690209 DOI: 10.1101/2023.11.20.567978] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/05/2023]
Abstract
Rapid delivery of glutamate receptors to the postsynaptic membrane via vesicle fusion is a central component of synaptic plasticity. However, it is unknown how this process supports specific neural computations during behavior. To bridge this gap, we combined conditional genetic deletion of a component of the postsynaptic membrane fusion machinery, Syntaxin3 (Stx3), in hippocampal CA1 neurons of mice with population in vivo calcium imaging. This approach revealed that Stx3 is necessary for forming the neural dynamics that support novelty processing, spatial reward memory and offline memory consolidation. In contrast, CA1 Stx3 was dispensable for maintaining aspects of the neural code that exist presynaptic to CA1 such as representations of context and space. Thus, manipulating postsynaptic membrane fusion identified computations that specifically require synaptic restructuring via membrane trafficking in CA1 and distinguished them from neural representation that could be inherited from upstream brain regions or learned through other mechanisms.
Collapse
|
5
|
Mehrotra D, Dubé L. Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus. Front Neurosci 2023; 17:1200842. [PMID: 37732307 PMCID: PMC10508350 DOI: 10.3389/fnins.2023.1200842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 08/25/2023] [Indexed: 09/22/2023] Open
Abstract
For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.
Collapse
Affiliation(s)
- Dhruv Mehrotra
- Integrated Program in Neuroscience, McGill University, Montréal, QC, Canada
- Montréal Neurological Institute, McGill University, Montréal, QC, Canada
| | - Laurette Dubé
- Desautels Faculty of Management, McGill University, Montréal, QC, Canada
- McGill Center for the Convergence of Health and Economics, McGill University, Montréal, QC, Canada
| |
Collapse
|
6
|
George TM, de Cothi W, Stachenfeld KL, Barry C. Rapid learning of predictive maps with STDP and theta phase precession. eLife 2023; 12:80663. [PMID: 36927826 PMCID: PMC10019887 DOI: 10.7554/elife.80663] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Accepted: 02/26/2023] [Indexed: 03/18/2023] Open
Abstract
The predictive map hypothesis is a promising candidate principle for hippocampal function. A favoured formalisation of this hypothesis, called the successor representation, proposes that each place cell encodes the expected state occupancy of its target location in the near future. This predictive framework is supported by behavioural as well as electrophysiological evidence and has desirable consequences for both the generalisability and efficiency of reinforcement learning algorithms. However, it is unclear how the successor representation might be learnt in the brain. Error-driven temporal difference learning, commonly used to learn successor representations in artificial agents, is not known to be implemented in hippocampal networks. Instead, we demonstrate that spike-timing dependent plasticity (STDP), a form of Hebbian learning, acting on temporally compressed trajectories known as 'theta sweeps', is sufficient to rapidly learn a close approximation to the successor representation. The model is biologically plausible - it uses spiking neurons modulated by theta-band oscillations, diffuse and overlapping place cell-like state representations, and experimentally matched parameters. We show how this model maps onto known aspects of hippocampal circuitry and explains substantial variance in the temporal difference successor matrix, consequently giving rise to place cells that demonstrate experimentally observed successor representation-related phenomena including backwards expansion on a 1D track and elongation near walls in 2D. Finally, our model provides insight into the observed topographical ordering of place field sizes along the dorsal-ventral axis by showing this is necessary to prevent the detrimental mixing of larger place fields, which encode longer timescale successor representations, with more fine-grained predictions of spatial location.
Collapse
Affiliation(s)
- Tom M George
- Sainsbury Wellcome Centre for Neural Circuits and Behaviour, University College LondonLondonUnited Kingdom
| | - William de Cothi
- Research Department of Cell and Developmental Biology, University College LondonLondonUnited Kingdom
| | | | - Caswell Barry
- Research Department of Cell and Developmental Biology, University College LondonLondonUnited Kingdom
| |
Collapse
|
7
|
Fang C, Aronov D, Abbott LF, Mackevicius EL. Neural learning rules for generating flexible predictions and computing the successor representation. eLife 2023; 12:e80680. [PMID: 36928104 PMCID: PMC10019889 DOI: 10.7554/elife.80680] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Accepted: 10/26/2022] [Indexed: 03/18/2023] Open
Abstract
The predictive nature of the hippocampus is thought to be useful for memory-guided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.
Collapse
Affiliation(s)
- Ching Fang
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - Dmitriy Aronov
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - LF Abbott
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - Emily L Mackevicius
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
- Basis Research InstituteNew YorkUnited States
| |
Collapse
|