1
|
Youn S, Hwang Y, Kim TH, Kim S, Hwang H, Park J, Kim H. Threshold learning algorithm for memristive neural network with binary switching behavior. Neural Netw 2024; 176:106355. [PMID: 38759411 DOI: 10.1016/j.neunet.2024.106355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Revised: 02/04/2024] [Accepted: 04/29/2024] [Indexed: 05/19/2024]
Abstract
On-chip learning is an effective method for adjusting artificial neural networks in neuromorphic computing systems by considering hardware intrinsic properties. However, it faces challenges due to hardware nonidealities, such as the nonlinearity of potentiation and depression and limitations on fine weight adjustment. In this study, we propose a threshold learning algorithm for a variation-tolerant ternary neural network in a memristor crossbar array. This algorithm utilizes two tightly separated resistance states in memristive devices to represent weight values. The high-resistance state (HRS) and low-resistance state (LRS) defined as read current of < 0.1 μA and > 1 μA, respectively, were successfully programmed in a 32 × 32 crossbar array, and exhibited half-normal distributions due to the programming method. To validate our approach experimentally, a 64 × 10 single-layer fully connected network were trained in the fabricated crossbar for an 8 × 8 MNIST dataset using the threshold learning algorithm, where the weight value is updated when a gradient determined by backpropagation exceeds a threshold value. Thanks to the large margin between the two states of the memristor, we observed only a 0.42 % drop in classification accuracy compared to the baseline network results. The threshold learning algorithm is expected to alleviate the programming burden and be utilized in variation-tolerant neuromorphic architectures.
Collapse
Affiliation(s)
- Sangwook Youn
- Division of Materials Science and Engineering, Seoul 04763, Korea
| | - Yeongjin Hwang
- Department of Electrical and Computer Engineering, Inha University, Incheon 22212, Korea
| | - Tae-Hyeon Kim
- Department of Semiconductor Engineering, Seoul National University of Science and Technology, Seoul 01811, Korea
| | - Sungjoon Kim
- Department of AI Semiconductor Engineering, Korea University, Sejong 30019, Korea
| | - Hwiho Hwang
- Division of Materials Science and Engineering, Seoul 04763, Korea
| | - Jinwoo Park
- Division of Materials Science and Engineering, Seoul 04763, Korea
| | - Hyungjin Kim
- Division of Materials Science and Engineering, Seoul 04763, Korea.
| |
Collapse
|
2
|
Zhou X, Zhao L, Yan C, Zhen W, Lin Y, Li L, Du G, Lu L, Zhang ST, Lu Z, Li D. Thermally stable threshold selector based on CuAg alloy for energy-efficient memory and neuromorphic computing applications. Nat Commun 2023; 14:3285. [PMID: 37280223 DOI: 10.1038/s41467-023-39033-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Accepted: 05/25/2023] [Indexed: 06/08/2023] Open
Abstract
As a promising candidate for high-density data storage and neuromorphic computing, cross-point memory arrays provide a platform to overcome the von Neumann bottleneck and accelerate neural network computation. In order to suppress the sneak-path current problem that limits their scalability and read accuracy, a two-terminal selector can be integrated at each cross-point to form the one-selector-one-memristor (1S1R) stack. In this work, we demonstrate a CuAg alloy-based, thermally stable and electroforming-free selector device with tunable threshold voltage and over 7 orders of magnitude ON/OFF ratio. A vertically stacked 64 × 64 1S1R cross-point array is further implemented by integrating the selector with SiO2-based memristors. The 1S1R devices exhibit extremely low leakage currents and proper switching characteristics, which are suitable for both storage class memory and synaptic weight storage. Finally, a selector-based leaky integrate-and-fire neuron is designed and experimentally implemented, which expands the application prospect of CuAg alloy selectors from synapses to neurons.
Collapse
Affiliation(s)
- Xi Zhou
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
- College of Information Science and Electronic Engineering, Zhejiang University, 38 Zheda Road, 310007, Hangzhou, China
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China
| | - Liang Zhao
- College of Information Science and Electronic Engineering, Zhejiang University, 38 Zheda Road, 310007, Hangzhou, China.
- Hefei Reliance Memory Ltd., Bldg. F4-11F, Innovation Industrial Park Phase II, 230088, Hefei, China.
| | - Chu Yan
- College of Information Science and Electronic Engineering, Zhejiang University, 38 Zheda Road, 310007, Hangzhou, China
| | - Weili Zhen
- High Magnetic Field Laboratory, Chinese Academy of Sciences, 230031, Hefei, China
| | - Yinyue Lin
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China
| | - Le Li
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China
| | - Guanlin Du
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China
| | - Linfeng Lu
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China
| | - Shan-Ting Zhang
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China
- Zhangjiang Laboratory, 100 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China
| | - Zhichao Lu
- Hefei Reliance Memory Ltd., Bldg. F4-11F, Innovation Industrial Park Phase II, 230088, Hefei, China
| | - Dongdong Li
- The Interdisciplinary Research Center, Shanghai Advanced Research Institute, Chinese Academy of Sciences, 99 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China.
- School of Microelectronics, University of Chinese Academy of Sciences, 19 Yuquan Road, 100049, Beijing, China.
- Zhangjiang Laboratory, 100 Haike Road, Zhangjiang Hi-Tech Park, 201210, Pudong, Shanghai, China.
| |
Collapse
|
3
|
A Memristor-Based High-Resolution A/D Converter. ELECTRONICS 2022. [DOI: 10.3390/electronics11091470] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Based on the voltage threshold adaptive memristor (VTEAM) model, this paper proposes a circuit design of a memristor-based A/D converter, which can achieve high-resolution conversion by simple configuration. For this A/D converter, there are the input voltage stage and the reference voltage stage in one conversion. According to the memristance change in the two stages, the input analog voltage is converted as the corresponding digital value. In the input voltage stage, the memristance increases from the initial memristance. Meanwhile, the counter rises its value from zero to the maximum. Next, the memristance returns to the initial memristance in the reference voltage stage. At the same time, the counting value starts to increase from zero again. Then, the input analog voltage is mapped to the eventual counting value of the reference voltage stage. The simulations of the memristor-based A/D converter demonstrate that it has good conversion performance. The proposed memristor-based A/D converter not only has more brilliant performance than the CMOS A/D converter, but also has the advantages over existing memristor-based A/D converters of anti-interference ability and high resolution.
Collapse
|
4
|
Suzuki Y, Asakawa N. Stochastic Resonance in Organic Electronic Devices. Polymers (Basel) 2022; 14:polym14040747. [PMID: 35215663 PMCID: PMC8878602 DOI: 10.3390/polym14040747] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2022] [Revised: 02/07/2022] [Accepted: 02/09/2022] [Indexed: 01/27/2023] Open
Abstract
Stochastic Resonance (SR) is a phenomenon in which noise improves the performance of a system. With the addition of noise, a weak input signal to a nonlinear system, which may exceed its threshold, is transformed into an output signal. In the other words, noise-driven signal transfer is achieved. SR has been observed in nonlinear response systems, such as biological and artificial systems, and this review will focus mainly on examples of previous studies of mathematical models and experimental realization of SR using poly(hexylthiophene)-based organic field-effect transistors (OFETs). This phenomenon may contribute to signal processing with low energy consumption. However, the generation of SR requires a noise source. Therefore, the focus is on OFETs using materials such as organic materials with unstable electrical properties and critical elements due to unidirectional signal transmission, such as neural synapses. It has been reported that SR can be observed in OFETs by application of external noise. However, SR does not occur under conditions where the input signal exceeds the OFET threshold without external noise. Here, we present an example of a study that analyzes the behavior of SR in OFET systems and explain how SR can be made observable. At the same time, the role of internal noise in OFETs will be explained.
Collapse
|
5
|
Fahimi Z, Mahmoodi MR, Nili H, Polishchuk V, Strukov DB. Combinatorial optimization by weight annealing in memristive hopfield networks. Sci Rep 2021; 11:16383. [PMID: 34385475 PMCID: PMC8361025 DOI: 10.1038/s41598-020-78944-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2020] [Accepted: 11/17/2020] [Indexed: 11/16/2022] Open
Abstract
The increasing utility of specialized circuits and growing applications of optimization call for the development of efficient hardware accelerator for solving optimization problems. Hopfield neural network is a promising approach for solving combinatorial optimization problems due to the recent demonstrations of efficient mixed-signal implementation based on emerging non-volatile memory devices. Such mixed-signal accelerators also enable very efficient implementation of various annealing techniques, which are essential for finding optimal solutions. Here we propose a “weight annealing” approach, whose main idea is to ease convergence to the global minima by keeping the network close to its ground state. This is achieved by initially setting all synaptic weights to zero, thus ensuring a quick transition of the Hopfield network to its trivial global minima state and then gradually introducing weights during the annealing process. The extensive numerical simulations show that our approach leads to a better, on average, solutions for several representative combinatorial problems compared to prior Hopfield neural network solvers with chaotic or stochastic annealing. As a proof of concept, a 13-node graph partitioning problem and a 7-node maximum-weight independent set problem are solved experimentally using mixed-signal circuits based on, correspondingly, a 20 × 20 analog-grade TiO2 memristive crossbar and a 12 × 10 eFlash memory array.
Collapse
Affiliation(s)
- Z Fahimi
- UC Santa Barbara, Santa Barbara, CA, 93106-9560, USA.
| | - M R Mahmoodi
- UC Santa Barbara, Santa Barbara, CA, 93106-9560, USA.
| | - H Nili
- UC Santa Barbara, Santa Barbara, CA, 93106-9560, USA
| | | | - D B Strukov
- UC Santa Barbara, Santa Barbara, CA, 93106-9560, USA
| |
Collapse
|
6
|
Berggren K, Xia Q, Likharev KK, Strukov DB, Jiang H, Mikolajick T, Querlioz D, Salinga M, Erickson JR, Pi S, Xiong F, Lin P, Li C, Chen Y, Xiong S, Hoskins BD, Daniels MW, Madhavan A, Liddle JA, McClelland JJ, Yang Y, Rupp J, Nonnenmann SS, Cheng KT, Gong N, Lastras-Montaño MA, Talin AA, Salleo A, Shastri BJ, de Lima TF, Prucnal P, Tait AN, Shen Y, Meng H, Roques-Carmes C, Cheng Z, Bhaskaran H, Jariwala D, Wang H, Shainline JM, Segall K, Yang JJ, Roy K, Datta S, Raychowdhury A. Roadmap on emerging hardware and technology for machine learning. NANOTECHNOLOGY 2021; 32:012002. [PMID: 32679577 DOI: 10.1088/1361-6528/aba70f] [Citation(s) in RCA: 29] [Impact Index Per Article: 9.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Recent progress in artificial intelligence is largely attributed to the rapid development of machine learning, especially in the algorithm and neural network models. However, it is the performance of the hardware, in particular the energy efficiency of a computing system that sets the fundamental limit of the capability of machine learning. Data-centric computing requires a revolution in hardware systems, since traditional digital computers based on transistors and the von Neumann architecture were not purposely designed for neuromorphic computing. A hardware platform based on emerging devices and new architecture is the hope for future computing with dramatically improved throughput and energy efficiency. Building such a system, nevertheless, faces a number of challenges, ranging from materials selection, device optimization, circuit fabrication and system integration, to name a few. The aim of this Roadmap is to present a snapshot of emerging hardware technologies that are potentially beneficial for machine learning, providing the Nanotechnology readers with a perspective of challenges and opportunities in this burgeoning field.
Collapse
Affiliation(s)
- Karl Berggren
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, United States of America
| | - Qiangfei Xia
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, United States of America
| | | | - Dmitri B Strukov
- Department of Electrical and Computer Engineering, University of California at Santa Barbara, Santa Barbara, CA 93106, United States of America
| | - Hao Jiang
- School of Engineering & Applied Science Yale University, CT, United States of America
| | | | | | - Martin Salinga
- Institut für Materialphysik, Westfälische Wilhelms-Universität Münster, Germany
| | - John R Erickson
- Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15261, United States of America
| | - Shuang Pi
- Lam Research, Fremont, CA, United States of America
| | - Feng Xiong
- Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA 15261, United States of America
| | - Peng Lin
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, United States of America
| | - Can Li
- Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China
| | - Yu Chen
- School of information science and technology, Fudan University, Shanghai, People's Republic of China
| | - Shisheng Xiong
- School of information science and technology, Fudan University, Shanghai, People's Republic of China
| | - Brian D Hoskins
- Physical Measurements Laboratory, National Institute of Standards and Technology, Gaithersburg, MD 20899, United States of America
| | - Matthew W Daniels
- Physical Measurements Laboratory, National Institute of Standards and Technology, Gaithersburg, MD 20899, United States of America
| | - Advait Madhavan
- Physical Measurements Laboratory, National Institute of Standards and Technology, Gaithersburg, MD 20899, United States of America
- Institute for Research in Electronics and Applied Physics, University of Maryland, College Park, MD, United States of America
| | - James A Liddle
- Physical Measurements Laboratory, National Institute of Standards and Technology, Gaithersburg, MD 20899, United States of America
| | - Jabez J McClelland
- Physical Measurements Laboratory, National Institute of Standards and Technology, Gaithersburg, MD 20899, United States of America
| | - Yuchao Yang
- School of Electronics Engineering and Computer Science, Peking University, Beijing, People's Republic of China
| | - Jennifer Rupp
- Department of Materials Science and Engineering and Department of Electrical Engineering & Computer Science, Massachusetts Institute of Technology, Cambridge, MA 02139, United States of America
- Electrochemical Materials, ETHZ Department of Materials, Hönggerbergring 64, Zürich 8093, Switzerland
| | - Stephen S Nonnenmann
- Department of Mechanical & Industrial Engineering, University of Massachusetts-Amherst, MA, United States of America
| | - Kwang-Ting Cheng
- School of Engineering, Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, People's Republic of China
| | - Nanbo Gong
- IBM T J Watson Research Center, Yorktown Heights, NY 10598, United States of America
| | - Miguel Angel Lastras-Montaño
- Instituto de Investigación en Comunicación Óptica, Facultad de Ciencias, Universidad Autónoma de San Luis Potosí, México
| | - A Alec Talin
- Sandia National Laboratories, Livermore, CA 94551, United States of America
| | - Alberto Salleo
- Department of Materials Science and Engineering, Stanford University, Stanford, California, United States of America
| | - Bhavin J Shastri
- Department of Physics, Engineering Physics & Astronomy, Queen's University, Kingston ON KL7 3N6, Canada
| | - Thomas Ferreira de Lima
- Department of Electrical Engineering, Princeton University, Princeton, NJ 08544, United States of America
| | - Paul Prucnal
- Department of Electrical Engineering, Princeton University, Princeton, NJ 08544, United States of America
| | - Alexander N Tait
- Physical Measurement Laboratory, National Institute of Standards and Technology (NIST), Boulder, CO 80305, United States of America
| | - Yichen Shen
- Lightelligence, 268 Summer Street, Boston, MA 02210, United States of America
| | - Huaiyu Meng
- Lightelligence, 268 Summer Street, Boston, MA 02210, United States of America
| | - Charles Roques-Carmes
- Research Laboratory of Electronics, Massachusetts Institute of Technology, Cambridge, MA 02139, United States of America
| | - Zengguang Cheng
- Department of Materials, University of Oxford, Oxford OX1 3PH, United Kingdom
- State Key Laboratory of ASIC and System, School of Microelectronics, Fudan University, Shanghai 200433, People's Republic of China
| | - Harish Bhaskaran
- Department of Materials, University of Oxford, Oxford OX1 3PH, United Kingdom
| | - Deep Jariwala
- Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia PA 19104, United States of America
| | - Han Wang
- University of Southern California, Los Angeles, CA 90089, United States of America
| | - Jeffrey M Shainline
- Physical Measurement Laboratory, National Institute of Standards and Technology (NIST), Boulder, CO 80305, United States of America
| | - Kenneth Segall
- Department of Physics and Astronomy, Colgate University, NY 13346, United States of America
| | - J Joshua Yang
- Department of Electrical and Computer Engineering, University of Massachusetts, Amherst, MA, United States of America
| | - Kaushik Roy
- School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, United States of America
| | - Suman Datta
- University of Notre Dame, Notre Dame, IN 46556, United States of America
| | - Arijit Raychowdhury
- Georgia Institute of Technology, Atlanta, GA 30332, United States of America
| |
Collapse
|
7
|
Yang K, Duan Q, Wang Y, Zhang T, Yang Y, Huang R. Transiently chaotic simulated annealing based on intrinsic nonlinearity of memristors for efficient solution of optimization problems. SCIENCE ADVANCES 2020; 6:eaba9901. [PMID: 32851168 PMCID: PMC7428342 DOI: 10.1126/sciadv.aba9901] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/21/2020] [Accepted: 07/01/2020] [Indexed: 05/04/2023]
Abstract
Optimization problems are ubiquitous in scientific research, engineering, and daily lives. However, solving a complex optimization problem often requires excessive computing resource and time and faces challenges in easily getting trapped into local optima. Here, we propose a memristive optimizer hardware based on a Hopfield network, which introduces transient chaos to simulated annealing in aid of jumping out of the local optima while ensuring convergence. A single memristor crossbar is used to store the weight parameters of a fully connected Hopfield network and adjust the network dynamics in situ. Furthermore, we harness the intrinsic nonlinearity of memristors within the crossbar to implement an efficient and simplified annealing process for the optimization. Solutions of continuous function optimizations on sphere function and Matyas function as well as combinatorial optimization on Max-cut problem are experimentally demonstrated, indicating great potential of the transiently chaotic memristive network in solving optimization problems in general.
Collapse
Affiliation(s)
- Ke Yang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), Department of Micro/nanoelectronics, Peking University, Beijing 100871, China
| | - Qingxi Duan
- Key Laboratory of Microelectronic Devices and Circuits (MOE), Department of Micro/nanoelectronics, Peking University, Beijing 100871, China
| | - Yanghao Wang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), Department of Micro/nanoelectronics, Peking University, Beijing 100871, China
| | - Teng Zhang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), Department of Micro/nanoelectronics, Peking University, Beijing 100871, China
| | - Yuchao Yang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), Department of Micro/nanoelectronics, Peking University, Beijing 100871, China
- Center for Brain Inspired Chips, Institute for Artificial Intelligence, Peking University, Beijing 100871, China
- Frontiers Science Center for Nano-optoelectronics, Peking University, Beijing 100871, China
- Corresponding author. (Y.Y.); (R.H.)
| | - Ru Huang
- Key Laboratory of Microelectronic Devices and Circuits (MOE), Department of Micro/nanoelectronics, Peking University, Beijing 100871, China
- Center for Brain Inspired Chips, Institute for Artificial Intelligence, Peking University, Beijing 100871, China
- Frontiers Science Center for Nano-optoelectronics, Peking University, Beijing 100871, China
- Corresponding author. (Y.Y.); (R.H.)
| |
Collapse
|
8
|
Versatile stochastic dot product circuits based on nonvolatile memories for high performance neurocomputing and neurooptimization. Nat Commun 2019; 10:5113. [PMID: 31704925 PMCID: PMC6841978 DOI: 10.1038/s41467-019-13103-7] [Citation(s) in RCA: 50] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2019] [Accepted: 10/10/2019] [Indexed: 11/22/2022] Open
Abstract
The key operation in stochastic neural networks, which have become the state-of-the-art approach for solving problems in machine learning, information theory, and statistics, is a stochastic dot-product. While there have been many demonstrations of dot-product circuits and, separately, of stochastic neurons, the efficient hardware implementation combining both functionalities is still missing. Here we report compact, fast, energy-efficient, and scalable stochastic dot-product circuits based on either passively integrated metal-oxide memristors or embedded floating-gate memories. The circuit’s high performance is due to mixed-signal implementation, while the efficient stochastic operation is achieved by utilizing circuit’s noise, intrinsic and/or extrinsic to the memory cell array. The dynamic scaling of weights, enabled by analog memory devices, allows for efficient realization of different annealing approaches to improve functionality. The proposed approach is experimentally verified for two representative applications, namely by implementing neural network for solving a four-node graph-partitioning problem, and a Boltzmann machine with 10-input and 8-hidden neurons. Providing efficient and scalable specialized hardware for stochastic neural networks remains a challenge. Here, the authors propose a fast, energy-efficient and scalable stochastic dot-product circuit that may use either of two types of memory devices – metal-oxide memristors and floating-gate memories.
Collapse
|
9
|
Chou J, Bramhavar S, Ghosh S, Herzog W. Analog Coupled Oscillator Based Weighted Ising Machine. Sci Rep 2019; 9:14786. [PMID: 31615999 PMCID: PMC6794317 DOI: 10.1038/s41598-019-49699-5] [Citation(s) in RCA: 44] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2019] [Accepted: 08/27/2019] [Indexed: 11/08/2022] Open
Abstract
We report on an analog computing system with coupled non-linear oscillators which is capable of solving complex combinatorial optimization problems using the weighted Ising model. The circuit is composed of a fully-connected 4-node LC oscillator network with low-cost electronic components and compatible with traditional integrated circuit technologies. We present the theoretical modeling, experimental characterization, and statistical analysis our system, demonstrating single-run ground state accuracies of 98% on randomized MAX-CUT problem sets with binary weights and 84% with 5-bit weight resolutions. Solutions are obtained within 5 oscillator cycles, and the time-to-solution has been demonstrated to scale directly with oscillator frequency. We present scaling analysis which suggests that large coupled oscillator networks may be used to solve computationally intensive problems faster and more efficiently than conventional algorithms. The proof-of-concept system presented here provides the foundation for realizing such larger scale systems using existing hardware technologies and could pave the way towards an entirely novel computing paradigm.
Collapse
Affiliation(s)
- Jeffrey Chou
- Massachusetts Institute of Technology Lincoln Laboratory, Lexington, Massachusetts, USA
| | - Suraj Bramhavar
- Massachusetts Institute of Technology Lincoln Laboratory, Lexington, Massachusetts, USA
| | - Siddhartha Ghosh
- Massachusetts Institute of Technology Lincoln Laboratory, Lexington, Massachusetts, USA
| | - William Herzog
- Massachusetts Institute of Technology Lincoln Laboratory, Lexington, Massachusetts, USA.
| |
Collapse
|
10
|
Binarized Neural Network with Silicon Nanosheet Synaptic Transistors for Supervised Pattern Classification. Sci Rep 2019; 9:11705. [PMID: 31406242 PMCID: PMC6690903 DOI: 10.1038/s41598-019-48048-w] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2019] [Accepted: 07/29/2019] [Indexed: 11/24/2022] Open
Abstract
In the biological neural network, the learning process is achieved through massively parallel synaptic connections between neurons that can be adjusted in an analog manner. Recent developments in emerging synaptic devices and their networks can emulate the functionality of a biological neural network, which will be the fundamental building block for a neuromorphic computing architecture. However, on-chip implementation of a large-scale artificial neural network is still very challenging due to unreliable analog weight modulation in current synaptic device technology. Here, we demonstrate a binarized neural network (BNN) based on a gate-all-around silicon nanosheet synaptic transistor, where reliable digital-type weight modulation can contribute to improve the sustainability of the entire network. BNN is applied to three proof-of-concept examples: (1) handwritten digit classification (MNIST dataset), (2) face image classification (Yale dataset), and (3) experimental 3 × 3 binary pattern classifications using an integrated synaptic transistor network (total 9 × 9 × 2 162 cells) through a supervised online training procedure. The results consolidate the feasibility of binarized neural networks and pave the way toward building a reliable and large-scale artificial neural network by using more advanced conventional digital device technologies.
Collapse
|
11
|
Lithium ion trapping mechanism of SiO 2 in LiCoO 2 based memristors. Sci Rep 2019; 9:5081. [PMID: 30911041 PMCID: PMC6434038 DOI: 10.1038/s41598-019-41508-3] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2018] [Accepted: 03/11/2019] [Indexed: 11/09/2022] Open
Abstract
Pt/LiCoO2/SiO2/Si stacks with different SiO2 thicknesses are fabricated and the influence of SiO2 on memristive behavior is investigated. It is demonstrated that SiO2 can serve as Li ion trapping layer benefiting device retention, and the thickness of SiO2 must be controlled to avoid large SET voltage and state instability. Simulation model based on Nernst potential and diffusion potential is postulated for electromotive force in LiCoO2 based memristors. The simulation results show that SiO2 trapping layer decreases the total electromotive field of device and thereby prevents Li ions from migrating back to LiCoO2. This model shows a good agreement with experimental data and reveals the Li ion trapping mechanism of SiO2 in LiCoO2 based memristors.
Collapse
|
12
|
Danial L, Wainstein N, Kraus S, Kvatinsky S. Breaking Through the Speed-Power-Accuracy Tradeoff in ADCs Using a Memristive Neuromorphic Architecture. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE 2018. [DOI: 10.1109/tetci.2018.2849109] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
13
|
Kim S, Choi B, Lim M, Kim Y, Kim HD, Choi SJ. Synaptic Device Network Architecture with Feature Extraction for Unsupervised Image Classification. SMALL (WEINHEIM AN DER BERGSTRASSE, GERMANY) 2018; 14:e1800521. [PMID: 30009414 DOI: 10.1002/smll.201800521] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/06/2018] [Revised: 05/07/2018] [Indexed: 06/08/2023]
Abstract
For the efficient recognition and classification of numerous images, neuroinspired deep learning algorithms have demonstrated their substantial performance. Nevertheless, current deep learning algorithms that are performed on von Neumann machines face significant limitations due to their inherent inefficient energy consumption. Thus, alternative approaches (i.e., neuromorphic systems) are expected to provide more energy-efficient computing units. However, the implementation of the neuromorphic system is still challenging due to the uncertain impacts of synaptic device specifications on system performance. Moreover, only few studies are reported how to implement feature extraction algorithms on the neuromorphic system. Here, a synaptic device network architecture with a feature extraction algorithm inspired by the convolutional neural network is demonstrated. Its pattern recognition efficacy is validated using a device-to-system level simulation. The network can classify handwritten digits at up to a 90% recognition rate despite using fewer synaptic devices than the architecture without feature extraction.
Collapse
Affiliation(s)
- Sungho Kim
- Department of Electrical Engineering, Sejong University, Seoul, 05006, South Korea
| | - Bongsik Choi
- School of Electrical Engineering, Kookmin University, Seoul, 02707, South Korea
| | - Meehyun Lim
- Mechatronics R&D Center, Samsung Electronics, Gyonggi-do, 18448, South Korea
| | - Yeamin Kim
- School of Electrical Engineering, Kookmin University, Seoul, 02707, South Korea
| | - Hee-Dong Kim
- Department of Electrical Engineering, Sejong University, Seoul, 05006, South Korea
| | - Sung-Jin Choi
- School of Electrical Engineering, Kookmin University, Seoul, 02707, South Korea
| |
Collapse
|
14
|
Malakhova YN, Korovin AN, Lapkin DA, Malakhov SN, Shcherban VV, Pichkur EB, Yakunin SN, Demin VA, Chvalun SN, Erokhin V. Planar and 3D fibrous polyaniline-based materials for memristive elements. SOFT MATTER 2017; 13:7300-7306. [PMID: 28976529 DOI: 10.1039/c7sm01773a] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
The memristive elements constructed using polymers - polyaniline (PANI) and polyethyleneoxide (PEO) - could be assembled on planar thin films or on 3D fibrous materials. Planar conductive PANI-based materials were made using the Langmuir-Schaefer (LS) method, and the 3D materials - using the electrospinning method which is a scalable technique. We have analyzed the influence of PANI molar mass, natures of solvent and subphase on the crystalline structure, thickness and conductivity of planar LS films, and the influence of PANI molar mass and the PANI-PEO ratio on the morphological and structural characteristics of 3D fibrous materials.
Collapse
Affiliation(s)
- Yulia N Malakhova
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia and Moscow Technological University, Institute of Fine Chemical Technology, 86, Vernadskogo prosp., Moscow, 119571, Russia
| | - Alexei N Korovin
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia
| | - Dmitry A Lapkin
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia
| | - Sergey N Malakhov
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia
| | - Valentina V Shcherban
- Moscow Technological University, Institute of Fine Chemical Technology, 86, Vernadskogo prosp., Moscow, 119571, Russia
| | - Eugene B Pichkur
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia
| | - Sergey N Yakunin
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia
| | - Vyacheslav A Demin
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia
| | - Sergey N Chvalun
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia and Moscow Technological University, Institute of Fine Chemical Technology, 86, Vernadskogo prosp., Moscow, 119571, Russia
| | - Victor Erokhin
- National Research Centre, Kurchatov Institute, 1, pl. Akademika Kurchatova, Moscow, 123182, Russia and CNR-IMEM (National Research Council, Institute of Materials for Electronics and Magnetism) and University of Parma, Viale Usberti 7A, Parma, 42124, Italy.
| |
Collapse
|
15
|
Sheridan PM, Cai F, Du C, Ma W, Zhang Z, Lu WD. Sparse coding with memristor networks. NATURE NANOTECHNOLOGY 2017; 12:784-789. [PMID: 28530717 DOI: 10.1038/nnano.2017.83] [Citation(s) in RCA: 170] [Impact Index Per Article: 24.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/16/2016] [Accepted: 03/28/2017] [Indexed: 05/04/2023]
Abstract
Sparse representation of information provides a powerful means to perform feature extraction on high-dimensional data and is of broad interest for applications in signal processing, computer vision, object recognition and neurobiology. Sparse coding is also believed to be a key mechanism by which biological neural systems can efficiently process a large amount of complex sensory data while consuming very little power. Here, we report the experimental implementation of sparse coding algorithms in a bio-inspired approach using a 32 × 32 crossbar array of analog memristors. This network enables efficient implementation of pattern matching and lateral neuron inhibition and allows input data to be sparsely encoded using neuron activities and stored dictionary elements. Different dictionary sets can be trained and stored in the same system, depending on the nature of the input signals. Using the sparse coding algorithm, we also perform natural image processing based on a learned dictionary.
Collapse
Affiliation(s)
- Patrick M Sheridan
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Fuxi Cai
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Chao Du
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Wen Ma
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Zhengya Zhang
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Wei D Lu
- Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, Michigan 48109, USA
| |
Collapse
|
16
|
Pedretti G, Milo V, Ambrogio S, Carboni R, Bianchi S, Calderoni A, Ramaswamy N, Spinelli AS, Ielmini D. Memristive neural network for on-line learning and tracking with brain-inspired spike timing dependent plasticity. Sci Rep 2017; 7:5288. [PMID: 28706303 PMCID: PMC5509735 DOI: 10.1038/s41598-017-05480-0] [Citation(s) in RCA: 117] [Impact Index Per Article: 16.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2017] [Accepted: 05/30/2017] [Indexed: 11/09/2022] Open
Abstract
Brain-inspired computation can revolutionize information technology by introducing machines capable of recognizing patterns (images, speech, video) and interacting with the external world in a cognitive, humanlike way. Achieving this goal requires first to gain a detailed understanding of the brain operation, and second to identify a scalable microelectronic technology capable of reproducing some of the inherent functions of the human brain, such as the high synaptic connectivity (~104) and the peculiar time-dependent synaptic plasticity. Here we demonstrate unsupervised learning and tracking in a spiking neural network with memristive synapses, where synaptic weights are updated via brain-inspired spike timing dependent plasticity (STDP). The synaptic conductance is updated by the local time-dependent superposition of pre- and post-synaptic spikes within a hybrid one-transistor/one-resistor (1T1R) memristive synapse. Only 2 synaptic states, namely the low resistance state (LRS) and the high resistance state (HRS), are sufficient to learn and recognize patterns. Unsupervised learning of a static pattern and tracking of a dynamic pattern of up to 4 × 4 pixels are demonstrated, paving the way for intelligent hardware technology with up-scaled memristive neural networks.
Collapse
Affiliation(s)
- G Pedretti
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - V Milo
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - S Ambrogio
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - R Carboni
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - S Bianchi
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - A Calderoni
- Micron Technology, Inc., Boise, ID, 83707, USA
| | - N Ramaswamy
- Micron Technology, Inc., Boise, ID, 83707, USA
| | - A S Spinelli
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy
| | - D Ielmini
- Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano and IU.NET, Piazza L. da Vinci 32, 20133, Milano, Italy.
| |
Collapse
|
17
|
Fuller EJ, Gabaly FE, Léonard F, Agarwal S, Plimpton SJ, Jacobs-Gedrim RB, James CD, Marinella MJ, Talin AA. Li-Ion Synaptic Transistor for Low Power Analog Computing. ADVANCED MATERIALS (DEERFIELD BEACH, FLA.) 2017; 29:1604310. [PMID: 27874238 DOI: 10.1002/adma.201604310] [Citation(s) in RCA: 168] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/12/2016] [Revised: 10/10/2016] [Indexed: 05/22/2023]
Abstract
Nonvolatile redox transistors (NVRTs) based upon Li-ion battery materials are demonstrated as memory elements for neuromorphic computer architectures with multi-level analog states, "write" linearity, low-voltage switching, and low power dissipation. Simulations of backpropagation using the device properties reach ideal classification accuracy. Physics-based simulations predict energy costs per "write" operation of <10 aJ when scaled to 200 nm × 200 nm.
Collapse
Affiliation(s)
| | | | | | - Sapan Agarwal
- Sandia National Laboratories, Albuquerque, NM, 87185, USA
| | | | | | - Conrad D James
- Sandia National Laboratories, Albuquerque, NM, 87185, USA
| | | | - A Alec Talin
- Sandia National Laboratories, Livermore, CA, 94551, USA
| |
Collapse
|