A New Stochastic Computing Methodology for Efficient Neural Network Implementation

An Energy-Efficient Bayesian Neural Network Implementation Using Stochastic Computing Method

The robustness of Bayesian neural networks (BNNs) to real-world uncertainties and incompleteness has led to their application in some safety-critical fields. However, evaluating uncertainty during BNN inference requires repeated sampling and feed-forward computing, making them challenging to deploy in low-power or embedded devices. This article proposes the use of stochastic computing (SC) to optimize the hardware performance of BNN inference in terms of energy consumption and hardware utilization. The proposed approach adopts bitstream to represent Gaussian random number and applies it in the inference phase. This allows for the omission of complex transformation computations in the central limit theorem-based Gaussian random number generating (CLT-based GRNG) method and the simplification of multipliers as AND operations. Furthermore, an asynchronous parallel pipeline calculation technique is proposed in computing block to enhance operation speed. Compared with conventional binary radix-based BNN, SC-based BNN (StocBNN) realized by FPGA with 128-bit bitstream consumes much less energy consumption and hardware resources with less than 0.1% accuracy decrease when dealing with MNIST/Fashion-MNIST datasets.

Fully Parallel Stochastic Computing Hardware Implementation of Convolutional Neural Networks for Edge Computing Applications

Edge artificial intelligence (AI) is receiving a tremendous amount of interest from the machine learning community due to the ever-increasing popularization of the Internet of Things (IoT). Unfortunately, the incorporation of AI characteristics to edge computing devices presents the drawbacks of being power and area hungry for typical deep learning techniques such as convolutional neural networks (CNNs). In this work, we propose a power-and-area efficient architecture based on the exploitation of the correlation phenomenon in stochastic computing (SC) systems. The proposed architecture solves the challenges that a CNN implementation with SC (SC-CNN) may present, such as the high resources used in binary-to-stochastic conversion, the inaccuracy produced by undesired correlation between signals, and the complexity of the stochastic maximum function implementation. To prove that our architecture meets the requirements of edge intelligence realization, we embed a fully parallel CNN in a single field-programmable gate array (FPGA) chip. The results obtained showed a better performance than traditional binary logic and other SC implementations. In addition, we performed a full VLSI synthesis of the proposed design, showing that it presents better overall characteristics than other recently published VLSI architectures.

Entropy considerations in improved circuits for a biologically-inspired random pulse computer

We present five novel or modified circuits intended for building a universal computer based on random pulse computing (RPC) paradigm, a biologically-inspired way of computation in which variable is represented by a frequency of a random pulse train (RPT) rather than by a logic state. For the first time we investigate operation of RPC circuits from the point of entropy. In particular, we introduce entropy budget criterion (EBC) to reliably predict whether it is even possible to create a deterministic circuit for a given mathematical operation and show its relevance to numerical precision of calculations. Based on insights gained from the EBC, unlike in the previous art, where randomness is obtained from electronics noise or a pseudorandom shift register while processing circuitry is deterministic, in our approach both variable generation and signal processing rely on the random flip-flop (RFF) whose randomness is derived from a fundamentally random quantum process. This approach offers an advantage in higher precision, better randomness of the output and conceptual simplicity of circuits.

A Survey of Stochastic Computing Neural Networks for Machine Learning Applications

Neural networks (NNs) are effective machine learning models that require significant hardware and energy consumption in their computing process. To implement NNs, stochastic computing (SC) has been proposed to achieve a tradeoff between hardware efficiency and computing performance. In an SC NN, hardware requirements and power consumption are significantly reduced by moderately sacrificing the inference accuracy and computation speed. With recent developments in SC techniques, however, the performance of SC NNs has substantially been improved, making it comparable with conventional binary designs yet by utilizing less hardware. In this article, we begin with the design of a basic SC neuron and then survey different types of SC NNs, including multilayer perceptrons, deep belief networks, convolutional NNs, and recurrent NNs. Recent progress in SC designs that further improve the hardware efficiency and performance of NNs is subsequently discussed. The generality and versatility of SC NNs are illustrated for both the training and inference processes. Finally, the advantages and challenges of SC NNs are discussed with respect to binary counterparts.

Modeling-Based Design of Memristive Devices for Brain-Inspired Computing

Resistive switching random access memory (RRAM) has emerged for non-volatile memory application with the features of simple structure, low cost, high density, high speed, low power, and CMOS compatibility. In recent years, RRAM technology has made significant progress in brain-inspired computing paradigms by exploiting its unique physical characteristics, which attempts to eliminate the energy-intensive and time-consuming data transfer between the processing unit and the memory unit. The design of RRAM-based computing paradigms, however, requires a detailed description of the dominant physical effects correlated with the resistive switching processes to realize the interaction and optimization between devices and algorithms or architectures. This work provides an overview of the current progress on device-level resistive switching behaviors with detailed insights into the physical effects in the resistive switching layer and the multifunctional assistant layer. Then the circuit-level physics-based compact models will be reviewed in terms of typical binary RRAM and the emerging analog synaptic RRAM, which act as an interface between the device and circuit design. After that, the interaction between device and system performances will finally be addressed by reviewing the specific applications of brain-inspired computing systems including neuromorphic computing, in-memory logic, and stochastic computing.

Implementation of an efficient magnetic tunnel junction-based stochastic neural network with application to iris data classification

Stochastic neuromorphic computation (SNC) has the potential to enable a low power, error tolerant and scalable computing platform in comparison to its deterministic counterparts. However, the hardware implementation of complementary metal oxide semiconductor (CMOS)-based stochastic circuits involves conversion blocks that cost more than the actual processing circuits. The realization of the activation function for SNCs also requires a complicated circuit that results in a significant amount of power dissipation and area overhead. The inherent probabilistic switching behavior of nanomagnets provides an advantage to overcome these complexity issues for the realization of low power and area efficient SNC systems. This paper presents magnetic tunnel junction (MTJ)-based stochastic computing methodology for the implementation of a neural network. The stochastic switching behavior of the MTJ has been exploited to design a binary to stochastic converter to mitigate the complexity of the CMOS-based design. The paper also presents the technique for realizing stochastic sigmoid activation function using an MTJ. Such circuits are simpler than existing ones and use considerably less power. An image classification system employing the proposed circuits has been implemented to verify the effectiveness of the technique. The MTJ-based SNC system shows area and energy reduction by a factor of 13.5 and 2.5, respectively, while the prediction accuracy is 86.66%. Furthermore, this paper investigates how crucial parameters, such as stochastic bitstream length, number of hidden layers and number of nodes in a hidden layer, need to be set precisely to realize an efficient MTJ-based stochastic neural network (SNN). The proposed methodology can prove a promising alternative for highly efficient digital stochastic computing applications.

Stochastic computing in convolutional neural network implementation: a review

Stochastic computing (SC) is an alternative computing domain for ubiquitous deterministic computing whereby a single logic gate can perform the arithmetic operation by exploiting the nature of probability math. SC was proposed in the 1960s when binary computing was expensive. However, presently, SC started to regain interest after the widespread of deep learning application, specifically the convolutional neural network (CNN) algorithm due to its practicality in hardware implementation. Although not all computing functions can translate to the SC domain, several useful function blocks related to the CNN algorithm had been proposed and tested by researchers. An evolution of CNN, namely, binarised neural network, had also gained attention in the edge computing due to its compactness and computing efficiency. This study reviews various SC CNN hardware implementation methodologies. Firstly, we review the fundamental concepts of SC and the circuit structure and then compare the advantages and disadvantages amongst different SC methods. Finally, we conclude the overview of SC in CNN and make suggestions for widespread implementation.

Cost-effective stochastic MAC circuits for deep neural networks

Stochastic computing (SC) is a promising computing paradigm that can help address both the uncertainties of future process technology and the challenges of efficient hardware realization for deep neural networks (DNNs). However the impreciseness and long latency of SC have rendered previous SC-based DNN architectures less competitive against optimized fixed-point digital implementations, unless inference accuracy is significantly sacrificed. In this paper we propose a new SC-MAC (multiply-and-accumulate) algorithm, which is a key building block for SC-based DNNs, that is orders of magnitude more efficient and accurate than previous SC-MACs. We also show how our new SC-MAC can be extended to a vector version and used to accelerate both convolution and fully-connected layers of convolutional neural networks (CNNs) using the same hardware. Our experimental results using CNNs designed for MNIST and CIFAR-10 datasets demonstrate that not only is our SC-based CNNs more accurate and 40∼490× more energy-efficient for convolution layers than conventional SC-based ones, but ours can also achieve lower area-delay product and lower energy compared with precision-optimized fixed-point implementations without sacrificing accuracy. We also demonstrate the feasibility of our SC-based CNNs through FPGA prototypes.

A Telescopic Binary Learning Machine for Training Neural Networks

This paper proposes a new algorithm based on multiscale stochastic local search with binary representation for training neural networks [binary learning machine (BLM)]. We study the effects of neighborhood evaluation strategies, the effect of the number of bits per weight and that of the maximum weight range used for mapping binary strings to real values. Following this preliminary investigation, we propose a telescopic multiscale version of local search, where the number of bits is increased in an adaptive manner, leading to a faster search and to local minima of better quality. An analysis related to adapting the number of bits in a dynamic way is presented. The control on the number of bits, which happens in a natural manner in the proposed method, is effective to increase the generalization performance. The learning dynamics are discussed and validated on a highly nonlinear artificial problem and on real-world tasks in many application domains; BLM is finally applied to a problem requiring either feedforward or recurrent architectures for feedback control.