A Survey of Stochastic Computing Neural Networks for Machine Learning Applications

Stagewise Training With Exponentially Growing Training Sets

In the world of big data, training large-scale machine learning problems has gained considerable attention. Numerous innovative optimization strategies have been presented in recent years to accelerate the large-scale training process. However, the possibility of further accelerating the training process of various optimization algorithms remains an unresolved subject. To begin addressing this difficult problem, we exploit the researched findings that when training data are independent and identically distributed, the learning problem on a smaller dataset is not significantly different from the original one. Upon that, we propose a stagewise training technique that grows the size of the training set exponentially while solving nonsmooth subproblem. We demonstrate that our stagewise training via exponentially growing the size of the training sets (STEGSs) are compatible with a large number of proximal gradient descent and gradient hard thresholding (GHT) techniques. Interestingly, we demonstrate that STEGS can greatly reduce overall complexity while maintaining statistical accuracy or even surpassing the intrinsic error introduced by GHT approaches. In addition, we analyze the effect of the training data growth rate on the overall complexity. The practical results of applying $l_{2,1}$ - and $l_{0}$ -norms to a variety of large-scale real-world datasets not only corroborate our theories but also demonstrate the benefits of our STEGS framework.

From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks

Deep neural networks (DNNs) have been widely used in many artificial intelligence (AI) tasks. However, deploying them brings significant challenges due to the huge cost of memory, energy, and computation. To address these challenges, researchers have developed various model compression techniques such as model quantization and model pruning. Recently, there has been a surge in research on compression methods to achieve model efficiency while retaining performance. Furthermore, more and more works focus on customizing the DNN hardware accelerators to better leverage the model compression techniques. In addition to efficiency, preserving security and privacy is critical for deploying DNNs. However, the vast and diverse body of related works can be overwhelming. This inspires us to conduct a comprehensive survey on recent research toward the goal of high-performance, cost-efficient, and safe deployment of DNNs. Our survey first covers the mainstream model compression techniques, such as model quantization, model pruning, knowledge distillation, and optimizations of nonlinear operations. We then introduce recent advances in designing hardware accelerators that can adapt to efficient model compression approaches. In addition, we discuss how homomorphic encryption can be integrated to secure DNN deployment. Finally, we discuss several issues, such as hardware evaluation, generalization, and integration of various compression approaches. Overall, we aim to provide a big picture of efficient DNNs from algorithm to hardware accelerators and security perspectives.

Design and Implementation of Low-Complexity Multiple Symbol Detection Algorithm Using Hybrid Stochastic Computing in Aircraft Wireless Communications

The Multiple Symbol Detection (MSD) algorithm can effectively lower the demodulation threshold in Frequency Modulation (FM) technology, which is widely used in aircraft wireless communications due to its insensitivity to large Doppler shifts. However, the high computational complexity of the MSD algorithm leads to considerable hardware resource overhead. In this paper, we propose a novel MSD architecture based on hybrid stochastic computing (SC), which allows for accurate signal detection while maintaining low hardware complexity. Given that the correlation calculation dominates the computational load in the MSD algorithm, we develop an SC-based, low-complexity unit to perform complex correlation operations using simple hardware circuits, significantly reducing the hardware overhead. Particularly, we integrate a flexible and scalable stochastic adder in the SC-based correlation calculation, which incorporates an adjustable scaling factor to enable high distinguishability in all possible correlation results. Additionally, for the symbol decision process of the MSD algorithm, we design a binary computing-based pipeline architecture to execute the computing process serially, which leverages the inherent low update rate of SC-based correlation results to further reduce the overall resource overhead. Experimental results show that, compared to an 8-bit quantization MSD implementation, our proposed hybrid SC-based MSD architecture achieves a comparable bit error rate while reducing the hardware resources to 69%, 45%, and 36% of those required for the three-, five-, and seven-symbol MSD algorithms, respectively.

A Gradient-Guided Evolutionary Neural Architecture Search

Neural architecture search (NAS) is a popular method that can automatically design deep neural network structures. However, designing a neural network using NAS is computationally expensive. This article proposes a gradient-guided evolutionary NAS (GENAS) to design convolutional neural networks (CNNs) for image classification. GENAS is a hybrid algorithm that combines evolutionary global and local search operators to evolve a population of subnets sampled from a supernet. Each candidate architecture is encoded as a table describing which operations are associated with the edges between nodes signifying feature maps. Besides, evolutionary optimization uses novel crossover and mutation operators to manipulate the subnets using the proposed tabular encoding. Every generations, the candidate architectures undergo a local search inspired by differentiable NAS. GENAS is designed to overcome the limitations of both evolutionary and gradient descent NAS. This algorithmic structure enables the performance assessment of the candidate architecture without retraining, thus limiting the NAS calculation time. Furthermore, subnet individuals are decoupled during evaluation to prevent strong coupling of operations in the supernet. The experimental results indicate that the searched structures achieve test errors of 2.45%, 16.86%, and 23.9% on CIFAR-10/100/ImageNet datasets and it costs only 0.26 GPU days on a graphic card. GENAS can effectively expedite the training and evaluation processes and obtain high-performance network structures.

Quantum-limited stochastic optical neural networks operating at a few quanta per activation

Energy efficiency in computation is ultimately limited by noise, with quantum limits setting the fundamental noise floor. Analog physical neural networks hold promise for improved energy efficiency compared to digital electronic neural networks. However, they are typically operated in a relatively high-power regime so that the signal-to-noise ratio (SNR) is large (>10), and the noise can be treated as a perturbation. We study optical neural networks where all layers except the last are operated in the limit that each neuron can be activated by just a single photon, and as a result the noise on neuron activations is no longer merely perturbative. We show that by using a physics-based probabilistic model of the neuron activations in training, it is possible to perform accurate machine-learning inference in spite of the extremely high shot noise (SNR  ~ 1). We experimentally demonstrated MNIST handwritten-digit classification with a test accuracy of 98% using an optical neural network with a hidden layer operating in the single-photon regime; the optical energy used to perform the classification corresponds to just 0.038 photons per multiply-accumulate (MAC) operation. Our physics-aware stochastic training approach might also prove useful with non-optical ultra-low-power hardware.

An Energy-Efficient Bayesian Neural Network Implementation Using Stochastic Computing Method

The robustness of Bayesian neural networks (BNNs) to real-world uncertainties and incompleteness has led to their application in some safety-critical fields. However, evaluating uncertainty during BNN inference requires repeated sampling and feed-forward computing, making them challenging to deploy in low-power or embedded devices. This article proposes the use of stochastic computing (SC) to optimize the hardware performance of BNN inference in terms of energy consumption and hardware utilization. The proposed approach adopts bitstream to represent Gaussian random number and applies it in the inference phase. This allows for the omission of complex transformation computations in the central limit theorem-based Gaussian random number generating (CLT-based GRNG) method and the simplification of multipliers as AND operations. Furthermore, an asynchronous parallel pipeline calculation technique is proposed in computing block to enhance operation speed. Compared with conventional binary radix-based BNN, SC-based BNN (StocBNN) realized by FPGA with 128-bit bitstream consumes much less energy consumption and hardware resources with less than 0.1% accuracy decrease when dealing with MNIST/Fashion-MNIST datasets.

Machine learning insights on intensive care unit-acquired weakness

Intensive care unit-acquired weakness (ICU-AW) significantly hampers patient recovery and increases morbidity. With the absence of established preventive strategies, this study utilizes advanced machine learning methodologies to unearth key predictors of ICU-AW. Employing a sophisticated multilayer perceptron neural network, the research methodically assesses the predictive power for ICU-AW, pinpointing the length of ICU stay and duration of mechanical ventilation as pivotal risk factors. The findings advocate for minimizing these elements as a preventive approach, offering a novel perspective on combating ICU-AW. This research illuminates critical risk factors and lays the groundwork for future explorations into effective prevention and intervention strategies.

Mott memristor based stochastic neurons for probabilistic computing

Many studies suggest that probabilistic spiking in biological neural systems is beneficial as it aids learning and provides Bayesian inference-like dynamics. If appropriately utilised, noise and stochasticity in nanoscale devices can benefit neuromorphic systems. In this paper, we build a stochastic leaky integrate and fire (LIF) neuron, utilising a Mott memristor's inherent stochastic switching dynamics. We demonstrate that the developed LIF neuron is capable of biological neural dynamics. We leverage these characteristics of the proposed LIF neuron by integrating it into a population-coded spiking neural network and a spiking restricted Boltzmann machine (sRBM), thereby showcasing its ability to implement probabilistic learning and inference. The sRBM achieves a software-comparable accuracy of 87.13%. Unlike CMOS-based probabilistic neurons, our design does not require any external noise sources. The designed neurons are highly energy efficient and ultra-compact, requiring only three components: a resistor, a capacitor and a memristor device.

Significant risk factors for intensive care unit-acquired weakness: A processing strategy based on repeated machine learning

BACKGROUND

Intensive care unit-acquired weakness (ICU-AW) is a common complication that significantly impacts the patient's recovery process, even leading to adverse outcomes. Currently, there is a lack of effective preventive measures.

AIM

To identify significant risk factors for ICU-AW through iterative machine learning techniques and offer recommendations for its prevention and treatment.

METHODS

Patients were categorized into ICU-AW and non-ICU-AW groups on the 14th day post-ICU admission. Relevant data from the initial 14 d of ICU stay, such as age, comorbidities, sedative dosage, vasopressor dosage, duration of mechanical ventilation, length of ICU stay, and rehabilitation therapy, were gathered. The relationships between these variables and ICU-AW were examined. Utilizing iterative machine learning techniques, a multilayer perceptron neural network model was developed, and its predictive performance for ICU-AW was assessed using the receiver operating characteristic curve.

RESULTS

Within the ICU-AW group, age, duration of mechanical ventilation, lorazepam dosage, adrenaline dosage, and length of ICU stay were significantly higher than in the non-ICU-AW group. Additionally, sepsis, multiple organ dysfunction syndrome, hypoalbuminemia, acute heart failure, respiratory failure, acute kidney injury, anemia, stress-related gastrointestinal bleeding, shock, hypertension, coronary artery disease, malignant tumors, and rehabilitation therapy ratios were significantly higher in the ICU-AW group, demonstrating statistical significance. The most influential factors contributing to ICU-AW were identified as the length of ICU stay (100.0%) and the duration of mechanical ventilation (54.9%). The neural network model predicted ICU-AW with an area under the curve of 0.941, sensitivity of 92.2%, and specificity of 82.7%.

CONCLUSION

The main factors influencing ICU-AW are the length of ICU stay and the duration of mechanical ventilation. A primary preventive strategy, when feasible, involves minimizing both ICU stay and mechanical ventilation duration.

Data-Knowledge-Driven Self-Organizing Fuzzy Neural Network

Fuzzy neural networks (FNNs) hold the advantages of knowledge leveraging and adaptive learning, which have been widely used in nonlinear system modeling. However, it is difficult for FNNs to obtain the appropriate structure in the situation of insufficient data, which limits its generalization performance. To solve this problem, a data-knowledge-driven self-organizing FNN (DK-SOFNN) with a structure compensation strategy and a parameter reinforcement mechanism is proposed in this article. First, a structure compensation strategy is proposed to mine structural information from empirical knowledge to learn the structure of DK-SOFNN. Then, a complete model structure can be acquired by sufficient structural information. Second, a parameter reinforcement mechanism is developed to determine the parameter evolution direction of DK-SOFNN that is most suitable for the current model structure. Then, a robust model can be obtained by the interaction between parameters and dynamic structure. Finally, the proposed DK-SOFNN is theoretically analyzed on the fixed structure case and dynamic structure case. Then, the convergence conditions can be obtained to guide practical applications. The merits of DK-SOFNN are demonstrated by some benchmark problems and industrial applications.

Fully Parallel Stochastic Computing Hardware Implementation of Convolutional Neural Networks for Edge Computing Applications

Edge artificial intelligence (AI) is receiving a tremendous amount of interest from the machine learning community due to the ever-increasing popularization of the Internet of Things (IoT). Unfortunately, the incorporation of AI characteristics to edge computing devices presents the drawbacks of being power and area hungry for typical deep learning techniques such as convolutional neural networks (CNNs). In this work, we propose a power-and-area efficient architecture based on the exploitation of the correlation phenomenon in stochastic computing (SC) systems. The proposed architecture solves the challenges that a CNN implementation with SC (SC-CNN) may present, such as the high resources used in binary-to-stochastic conversion, the inaccuracy produced by undesired correlation between signals, and the complexity of the stochastic maximum function implementation. To prove that our architecture meets the requirements of edge intelligence realization, we embed a fully parallel CNN in a single field-programmable gate array (FPGA) chip. The results obtained showed a better performance than traditional binary logic and other SC implementations. In addition, we performed a full VLSI synthesis of the proposed design, showing that it presents better overall characteristics than other recently published VLSI architectures.

Stochastic Adder Circuits with Improved Entropy Output

Random pulse computing (RPC), the third paradigm along with digital and quantum computing, draws inspiration from biology, particularly the functioning of neurons. Here, we study information processing in random pulse computing circuits intended for the summation of numbers. Based on the information-theoretic merits of entropy budget and relative Kolmogorov-Sinai entropy, we investigate the prior art and propose new circuits: three deterministic adders with significantly improved output entropy and one exact nondeterministic adder that requires much less additional entropy than the previous art. All circuits are realized and tested experimentally, using quantum entropy sources and reconfigurable logic devices. Not only the proposed circuits yield a precise mathematical result and have output entropy near maximum, which satisfies the need for building a programmable random pulse computer, but also they provide affordable hardware options for generating additional entropy.

A Survey of Hardware Self-Organizing Maps

Self-organizing feature maps (SOMs) are commonly used technique for clustering and data dimensionality reduction in many application fields. Indeed, their inherent property of topology preservation and unsupervised learning of processed data without any prior knowledge put them in the front of candidates for data reduction in the Internet of Things (IoT) and big data (BD) technologies. However, the high computational cost of SOMs limits their use to offline approaches and makes the online real-time high-performance SOM processing more challenging and mostly reserved to specific hardware implementations. In this article, we present a survey of hardware (HW) SOM implementations found in the literature so far: the most widely used computing blocks, architectures, design choices, adaptation, and optimization techniques that have been reported in the field of hardware SOMs. Moreover, we give an overview of main challenges and trends for their ubiquitous adoption as hardware accelerators in many application fields. This article is expected to be useful for researchers in the areas of artificial intelligence, hardware architecture, and system design.

Artificial Intelligence Enhanced Reliability Assessment Methodology With Small Samples

Due to the high price of the product and the limitation of laboratory conditions, reliability tests often get a small number of failed samples. If the data are not handled properly, the reliability evaluation results will incur grave errors. In order to solve this problem, this work proposes an artificial intelligence (AI) enhanced reliability assessment methodology by combining Bayesian neural networks (BNNs) and differential evolution (DE) algorithms. First, a single hidden layer BNN model is constructed by fusing small samples and prior information to obtain the 95% confidence interval (CI) of the posterior distribution. Then, the DE algorithm is used to iteratively generate optimal virtual samples based on the 95% CI and small samples trends. A reliability assessment model is reconstructed based on double hidden layers BNN model by combining virtual samples and test samples in the last stage. In order to verify the effectiveness of the proposed method, an accelerated life test (ALT) of the subsurface electronic control unit (S-ECU) was carried out. The verification test results show that the proposed method can accurately evaluate the reliability life of a product. And compared with the two existing methods, the results show that this method can effectively improve the accuracy of the reliability assessment of a test product.

Machine Learning for Microcontroller-Class Hardware: A Review

The advancements in machine learning opened a new opportunity to bring intelligence to the low-end Internet-of-Things nodes such as microcontrollers. Conventional machine learning deployment has high memory and compute footprint hindering their direct deployment on ultra resource-constrained microcontrollers. This paper highlights the unique requirements of enabling onboard machine learning for microcontroller class devices. Researchers use a specialized model development workflow for resource-limited applications to ensure the compute and latency budget is within the device limits while still maintaining the desired performance. We characterize a closed-loop widely applicable workflow of machine learning model development for microcontroller class devices and show that several classes of applications adopt a specific instance of it. We present both qualitative and numerical insights into different stages of model development by showcasing several use cases. Finally, we identify the open research challenges and unsolved questions demanding careful considerations moving forward.

Interval type-2 fuzzy computational model for real time Kalman filtering and forecasting of the dynamic spreading behavior of novel Coronavirus 2019

This paper presents a computational model based on interval type-2 fuzzy systems for analysis and forecasting of COVID-19 dynamic spreading behavior. The proposed methodology is related to interval type-2 fuzzy Kalman filters design from experimental data of daily deaths reports. Initially, a recursive spectral decomposition is performed on the experimental dataset to extract relevant unobservable components for parametric estimation of the interval type-2 fuzzy Kalman filter. The antecedent propositions of fuzzy rules are obtained by formulating a type-2 fuzzy clustering algorithm. The state space submodels and the interval Kalman gains in consequent propositions of fuzzy rules are recursively updated by a proposed interval type-2 fuzzy Observer/Kalman Filter Identification (OKID) algorithm, taking into account the unobservable components obtained by recursive spectral decomposition of epidemiological experimental data of COVID-19. For validation purposes, through a comparative analysis with relevant references of literature, the proposed methodology is evaluated from the adaptive tracking and forecasting of COVID-19 dynamic spreading behavior, in Brazil, with the better results for RMSE of 1.24×10-5, MAE of 2.62×10-6, R2 of 0.99976, and MAPE of 6.33×10-6.

Efficient Stochastic Computing FIR Filtering Using Sigma-Delta Modulated Signals

This work presents a soft-filtering digital signal processing architecture based on sigma-delta modulators and stochastic computing. A sigma-delta modulator converts the input high-resolution signal to a single-bit stream enabling filtering structures to be realized using stochastic computing’s negligible-area multipliers. Simulation in the spectral domain demonstrates the filter’s proper operation and its roll-off behavior, as well as the signal-to-noise ratio improvement using the sigma-delta modulator, compared to typical stochastic computing filter realizations. The proposed architecture’s hardware advantages are showcased with synthesis results for two FIR filters using FPGA and synopsys tools, while comparisons with standard stochastic computing-based hardware realizations, as well as with conventional binary ones, demonstrate its efficacy.