Roadmap on emerging hardware and technology for machine learning

Recent progress in artificial intelligence is largely attributed to the rapid development of machine learning, especially in the algorithm and neural network models. However, it is the performance of the hardware, in particular the energy efficiency of a computing system that sets the fundamental limit of the capability of machine learning. Data-centric computing requires a revolution in hardware systems, since traditional digital computers based on transistors and the von Neumann architecture were not purposely designed for neuromorphic computing. A hardware platform based on emerging devices and new architecture is the hope for future computing with dramatically improved throughput and energy efficiency. Building such a system, nevertheless, faces a number of challenges, ranging from materials selection, device optimization, circuit fabrication and system integration, to name a few. The aim of this Roadmap is to present a snapshot of emerging hardware technologies that are potentially beneficial for machine learning, providing the Nanotechnology readers with a perspective of challenges and opportunities in this burgeoning field.

Deep neural network to detect COVID-19: one architecture for both CT Scans and Chest X-rays

Since December 2019, the novel COVID-19's spread rate is exponential, and AI-driven tools are used to prevent further spreading [1]. They can help predict, screen, and diagnose COVID-19 positive cases. Within this scope, imaging with Computed Tomography (CT) scans and Chest X-rays (CXRs) are widely used in mass triage situations. In the literature, AI-driven tools are limited to one data type either CT scan or CXR to detect COVID-19 positive cases. Integrating multiple data types could possibly provide more information in detecting anomaly patterns due to COVID-19. Therefore, in this paper, we engineered a Convolutional Neural Network (CNN) -tailored Deep Neural Network (DNN) that can collectively train/test both CT scans and CXRs. In our experiments, we achieved an overall accuracy of 96.28% (AUC = 0.9808 and false negative rate = 0.0208). Further, major existing DNNs provided coherent results while integrating CT scans and CXRs to detect COVID-19 positive cases.

Improving Resident Self-Efficacy in Tracheostomy Management Using a Novel Curriculum

INTRODUCTION

Patients receiving pediatric tracheostomy have significant risk for mortality due to compromised airway. Timely management of airway emergencies in children with tracheostomies is an important clinical skill for pediatricians. We developed this curriculum to improve residents' self-efficacy with tracheostomy management.

METHODS

We collected baseline data on 67 residents from two hospitals while creating a blended curriculum with video-based instruction on routine tracheostomy change and team management of tracheostomy emergency. Forty residents enrolled in the curriculum. During an ICU rotation, they received face-to-face instruction on routine tracheostomy change in small groups, followed by assessment of managing a tracheostomy emergency during a simulation. A video completed prior to the simulation took 9 minutes, the routine tracheostomy change didactic session took 15 minutes, and the simulation instruction was completed in 10-15 minutes. We collected feedback on the effectiveness of the curriculum from the participants.

RESULTS

All 107 residents from the baseline and intervention groups completed the self-efficacy survey. The intervention group had significantly higher changes in scores across all self-efficacy domains than the baseline group. On the curriculum feedback survey, residents rated the curriculum very highly, between 4.4 and 4.8 on a 5-point Likert scale.

DISCUSSION

Our blended curriculum increased learners' self-efficacy and promoted learner competence in tracheostomy management. Residents scored more than 80% across all aspects of simulation assessment and reported higher self-efficacy scores following our curricular intervention.

CESS-A System to Categorize Bangla Web Text Documents

Technology has evolved remarkably, which has led to an exponential increase in the availability of digital text documents of disparate domains over the Internet. This makes the retrieval of the information a very much time- and resource-consuming task. Thus, a system that can categorize such documents based on their domains can truly help the users in obtaining the required information with relative ease and also reduce the workload of the search engines. This article presents a text categorization system (CESS) that categorizes text document using newly proposed hybrid features that combines term frequency-inverse document frequency-inverse class frequency and modified chi-square methods. Experiments were performed on real-world Bangla documents from eight domains comprises of 24,29,857 tokens, and the highest accuracy of 99.91% has been obtained with multilayer perceptron-based classification. Also, the experiments were tested on Reuters-21578 and 20 Newsgroups datasets and obtained accuracies of 97.29% and 94.67%, respectively, to show the language-independent nature of the system.

Bayesian Multi-objective Hyperparameter Optimization for Accurate, Fast, and Efficient Neural Network Accelerator Design

In resource-constrained environments, such as low-power edge devices and smart sensors, deploying a fast, compact, and accurate intelligent system with minimum energy is indispensable. Embedding intelligence can be achieved using neural networks on neuromorphic hardware. Designing such networks would require determining several inherent hyperparameters. A key challenge is to find the optimum set of hyperparameters that might belong to the input/output encoding modules, the neural network itself, the application, or the underlying hardware. In this work, we present a hierarchical pseudo agent-based multi-objective Bayesian hyperparameter optimization framework (both software and hardware) that not only maximizes the performance of the network, but also minimizes the energy and area requirements of the corresponding neuromorphic hardware. We validate performance of our approach (in terms of accuracy and computation speed) on several control and classification applications on digital and mixed-signal (memristor-based) neural accelerators. We show that the optimum set of hyperparameters might drastically improve the performance of one application (i.e., 52–71% for Pole-Balance), while having minimum effect on another (i.e., 50–53% for RoboNav). In addition, we demonstrate resiliency of different input/output encoding, training neural network, or the underlying accelerator modules in a neuromorphic system to the changes of the hyperparameters.

Biologically Plausible Class Discrimination Based Recurrent Neural Network Training for Motor Pattern Generation

Biological brain stores massive amount of information. Inspired by features of the biological memory, we propose an algorithm to efficiently store different classes of spatio-temporal information in a Recurrent Neural Network (RNN). A given spatio-temporal input triggers a neuron firing pattern, known as an attractor, and it conveys information about the class to which the input belongs. These attractors are the basic elements of the memory in our RNN. Preparing a set of good attractors is the key to efficiently storing temporal information in an RNN. We achieve this by means of enhancing the “separation” and “approximation” properties associated with the attractors, during the RNN training. We furthermore elaborate how these attractors can trigger an action via the readout in the RNN, similar to the sensory motor action processing in the cerebellum cortex. We show how different voice commands by different speakers trigger hand drawn impressions of the spoken words, by means of our separation and approximation based learning. The method further recognizes the gender of the speaker. The method is evaluated on the TI-46 speech data corpus, and we have achieved 98.6% classification accuracy on the TI-46 digit corpus.

Development of a simple, interpretable and easily transferable QSAR model for quick screening antiviral databases in search of novel 3C-like protease (3CLpro) enzyme inhibitors against SARS-CoV diseases

In the context of recently emerged pandemic of COVID-19, we have performed two-dimensional quantitative structure-activity relationship (2D-QSAR) modelling using SARS-CoV-3CLpro enzyme inhibitors for the development of a multiple linear regression (MLR) based model. We have used 2D descriptors with an aim to develop an easily interpretable, transferable and reproducible model which may be used for quick prediction of SAR-CoV-3CLpro inhibitory activity for query compounds in the screening process. Based on the insights obtained from the developed 2D-QSAR model, we have identified the structural features responsible for the enhancement of the inhibitory activity against 3CLpro enzyme. Moreover, we have performed the molecular docking analysis using the most and least active molecules from the dataset to understand the molecular interactions involved in binding, and the results were then correlated with the essential structural features obtained from the 2D-QSAR model. Additionally, we have performed in silico predictions of SARS-CoV 3CLpro enzyme inhibitory activity of a total of 50,437 compounds obtained from two anti-viral drug databases (CAS COVID-19 antiviral candidate compound database and another recently reported list of prioritized compounds from the ZINC15 database) using the developed model and provided prioritized compounds for experimental detection of their performance for SARS-CoV 3CLpro enzyme inhibition.

Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization

Spiking Neural Networks (SNNs) may offer an energy-efficient alternative for implementing deep learning applications. In recent years, there have been several proposals focused on supervised (conversion, spike-based gradient descent) and unsupervised (spike timing dependent plasticity) training methods to improve the accuracy of SNNs on large-scale tasks. However, each of these methods suffer from scalability, latency, and accuracy limitations. In this paper, we propose novel algorithmic techniques of modifying the SNN configuration with backward residual connections, stochastic softmax, and hybrid artificial-and-spiking neuronal activations to improve the learning ability of the training methodologies to yield competitive accuracy, while, yielding large efficiency gains over their artificial counterparts. Note, artificial counterparts refer to conventional deep learning/artificial neural networks. Our techniques apply to VGG/Residual architectures, and are compatible with all forms of training methodologies. Our analysis reveals that the proposed solutions yield near state-of-the-art accuracy with significant energy-efficiency and reduced parameter overhead translating to hardware improvements on complex visual recognition tasks, such as, CIFAR10, Imagenet datatsets.

DeepSuccinylSite: a deep learning based approach for protein succinylation site prediction

Background

Protein succinylation has recently emerged as an important and common post-translation modification (PTM) that occurs on lysine residues. Succinylation is notable both in its size (e.g., at 100 Da, it is one of the larger chemical PTMs) and in its ability to modify the net charge of the modified lysine residue from + 1 to − 1 at physiological pH. The gross local changes that occur in proteins upon succinylation have been shown to correspond with changes in gene activity and to be perturbed by defects in the citric acid cycle. These observations, together with the fact that succinate is generated as a metabolic intermediate during cellular respiration, have led to suggestions that protein succinylation may play a role in the interaction between cellular metabolism and important cellular functions. For instance, succinylation likely represents an important aspect of genomic regulation and repair and may have important consequences in the etiology of a number of disease states. In this study, we developed DeepSuccinylSite, a novel prediction tool that uses deep learning methodology along with embedding to identify succinylation sites in proteins based on their primary structure.

Results

Using an independent test set of experimentally identified succinylation sites, our method achieved efficiency scores of 79%, 68.7% and 0.48 for sensitivity, specificity and MCC respectively, with an area under the receiver operator characteristic (ROC) curve of 0.8. In side-by-side comparisons with previously described succinylation predictors, DeepSuccinylSite represents a significant improvement in overall accuracy for prediction of succinylation sites.

Conclusion

Together, these results suggest that our method represents a robust and complementary technique for advanced exploration of protein succinylation.

Cardiotocograph-based labor stage classification from uterine contraction pressure during ante-partum and intra-partum period: a fuzzy theoretic approach

Computerized techniques for Cardiotocograph (CTG) based labor stage classification would support obstetrician for advance CTG analysis and would improve their predictive power for fetal heart rate (FHR) monitoring. Intrapartum fetal monitoring is necessary as it can detect the event, which ultimately leads to hypoxic ischemic encephalopathy, cerebral palsy or even fetal demise. To bridge this gap, in this paper, we propose an automated decision support system that will help the obstetrician identify the status of the fetus during ante-partum and intra-partum period. The proposed algorithm takes 30 min of 275 Cardiotocograph data and applies a fuzzy-rule based approach for identification and classification of labor from 'toco' signal. Since there is no gold standard to validate the outcome of the proposed algorithm, the authors used various statistical means to establish the cogency of the proposed algorithm and the degree of agreement with visual estimation were using Bland-Altman plot, Fleiss kappa (0.918 ± 0.0164 at 95% CI) and Kendall's coefficient of concordance (W = 0.845). Proposed method was also compared against some standard machine learning classifiers like SVM, Random Forest and Naïve Bayes using weighted kappa (0.909), Bland-Altman plot (Limits of Agreement 0.094 to 0.0155 at 95% CI) and AUC-ROC (0.938). The proposed algorithm was found to be as efficient as visual estimation compared to the standard machine learning algorithms and thus can be incorporated into the automated decision support system.

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN, and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Controlled Forgetting: Targeted Stimulation and Dopaminergic Plasticity Modulation for Unsupervised Lifelong Learning in Spiking Neural Networks

Stochastic gradient descent requires that training samples be drawn from a uniformly random distribution of the data. For a deployed system that must learn online from an uncontrolled and unknown environment, the ordering of input samples often fails to meet this criterion, making lifelong learning a difficult challenge. We exploit the locality of the unsupervised Spike Timing Dependent Plasticity (STDP) learning rule to target local representations in a Spiking Neural Network (SNN) to adapt to novel information while protecting essential information in the remainder of the SNN from catastrophic forgetting. In our Controlled Forgetting Networks (CFNs), novel information triggers stimulated firing and heterogeneously modulated plasticity, inspired by biological dopamine signals, to cause rapid and isolated adaptation in the synapses of neurons associated with outlier information. This targeting controls the forgetting process in a way that reduces the degradation of accuracy for older tasks while learning new tasks. Our experimental results on the MNIST dataset validate the capability of CFNs to learn successfully over time from an unknown, changing environment, achieving 95.24% accuracy, which we believe is the best unsupervised accuracy ever achieved by a fixed-size, single-layer SNN on a completely disjoint MNIST dataset.

In situ unsupervised learning using stochastic switching in magneto-electric magnetic tunnel junctions

Spiking neural networks (SNNs) offer a bio-plausible and potentially power-efficient alternative to conventional deep learning. Although there has been progress towards implementing SNN functionalities in custom CMOS-based hardware using beyond Von Neumann architectures, the power-efficiency of the human brain has remained elusive. This has necessitated investigations of novel material systems which can efficiently mimic the functional units of SNNs, such as neurons and synapses. In this paper, we present a magnetoelectric-magnetic tunnel junction (ME-MTJ) device as a synapse. We arrange these synapses in a crossbar fashion and perform in situ unsupervised learning. We leverage the capacitive nature of write-ports in ME-MTJs, wherein by applying appropriately shaped voltage pulses across the write-port, the ME-MTJ can be switched in a probabilistic manner. We further exploit the sigmoidal switching characteristics of ME-MTJ to tune the synapses to follow the well-known spike timing-dependent plasticity (STDP) rule in a stochastic fashion. Finally, we use the stochastic STDP rule in ME-MTJ synapses to simulate a two-layered SNN to perform image classification tasks on a handwritten digit dataset. Thus, the capacitive write-port and the decoupled-nature of read-write path of ME-MTJs allow us to construct a transistor-less crossbar, suitable for energy-efficient implementation of in situ learning in SNNs. This article is part of the theme issue 'Harmonizing energy-autonomous computing and intelligence'.

Comparative study of expandable cage with integrated plate versus non expandable cage in cervical spine corpectomies

Background: The use of expandable cages in cervical spine has gained popularity over the last decade. They have been used in dorsal spine since long but were rarely used in cervical spine due to their high cost. Now, with more insight into their mechanics, many advantages have been noted over the fixed cages along with similar efficacy and with no added complications. Aims and Objectives: To study the benefits of expandable cage with incorporated anterior cervical plate over non expandable cage in cervical spine corpectomies. Materials and Methods: Ten cases of two level corpectomy were operated in each group and compared for intraoperative time, postoperative fusion rates and complications. Results: Intraoperative time was less in the expandable cage group. Fusion rates were comparable at 6 month follow up. No reported long term complication in both groups. Conclusion: Expandable cages are less frequently used in cervical spine due to their significantly higher cost but there are advantages such as decreased intraoperative manipulation and operative time, less damage to end plates and also useful in cases of poor bone quality.

Exploring 2D-QSAR for prediction of beta-secretase 1 (BACE1) inhibitory activity against Alzheimer's disease

We have developed a robust quantitative structure-activity relationship (QSAR) model employing a dataset of 98 heterocycle compounds to identify structural features responsible for BACE1 (beta-secretase 1) enzyme inhibition. We have used only 2D descriptors for model development purpose thus avoiding the conformational complications arising due to 3D geometry considerations. Following the strict Organization for Economic Co-operation and Development (OECD) guidelines, we have developed models using stepwise regression analysis followed by the best subset selection, while the final model was developed by partial least squares regression technique. The model was validated using various internationally accepted stringent validation parameters. From the insights obtained from the developed model, we have concluded that heteroatoms (nitrogen, oxygen, etc.) present within to an aromatic nucleus and the structural features such as hydrophobic, ring aromatic and hydrogen bond acceptor/donor are responsible for the enhancement of the BACE1 enzyme inhibitory activity. Moreover, we have performed the pharmacophore modelling to unveil the structural requirements for the inhibitory activity against the BACE1 enzyme. Furthermore, molecular docking studies were carried out to understand the molecular interactions involved in binding, and the results are then correlated with the requisite structural features obtained from the QSAR and pharmacophore models.

An Empirical Evaluation of User Movement Data on Smartphones

Movement data can be collected and used to add new security features and functionality to users’ mobile devices. Measuring a user’s movement using mobile devices allows for the use of behavioral biometrics. This assessment could introduce a shift in our current methods for securing mobile devices: instead of physical attributes like fingerprints or our face, the use of behavioral attributes like the way we walk or perform some personal activity. In this paper, an empirical evaluation of different classification techniques is conducted on user movement data. The datasets used in this empirical evaluation contain accelerometer data that were collected during various experiments from several mobile devices, including smartphones, smart watches, and other accelerometer sensors. We aggregated the user movement data and provided them as input into five traditional machine learning algorithms. The classification performances of the data were compared with a deep learning technique, the Long Short-Term Memory-Recurrent Neural Network (LSTM-RNN). The LSTM-RNN achieved its highest accuracy at 89% compared to 97% from a traditional machine learning algorithm, specifically the k-Nearest Neighbor (k-NN) algorithm on wrist-worn accelerometer data, thus showing the LSTM to be a less viable option.

Touch-Based Active Cloud Authentication Using Traditional Machine Learning and LSTM on a Distributed Tensorflow Framework

In this modern world, mobile devices have been paired with the cloud environment to scale the voluminous amount of generated data. The implementation comes at the cost of privacy as proprietary data can be stolen in transit to the cloud, or victims’ phones can be seized along with synced data from cloud. The attacker can gain access to the phone through shoulder surfing, or even spoofing attacks. Our approach is to mitigate this issue by proposing an active cloud authentication framework using touch biometric pattern. To the best of our knowledge, active cloud authentication using touch dynamics for mobile cloud computing has not been explored in the literature. This research creates a proof of concept that will lead into a simulated cloud framework for active authentication. Given the amount of data captured by the mobile device from user activity, it can be a computationally intensive process for the mobile device to handle with such limited resources. To solve this, we simulated a post-transmission process of data to the cloud so that we could implement the authentication process within the cloud. We evaluated the touch data using traditional machine learning algorithms, such as Random Forest (RF), Support Vector Machine (SVM), and also using a deep learning classifier, the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) algorithms. The novelty of this work is two-fold. First, we develop a distributed tensorflow framework for cloud authentication using touch biometric pattern. This framework helps alleviate the drawback of the computationally intensive recognition of the substantial amount of raw data from the user. Second, we apply the RF, SVM, and a deep learning classifier, the LSTM-RNN, on the touch data to evaluate the performance of the proposed authentication scheme. The proposed approach shows a promising performance with an accuracy of 99.0361% using RF on the distributed tensorflow framework.

Linear Predictive Coefficients-Based Feature to Identify Top-Seven Spoken Languages

Speech recognition in multilingual scenario is not trivial in the case when multiple languages are used in one conversation. Language must be identified before we process speech recognition as such tools are language-dependent. We present a language identification system (or AI tool) to distinguish top-seven world languages namely Chinese, Spanish, English, Hindi, Arabic, Bangla and Portuguese [G. F. Simons and C. D. Fennig (eds.), Ethnologue: Laguage of the Americas and the Pacific, Twentieth Edn. (SIL Internatinal, 2017)]. The system uses linear predictive coefficients-based feature, i.e. the line spectral pair–grade ratio (LSP–GR) feature, and ensemble learning for classification. Experiments were performed on more than 200[Formula: see text]h of real-world YouTube data and the highest possible accuracy of 96.95% was received. The results can be compared with other machine learning classifiers.