A factorial-analysis-based Bayesian neural network method for quantifying China's CO2 emissions under dual-carbon target

Energy-structure transformation and CO2-emission reduction are becoming particularly urgent for China and many other countries. Development of effective methods that are capable of quantifying and predicting CO2 emissions to achieve carbon neutrality is desired. This study advances a factorial-analysis-based Bayesian neural network (abbreviated as FABNN) method to reflect the complex relationship between inputs and outputs as well as reveal the individual and interactive effects of multiple factors affecting CO2 emissions. FABNN is then applied to analyzing CO2 emissions of China (abbreviated as CEC), where multiple factors involve in energy (e.g., the consumption of natural gas, CONG), economic (e.g., Gross domestic product, GDP) and social (e.g., the rate of urbanization, ROU) aspects are investigated and 512 scenarios are designed to achieve the national dual carbon targets (i.e., carbon peak before 2030 and carbon neutrality by 2060). Comparing to the conventional machine learning methods, FABNN performs better in calibration and validation results, indicating that FABNN is suitable for CEC simulation and prediction. Results disclose that the top three factors affecting CEC under the dual‑carbon target are GDP, CONG, and ROU; energy, economic and social contributions are 43.5 %, 34.6 % and 21.9 %, respectively. CEC reaches its carbon peak during 2027-2032 and achieve carbon neutrality during 2053-2057 under all scenarios. Under the optimal scenario (S195), the CO2-emission reduction potential is about 772.2 million tonnes and the consumptions of coal, petroleum and natural gas can be respectively reduced by 3.1 %, 9.9 % and 23.0 % compared to the worst scenario (S466). The results can provide solid support for national energy-structure transformation and CO2-emission reduction to achieve carbon-peak and carbon-neutrality targets.

American society of anesthesiologists physical status classification significantly affects the performances of machine learning models in intraoperative hypotension inference

STUDY OBJECTIVE

To explore how American Society of Anesthesiologists (ASA) physical status classification affects different machine learning models in hypotension prediction and whether the prediction uncertainty could be quantified.

DESIGN

Observational Studies SETTING: UofL health hospital PATIENTS: This study involved 562 hysterectomy surgeries performed on patients (≥ 18 years) between June 2020 and July 2021.

INTERVENTIONS

None MEASUREMENTS: Preoperative and intraoperative data is collected. Three parametric machine learning models, including Bayesian generalized linear model (BGLM), Bayesian neural network (BNN), a newly proposed BNN with multivariate mixed responses (BNNMR), and one nonparametric model, Gaussian Process (GP), were explored to predict patients' diastolic and systolic blood pressures (continuous responses) and patients' hypotensive event (binary response) for the next five minutes. Data was separated into American Society of Anesthesiologists (ASA) physical status class 1- 4 before being read in by four machine learning models. Statistical analysis and models' constructions are performed in Python. Sensitivity, specificity, and the confidence/credible intervals were used to evaluate the prediction performance of each model for each ASA physical status class.

MAIN RESULTS

ASA physical status classes require distinct models to accurately predict intraoperative blood pressures and hypotensive events. Overall, high sensitivity (above 0.85) and low uncertainty can be achieved by all models for ASA class 4 patients. In contrast, models trained without controlling ASA classes yielded lower sensitivity (below 0.5) and larger uncertainty. Particularly, in terms of predicting binary hypotensive event, for ASA physical status class 1, BNNMR yields the highest sensitivity of 1. For classes 2 and 3, BNN has the highest sensitivity of 0.429 and 0.415, respectively. For class 4, BNNMR and GP are tied with the highest sensitivity of 0.857. On the other hand, the sensitivity is just 0.031, 0.429, 0.165 and 0.305 for BNNMR, BNN, GBLM and GP models respectively, when training data is not divided by ASA physical status classes. In terms of predicting systolic blood pressure, the GP regression yields the lowest root mean squared errors (RMSE) of 2.072, 7.539, 9.214 and 0.295 for ASA physical status classes 1, 2, 3 and 4, respectively, but a RMSE of 126.894 if model is trained without controlling the ASA physical status class. The RMSEs for other models are far higher. RMSEs are 2.175, 13.861, 17.560 and 22.426 for classes 1, 2, 3 and 4 respectively for the BGLM. In terms of predicting diastolic blood pressure, the GP regression yields the lowest RMSEs of 2.152, 6.573, 5.371 and 0.831 for ASA physical status classes 1, 2, 3 and 4, respectively; RMSE of 8.084 if model is trained without controlling the ASA physical status class. The RMSEs for other models are far higher. Finally, in terms of the width of the 95% confidence interval of the mean prediction for systolic and diastolic blood pressures, GP regression gives narrower confidence interval with much smaller margin of error across all four ASA physical status classes.

CONCLUSIONS

Different ASA physical status classes present different data distributions, and thus calls for distinct machine learning models to improve prediction accuracy and reduce predictive uncertainty. Uncertainty quantification enabled by Bayesian inference provides valuable information for clinicians as an additional metric to evaluate performance of machine learning models for medical decision making.

Large-scale spatiotemporal deep learning predicting urban residential indoor PM2.5 concentration

Indoor PM2.5 pollution is one of the leading causes of death and disease worldwide. As monitoring indoor PM2.5 concentrations on a large scale is challenging, it is urgent to assess population-level exposure and related health risks to develop an easy-to-use and generalized model to predict indoor PM2.5 concentrations and spatiotemporal variations at the global level. Existing machine learning models of indoor PM2.5 are prone to deliver single-point predictions, and their input strategies are not widely applicable. Here, we developed a Bayesian neural network (BNN) model for predicting the distribution of daily average urban residential PM2.5 concentration based on multiple data sources available from nationwide comprehensive sensor-monitoring records in China. The BNN model showed good performance with a 10-fold cross-validation R2 of 0.70, mean-absolute-error of 9.45 μg/m3, root-mean-square error of 13.3 μg/m3, and 95 % prediction interval coverage of 85 %. To demonstrate the application process, this model was applied to predict indoor PM2.5 concentrations on a large spatiotemporal scale. Our modeled population-weighted annual indoor PM2.5 concentration for China in 2019 was 22.8 μg/m3, far exceeding the WHO standard. The validity of the model at the population level can be further bolstered, making it valuable for assessing and managing indoor air pollution-related health risks.

Development and evaluation of a robotic system for lumbar puncture and epidural steroid injection

Introduction

Lumbar puncture is an important medical procedure for various diagnostics and therapies, but it can be hazardous due to individual variances in subcutaneous soft tissue, especially in the elderly and obese. Our research describes a novel robot-assisted puncture system that automatically controls and maintains the probe at the target tissue layer through a process of tissue recognition.

Methods

The system comprises a robotic system and a master computer. The robotic system is constructed based on a probe consisting of a pair of concentric electrodes. From the probe, impedance spectroscopy measures bio-impedance signals and transforms them into spectra that are communicated to the master computer. The master computer uses a Bayesian neural network to classify the bio-impedance spectra as corresponding to different soft tissues. By feeding the bio-impedance spectra of unknown tissues into the Bayesian neural network, we can determine their categories. Based on the recognition results, the master computer controls the motion of the robotic system.

Results

The proposed system is demonstrated on a realistic phantom made of ex vivo tissues to simulate the spinal environment. The findings indicate that the technology has the potential to increase the precision and security of lumbar punctures and associated procedures.

Discussion

In addition to lumbar puncture, the robotic system is suitable for related puncture operations such as discography, radiofrequency ablation, facet joint injection, and epidural steroid injection, as long as the required tissue recognition features are available. These operations can only be carried out once the puncture needle and additional instruments reach the target tissue layer, despite their ensuing processes being distinct.

Applications of Bayesian Neural Networks in Outlier Detection

Anomaly detection is crucial in a variety of domains, such as fraud detection, disease diagnosis, and equipment defect detection. With the development of deep learning, anomaly detection with Bayesian neural networks (BNNs) becomes a novel research topic in recent years. This article aims to propose a widely applicable method of outlier detection (a category of anomaly detection) using BNNs based on uncertainty measurement. There are three kinds of uncertainties generated in the prediction of BNNs: epistemic uncertainty, aleatoric uncertainty, and (model) misspecification uncertainty. Although the approaches in previous studies are adopted to measure epistemic and aleatoric uncertainty, a new method of utilizing loss functions to quantify misspecification uncertainty is proposed in this article. Then, these three uncertainty sources are merged together by specific combination models to construct total prediction uncertainty. In this study, the key idea is that the observations with high total prediction uncertainty should correspond to outliers in the data. The method of this research is applied to the experiments on Modified National Institute of Standards and Technology (MNIST) dataset and Taxi dataset, respectively. From the results, if the network is appropriately constructed and well-trained and model parameters are carefully tuned, most anomalous images in MNIST dataset and all the abnormal traffic periods in Taxi dataset can be nicely detected. In addition, the performance of this method is compared with the BNN anomaly detection methods proposed before and the classical Local Outlier Factor and Density-Based Spatial Clustering of Applications with Noise methods. This study links the classification of uncertainties in essence with anomaly detection and takes the lead to consider combining different uncertainty sources to reform detection outcomes instead of using only single uncertainty each time.

A Design Methodology for Fault-Tolerant Neuromorphic Computing Using Bayesian Neural Network

Memristor crossbar arrays are a promising platform for neuromorphic computing. In practical scenarios, the synapse weights represented by the memristors for the underlying system are subject to process variations, in which the programmed weight when read out for inference is no longer deterministic but a stochastic distribution. It is therefore highly desired to learn the weight distribution accounting for process variations, to ensure the same inference performance in memristor crossbar arrays as the design value. In this paper, we introduce a design methodology for fault-tolerant neuromorphic computing using a Bayesian neural network, which combines the variational Bayesian inference technique with a fault-aware variational posterior distribution. The proposed framework based on Bayesian inference incorporates the impacts of memristor deviations into algorithmic training, where the weight distributions of neural networks are optimized to accommodate uncertainties and minimize inference degradation. The experimental results confirm the capability of the proposed methodology to tolerate both process variations and noise, while achieving more robust computing in memristor crossbar arrays.

Prediction of Shooting Events in Soccer Videos Using Complete Bipartite Graphs and Players' Spatial-Temporal Relations

In soccer, quantitatively evaluating the performance of players and teams is essential to improve tactical coaching and players' decision-making abilities. To achieve this, some methods use predicted probabilities of shoot event occurrences to quantify player performances, but conventional shoot prediction models have not performed well and have failed to consider the reliability of the event probability. This paper proposes a novel method that effectively utilizes players' spatio-temporal relations and prediction uncertainty to predict shoot event occurrences with greater accuracy and robustness. Specifically, we represent players' relations as a complete bipartite graph, which effectively incorporates soccer domain knowledge, and capture latent features by applying a graph convolutional recurrent neural network (GCRNN) to the constructed graph. Our model utilizes a Bayesian neural network to predict the probability of shoot event occurrence, considering spatio-temporal relations between players and prediction uncertainty. In our experiments, we confirmed that the proposed method outperformed several other methods in terms of prediction performance, and we found that considering players' distances significantly affects the prediction accuracy.

Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability

Bayesian neural network (BNN) allows for uncertainty quantification in prediction, offering an advantage over regular neural networks that has not been explored in the differential privacy (DP) framework. We fill this important gap by leveraging recent development in Bayesian deep learning and privacy accounting to offer a more precise analysis of the trade-off between privacy and accuracy in BNN. We propose three DP-BNNs that characterize the weight uncertainty for the same network architecture in distinct ways, namely DP-SGLD (via the noisy gradient method), DP-BBP (via changing the parameters of interest) and DP-MC Dropout (via the model architecture). Interestingly, we show a new equivalence between DP-SGD and DP-SGLD, implying that some non-Bayesian DP training naturally allows for uncertainty quantification. However, the hyperparameters such as learning rate and batch size, can have different or even opposite effects in DP-SGD and DP-SGLD. Extensive experiments are conducted to compare DP-BNNs, in terms of privacy guarantee, prediction accuracy, uncertainty quantification, calibration, computation speed, and generalizability to network architecture. As a result, we observe a new tradeoff between the privacy and the reliability. When compared to non-DP and non-Bayesian approaches, DP-SGLD is remarkably accurate under strong privacy guarantee, demonstrating the great potential of DP-BNN in real-world tasks.

Multi-energy CT material decomposition using Bayesian deep convolutional neural network with explicit penalty of uncertainty and bias

Convolutional neural network (CNN)-based material decomposition has the potential to improve image quality (visual appearance) and quantitative accuracy of material maps. Most methods use deterministic CNNs with mean-square-error loss to provide point-estimates of mass densities. Point estimates can be over-confident as the reliability of CNNs is frequently compromised by bias and two major uncertainties - data and model uncertainties originating from noise in inputs and train-test data dissimilarity, respectively. Also, mean-square-error lacks explicit control of uncertainty and bias. To tackle these problems, a Bayesian dual-task CNN (BDT-CNN) with explicit penalization of uncertainty and bias was developed. It is a probabilistic CNN that concurrently conducts material classification and quantification and allows for pixel-wise modeling of bias, data uncertainty, and model uncertainty. CNN was trained with images of physical and simulated tissue-mimicking inserts at varying mass densities. Hydroxyapatite (nominal density 400mg/cc) and blood (nominal density 1095mg/cc) inserts were placed in different-sized body phantoms (30 - 45cm) and used to evaluate mean-absolute-bias (MAB) in predicted mass densities across different images at routine- and half-routine-dose. Patient CT exams were collected to assess generalizability of BDT-CNN in the presence of anatomical background. Noise insertion was used to simulate patient exams at half- and quarter-routine-dose. The deterministic dual-task CNN was used as baseline. In phantoms, BDT-CNN improved consistency of insert delineation, especially edges, and reduced overall bias (average MAB for hydroxyapatite: BDT-CNN 5.4mgHA/cc, baseline 11.0mgHA/cc and blood: BDT-CNN 8.9mgBlood/cc, baseline 14.0mgBlood/cc). In patient images, BDT-CNN improved detail preservation, lesion conspicuity, and structural consistency across different dose levels.

Patient-specific uncertainty and bias quantification of non-transparent convolutional neural network model through knowledge distillation and Bayesian deep learning

Assessing the reliability of convolutional neural network (CNN)-based CT imaging techniques is critical for reliable deployment in practice. Some evaluation methods exist but require full access to target CNN architecture and training data, something not available for proprietary or commercial algorithms. Moreover, there is a lack of systematic evaluation methods. To address these issues, we propose a patient-specific uncertainty and bias quantification (UNIQ) method that integrates knowledge distillation and Bayesian deep learning. Knowledge distillation creates a transparent CNN ("Student CNN") to approximate the target non-transparent CNN ("Teacher CNN"). Student CNN is built as a Bayesian-deep-learning-based probabilistic CNN that, for each input, always generates statistical distribution of the corresponding outputs, and characterizes predictive mean and two major uncertainties - data and model uncertainty. UNIQ was evaluated using a low-dose CT denoising task. Patient and phantom scans with routine-dose and synthetic quarter-dose were used to create training, validation, and testing sets. To demonstrate, Unet and Resnet were used as backbones of Teacher CNN and Student CNN respectively and were trained using independent training sets. Student Resnet was qualitatively and quantitatively evaluated. The pixel-wise predictive mean, data uncertainty, and model uncertainty from Student Resnet were very similar to the counterparts from Teacher Unet (mean-absolute-error: predictive mean 1.5HU, data uncertainty 1.8HU, model uncertainty 1.3HU; mean 2D correlation coefficient: total uncertainty 0.90, data uncertainty 0.86, model uncertainty 0.83). The proposed UNIQ can potentially systematically characterize the reliability of non-transparent CNN models used in CT.

RAMP: response-aware multi-task learning with contrastive regularization for cancer drug response prediction

The accurate prediction of cancer drug sensitivity according to the multiomics profiles of individual patients is crucial for precision cancer medicine. However, the development of prediction models has been challenged by the complex crosstalk of input features and the resistance-dominant drug response information contained in public databases. In this study, we propose a novel multidrug response prediction framework, response-aware multitask prediction (RAMP), via a Bayesian neural network and restrict it by soft-supervised contrastive regularization. To utilize network embedding vectors as representation learning features for heterogeneous networks, we harness response-aware negative sampling, which applies cell line-drug response information to the training of network embeddings. RAMP overcomes the prediction accuracy limitation induced by the imbalance of trained response data based on the comprehensive selection and utilization of drug response features. When trained on the Genomics of Drug Sensitivity in Cancer dataset, RAMP achieved an area under the receiver operating characteristic curve > 89%, an area under the precision-recall curve > 59% and an $\textrm{F}_1$ score > 52% and outperformed previously developed methods on both balanced and imbalanced datasets. Furthermore, RAMP predicted many missing drug responses that were not included in the public databases. Our results showed that RAMP will be suitable for the high-throughput prediction of cancer drug sensitivity and will be useful for guiding cancer drug selection processes. The Python implementation for RAMP is available at https://github.com/hvcl/RAMP.

Machine Learning Derived Lifting Techniques and Pain Self-Efficacy in People with Chronic Low Back Pain

This paper proposes an innovative methodology for finding how many lifting techniques people with chronic low back pain (CLBP) can demonstrate with camera data collected from 115 participants. The system employs a feature extraction algorithm to calculate the knee, trunk and hip range of motion in the sagittal plane, Ward’s method, a combination of K-means and Ensemble clustering method for classification algorithm, and Bayesian neural network to validate the result of Ward’s method and the combination of K-means and Ensemble clustering method. The classification results and effect size show that Ward clustering is the optimal method where precision and recall percentages of all clusters are above 90, and the overall accuracy of the Bayesian Neural Network is 97.9%. The statistical analysis reported a significant difference in the range of motion of the knee, hip and trunk between each cluster, F (9, 1136) = 195.67, p < 0.0001. The results of this study suggest that there are four different lifting techniques in people with CLBP. Additionally, the results show that even though the clusters demonstrated similar pain levels, one of the clusters, which uses the least amount of trunk and the most knee movement, demonstrates the lowest pain self-efficacy.

Bayesian deep learning-based 1 H-MRS of the brain: Metabolite quantification with uncertainty estimation using Monte Carlo dropout

PURPOSE

To develop a Bayesian convolutional neural network (BCNN) with Monte Carlo dropout sampling for metabolite quantification with simultaneous uncertainty estimation in deep learning-based proton MRS of the brain.

METHODS

Human brain spectra were simulated using basis spectra for 17 metabolites and macromolecules (N = 100 000) at 3.0 Tesla. In addition, actual in vivo spectra (N = 5) were modified by adjusting SNR and linewidth with increasing severity of spectral degradation (N = 50). A BCNN was trained on the simulated spectra to generate a noise-free, line-narrowed, macromolecule signal-removed, metabolite-only spectrum from a typical human brain spectrum. At inference, each input spectrum was Monte Carlo dropout sampled (50 times), and the resulting mean spectrum and variance spectrum were used for metabolite quantification and uncertainty estimation, respectively.

RESULTS

Using the simulated spectra, the mean absolute percent errors of the BCNN-predicted metabolite content were < 10% for Cr, Glu, Gln, mI, NAA, and Tau (< 5% for Glu, NAA, and mI). For all metabolites, the correlations (r's) between the ground-truth error and BCNN-predicted uncertainty ranged 0.72-0.94 (0.83 ± 0.06; p < 0.001). Using the modified in vivo spectra, the extent of variation in the estimated metabolite content against the increasing severity of spectral degradation tended to be smaller with BCNN than with linear combination of model spectra (LCModel). Overall, the variation in metabolite content tended to be more highly correlated with the uncertainty from BCNN than with the Cramér-Rao lower-bounds from LCModel (0.938 ± 0.019 vs. 0.881 ± 0.057 [p = 0.115]).

CONCLUSION

The BCNN with Monte Carlo dropout sampling may be used in deep learning-based MRS for the estimation of uncertainty in the machine-predicted metabolite content, which is important in the clinical application of deep learning-based MRS.

Learning Sparse Deep Neural Networks with a Spike-and-Slab Prior

Deep learning has achieved great successes in many machine learning tasks. However, the deep neural networks (DNNs) are often severely over-parameterized, making them computationally expensive, memory intensive, less interpretable and mis-calibrated. We study sparse DNNs under the Bayesian framework: we establish posterior consistency and structure selection consistency for Bayesian DNNs with a spike-and-slab prior, and illustrate their performance using examples on high-dimensional nonlinear variable selection, large network compression and model calibration. Our numerical results indicate that sparsity is essential for improving the prediction accuracy and calibration of the DNN.

Winsorization for Robust Bayesian Neural Networks

With the advent of big data and the popularity of black-box deep learning methods, it is imperative to address the robustness of neural networks to noise and outliers. We propose the use of Winsorization to recover model performances when the data may have outliers and other aberrant observations. We provide a comparative analysis of several probabilistic artificial intelligence and machine learning techniques for supervised learning case studies. Broadly, Winsorization is a versatile technique for accounting for outliers in data. However, different probabilistic machine learning techniques have different levels of efficiency when used on outlier-prone data, with or without Winsorization. We notice that Gaussian processes are extremely vulnerable to outliers, while deep learning techniques in general are more robust.

Prediction of anxiety disorders using a feature ensemble based bayesian neural network

Anxiety disorders are common among youth, posing risks to physical and mental health development. Early screening can help identify such disorders and pave the way for preventative treatment. To this end, the Youth Online Diagnostic Assessment (YODA) tool was developed and deployed to predict youth disorders using online screening questionnaires filled by parents. YODA facilitated collection of several novel unique datasets of self-reported anxiety disorder symptoms. Since the data is self-reported and often noisy, feature selection needs to be performed on the raw data to improve accuracy. However, a single set of selected features may not be informative enough. Consequently, in this work we propose and evaluate a novel feature ensemble based Bayesian Neural Network (FE-BNN) that exploits an ensemble of features for improving the accuracy of disorder predictions. We evaluate the performance of FE-BNN on three disorder-specific datasets collected by YODA. Our method achieved the AUC of 0.8683, 0.8769, 0.9091 for the predictions of Separation Anxiety Disorder, Generalized Anxiety Disorder and Social Anxiety Disorder, respectively. These results provide initial evidence that our method outperforms the original diagnostic scoring function of YODA and several other baseline methods for three anxiety disorders, which can practically help prioritizing diagnostic interviews. Our promising results call for investigation of interpretable methods maintaining high predictive accuracy.

A hybrid framework for improving uncertainty quantification in deep learning-based QSAR regression modeling

Reliable uncertainty quantification for statistical models is crucial in various downstream applications, especially for drug design and discovery where mistakes may incur a large amount of cost. This topic has therefore absorbed much attention and a plethora of methods have been proposed over the past years. The approaches that have been reported so far can be mainly categorized into two classes: distance-based approaches and Bayesian approaches. Although these methods have been widely used in many scenarios and shown promising performance with their distinct superiorities, being overconfident on out-of-distribution examples still poses challenges for the deployment of these techniques in real-world applications. In this study we investigated a number of consensus strategies in order to combine both distance-based and Bayesian approaches together with post-hoc calibration for improved uncertainty quantification in QSAR (Quantitative Structure-Activity Relationship) regression modeling. We employed a set of criteria to quantitatively assess the ranking and calibration ability of these models. Experiments based on 24 bioactivity datasets were designed to make critical comparison between the model we proposed and other well-studied baseline models. Our findings indicate that the hybrid framework proposed by us can robustly enhance the model ability of ranking absolute errors. Together with post-hoc calibration on the validation set, we show that well-calibrated uncertainty quantification results can be obtained in domain shift settings. The complementarity between different methods is also conceptually analyzed.

Enabling Training of Neural Networks on Noisy Hardware

Deep neural networks (DNNs) are typically trained using the conventional stochastic gradient descent (SGD) algorithm. However, SGD performs poorly when applied to train networks on non-ideal analog hardware composed of resistive device arrays with non-symmetric conductance modulation characteristics. Recently we proposed a new algorithm, the Tiki-Taka algorithm, that overcomes this stringent symmetry requirement. Here we build on top of Tiki-Taka and describe a more robust algorithm that further relaxes other stringent hardware requirements. This more robust second version of the Tiki-Taka algorithm (referred to as TTv2) 1. decreases the number of device conductance states requirement from 1000s of states to only 10s of states, 2. increases the noise tolerance to the device conductance modulations by about 100x, and 3. increases the noise tolerance to the matrix-vector multiplication performed by the analog arrays by about 10x. Empirical simulation results show that TTv2 can train various neural networks close to their ideal accuracy even at extremely noisy hardware settings. TTv2 achieves these capabilities by complementing the original Tiki-Taka algorithm with lightweight and low computational complexity digital filtering operations performed outside the analog arrays. Therefore, the implementation cost of TTv2 compared to SGD and Tiki-Taka is minimal, and it maintains the usual power and speed benefits of using analog hardware for training workloads. Here we also show how to extract the neural network from the analog hardware once the training is complete for further model deployment. Similar to Bayesian model averaging, we form analog hardware compatible averages over the neural network weights derived from TTv2 iterates. This model average then can be transferred to another analog or digital hardware with notable improvements in test accuracy, transcending the trained model itself. In short, we describe an end-to-end training and model extraction technique for extremely noisy crossbar-based analog hardware that can be used to accelerate DNN training workloads and match the performance of full-precision SGD.

A Bayesian Deep Neural Network for Safe Visual Servoing in Human-Robot Interaction

Safety is an important issue in human–robot interaction (HRI) applications. Various research works have focused on different levels of safety in HRI. If a human/obstacle is detected, a repulsive action can be taken to avoid the collision. Common repulsive actions include distance methods, potential field methods, and safety field methods. Approaches based on machine learning are less explored regarding the selection of the repulsive action. Few research works focus on the uncertainty of the data-based approaches and consider the efficiency of the executing task during collision avoidance. In this study, we describe a system that can avoid collision with human hands while the robot is executing an image-based visual servoing (IBVS) task. We use Monte Carlo dropout (MC dropout) to transform a deep neural network (DNN) to a Bayesian DNN, and learn the repulsive position for hand avoidance. The Bayesian DNN allows IBVS to converge faster than the opposite repulsive pose. Furthermore, it allows the robot to avoid undesired poses that the DNN cannot avoid. The experimental results show that Bayesian DNN has adequate accuracy and can generalize well on unseen data. The predictive interval coverage probability (PICP) of the predictions along x, y, and z directions are 0.84, 0.94, and 0.95, respectively. In the space which is unseen in the training data, the Bayesian DNN is also more robust than a DNN. We further implement the system on a UR10 robot, and test the robustness of the Bayesian DNN and the IBVS convergence speed. Results show that the Bayesian DNN can avoid the poses out of the reach range of the robot and it lets the IBVS task converge faster than the opposite repulsive pose.¹

Predictions of bitcoin prices through machine learning based frameworks

The high volatility of an asset in financial markets is commonly seen as a negative factor. However short-term trades may entail high profits if traders open and close the correct positions. The high volatility of cryptocurrencies, and in particular of Bitcoin, is what made cryptocurrency trading so profitable in these last years. The main goal of this work is to compare several frameworks each other to predict the daily closing Bitcoin price, investigating those that provide the best performance, after a rigorous model selection by the so-called k-fold cross validation method. We evaluated the performance of one stage frameworks, based only on one machine learning technique, such as the Bayesian Neural Network, the Feed Forward and the Long Short Term Memory Neural Networks, and that of two stages frameworks formed by the neural networks just mentioned in cascade to Support Vector Regression. Results highlight higher performance of the two stages frameworks with respect to the correspondent one stage frameworks, but for the Bayesian Neural Network. The one stage framework based on Bayesian Neural Network has the highest performance and the order of magnitude of the mean absolute percentage error computed on the predicted price by this framework is in agreement with those reported in recent literature works.

Compensating for visibility artefacts in photoacoustic imaging with a deep learning approach providing prediction uncertainties

Conventional photoacoustic imaging may suffer from the limited view and bandwidth of ultrasound transducers. A deep learning approach is proposed to handle these problems and is demonstrated both in simulations and in experiments on a multi-scale model of leaf skeleton. We employed an experimental approach to build the training and the test sets using photographs of the samples as ground truth images. Reconstructions produced by the neural network show a greatly improved image quality as compared to conventional approaches. In addition, this work aimed at quantifying the reliability of the neural network predictions. To achieve this, the dropout Monte-Carlo procedure is applied to estimate a pixel-wise degree of confidence on each predicted picture. Last, we address the possibility to use transfer learning with simulated data in order to drastically limit the size of the experimental dataset.

Left ventricle quantification with sample-level confidence estimation via Bayesian neural network

Quantification of cardiac left ventricle has become a hot topic due to its great significance in clinical practice. Many efforts have been devoted to LV quantification and obtained promising performance with the help of various deep neural networks when validated on a group of samples. However, none of them can provide sample-level confidence of the results, i.e., how reliable is the prediction result for one single sample, which would help clinicians make decisions of whether or not to accept the automatic results. In this paper, we achieve this by introducing the uncertainty analysis theory into our LV quantification network. Two types of uncertainty, Model Uncertainty, and Data Uncertainty are analyzed for the quantification performance and contribute to the sample-level confidence. Experiments with data of 145 subjects validate that our method not only improved the quantification performance with an uncertainty-weighted regression loss but also is capable of providing for each sample the confidence level of the estimation results for clinicians' further consideration.

Wearable IoT Smart-Log Patch: An Edge Computing-Based Bayesian Deep Learning Network System for Multi Access Physical Monitoring System

According to the survey on various health centres, smart log-based multi access physical monitoring system determines the health conditions of humans and their associated problems present in their lifestyle. At present, deficiency in significant nutrients leads to deterioration of organs, which creates various health problems, particularly for infants, children, and adults. Due to the importance of a multi access physical monitoring system, children and adolescents’ physical activities should be continuously monitored for eliminating difficulties in their life using a smart environment system. Nowadays, in real-time necessity on multi access physical monitoring systems, information requirements and the effective diagnosis of health condition is the challenging task in practice. In this research, wearable smart-log patch with Internet of Things (IoT) sensors has been designed and developed with multimedia technology. Further, the data computation in that smart-log patch has been analysed using edge computing on Bayesian deep learning network (EC-BDLN), which helps to infer and identify various physical data collected from the humans in an accurate manner to monitor their physical activities. Then, the efficiency of this wearable IoT system with multimedia technology is evaluated using experimental results and discussed in terms of accuracy, efficiency, mean residual error, delay, and less energy consumption. This state-of-the-art smart-log patch is considered as one of evolutionary research in health checking of multi access physical monitoring systems with multimedia technology.

Probabilistic Damage Detection of a Steel Truss Bridge Model by Optimally Designed Bayesian Neural Network

Excellent pattern matching capability makes artificial neural networks (ANNs) a very promising approach for vibration-based structural health monitoring (SHM). The proper design of the network architecture with the suitable complexity is vital to the ANN-based structural damage detection. In addition to the number of hidden neurons, the type of transfer function used in the hidden layer cannot be neglected for the ANN design. Neural network learning can be further presented in the framework of Bayesian statistics, but the issues of selection for the hidden layer transfer function with respect to the Bayesian neural network has not yet been reported in the literature. In addition, most of the research works in the literature for addressing the predictive distribution of neural network output is only for a single target variable, while multiple target variables are rarely involved. In the present paper, for the purpose of probabilistic structural damage detection, Bayesian neural networks with multiple target variables are optimally designed, and the selection of the number of neurons, and the transfer function in the hidden layer, are carried out simultaneously to achieve a neural network architecture with suitable complexity. Furthermore, the nonlinear network function can be approximately linear by assuming the posterior distribution of network parameters is a sufficiently narrow Gaussian, and then the input-dependent covariance matrix of the predictive distribution of network output can be obtained with the Gaussian assumption for the situation of multiple target variables. Structural damage detection is conducted for a steel truss bridge model to verify the proposed method through a set of numerical case studies.