Detection of Malicious Threats Exploiting Clock-Gating Hardware Using Machine Learning

Embedded system technologies are increasingly being incorporated into manufacturing, smart grid, industrial control systems, and transportation systems. However, the vast majority of today's embedded platforms lack the support of built-in security features which makes such systems highly vulnerable to a wide range of cyber-attacks. Specifically, they are vulnerable to malware injection code that targets the power distribution system of an ARM Cortex-M-based microcontroller chipset (ARM, Cambridge, UK). Through hardware exploitation of the clock-gating distribution system, an attacker is capable of disabling/activating various subsystems on the chip, compromising the reliability of the system during normal operation. This paper proposes the development of an Intrusion Detection System (IDS) capable of detecting clock-gating malware deployed on ARM Cortex-M-based embedded systems. To enhance the robustness and effectiveness of our approach, we fully implemented, tested, and compared six IDSs, each employing different methodologies. These include IDSs based on K-Nearest Classifier, Random Forest, Logistic Regression, Decision Tree, Naive Bayes, and Stochastic Gradient Descent. Each of these IDSs was designed to identify and categorize various variants of clock-gating malware deployed on the system. We have analyzed the performance of these IDSs in terms of detection accuracy against various types of clock-gating malware injection code. Power consumption data collected from the chipset during normal operation and malware code injection attacks were used for models' training and validation. Our simulation results showed that the proposed IDSs, particularly those based on K-Nearest Classifier and Logistic Regression, were capable of achieving high detection rates, with some reaching a detection rate of 0.99. These results underscore the effectiveness of our IDSs in protecting ARM Cortex-M-based embedded systems against clock-gating malware.

Control Software Design for a Multisensing Multicellular Microscale Gas Chromatography System

Microscale gas chromatography (μGC) systems are miniaturized instruments that typically incorporate one or several microfabricated fluidic elements; such systems are generally well suited for the automated sampling and analysis of gas-phase chemicals. Advanced μGC systems may incorporate more than 15 elements and operate these elements in different coordinated sequences to execute complex operations. In particular, the control software must manage the sampling and analysis operations of the μGC system in a time-sensitive manner; while operating multiple control loops, it must also manage error conditions, data acquisition, and user interactions when necessary. To address these challenges, this work describes the investigation of multithreaded control software and its evaluation with a representative μGC system. The μGC system is based on a progressive cellular architecture that uses multiple μGC cells to efficiently broaden the range of chemical analytes, with each cell incorporating multiple detectors. Implemented in Python language version 3.7.3 and executed by an embedded single-board computer, the control software enables the concurrent control of heaters, pumps, and valves while also gathering data from thermistors, pressure sensors, capacitive detectors, and photoionization detectors. A graphical user interface (UI) that operates on a laptop provides visualization of control parameters in real time. In experimental evaluations, the control software provided successful operation and readout for all the components, including eight sets of thermistors and heaters that form temperature feedback loops, two sets of pressure sensors and tunable gas pumps that form pressure head feedback loops, six capacitive detectors, three photoionization detectors, six valves, and an additional fixed-flow gas pump. A typical run analyzing 18 chemicals is presented. Although the operating system does not guarantee real-time operation, the relative standard deviations of the control loop timings were <0.5%. The control software successfully supported >1000 μGC runs that analyzed various chemical mixtures.

Study of the Impact of Data Compression on the Energy Consumption Required for Data Transmission in a Microcontroller-Based System

As the number of Internet of Things (IoT) devices continues to rise dramatically each day, the data generated and transmitted by them follow similar trends. Given that a significant portion of these embedded devices operate on battery power, energy conservation becomes a crucial factor in their design. This paper aims to investigate the impact of data compression on the energy consumption required for data transmission. To achieve this goal, we conduct a comprehensive study using various transmission modules in a severely resource-limited microcontroller-based system designed for battery power. Our study evaluates the performance of several compression algorithms, conducting a detailed analysis of computational and memory complexity, along with performance metrics. The primary finding of our study is that by carefully selecting an algorithm for compressing different types of data before transmission, a significant amount of energy can be saved. Moreover, our investigation demonstrates that for a battery-powered embedded device transmitting sensor data based on the STM32F411CE microcontroller, the recommended transmission module is the nRF24L01+ board, as it requires the least amount of energy to transmit one byte of data. This module is most effective when combined with the LZ78 algorithm for optimal energy and time efficiency. In the case of image data, our findings indicate that the use of the JPEG algorithm for compression yields the best results. Overall, our research underscores the importance of selecting appropriate compression algorithms tailored to specific data types, contributing to enhanced energy efficiency in IoT devices.

A Low Complexity Rolling Bearing Diagnosis Technique Based on Machine Learning and Smart Preprocessing

In this work, we present a diagnosis system for rolling bearings that leverages simultaneous measurements of vibrations and machine rotation speed. Our approach combines the robustness of simple time domain methods for fault detection with the potential of machine learning techniques for fault location. This research is based on a neural network classifier, which exploits a simple and novel preprocessing algorithm specifically designed for minimizing the dependency of the classifier performance on the machine working conditions, on the bearing model and on the acquisition system set-up. The overall diagnosis system is based on light algorithms with reduced complexity and hardware resource demand and is designed to be deployed in embedded electronics. The fault diagnosis system was trained using emulated data, exploiting an ad-hoc test bench thus avoiding the problem of generating enough data, achieving an overall classifier accuracy larger than 98%. Its noteworthy ability to generalize was proven by using data emulating different working conditions and acquisition set-ups and noise levels, obtaining in all the cases accuracies greater than 97%, thereby proving in this way that the proposed system can be applied in a wide spectrum of different applications. Finally, real data from an on-line database containing vibration signals obtained in a completely different scenario are used to demonstrate the distinctive capability of the proposed system to generalize.

Development of electronic nose for detection of micro-mechanical damages in strawberries

A self-developed portable electronic nose and its classification model were designed to detect and differentiate minor mechanical damage to strawberries. The electronic nose utilises four metal oxide sensors and four electrochemical sensors specifically calibrated for strawberry detection. The selected strawberries were subjected to simulated damage using an H2Q-C air bath oscillator at varying speeds and then stored at 4°C to mimic real-life mechanical damage scenarios. Multiple feature extraction methods have been proposed and combined with Principal Component Analysis (PCA) dimensionality reduction for comparative modelling. Following validation with various models such as SVM, KNN, LDA, naive Bayes, and subspace ensemble, the Grid Search-optimised SVM (GS-SVM) method achieved the highest classification accuracy of 0.84 for assessing the degree of strawberry damage. Additionally, the Feature Extraction ensemble classifier achieved the highest classification accuracy (0.89 in determining the time interval of strawberry damage). This experiment demonstrated the feasibility of the self-developed electronic nose for detecting minor mechanical damage in strawberries.

An Automated Sitting Posture Recognition System Utilizing Pressure Sensors

Prolonged sitting with poor posture can lead to various health problems, including upper back pain, lower back pain, and cervical pain. Maintaining proper sitting posture is crucial for individuals while working or studying. Existing pressure sensor-based systems have been proposed to recognize sitting postures, but their accuracy ranges from 80% to 90%, leaving room for improvement. In this study, we developed a sitting posture recognition system called SPRS. We identified key areas on the chair surface that capture essential characteristics of sitting postures and employed diverse machine learning technologies to recognize ten common sitting postures. To evaluate the accuracy and usability of SPRS, we conducted a ten-minute sitting session with arbitrary postures involving 20 volunteers. The experimental results demonstrated that SPRS achieved an impressive accuracy rate of up to 99.1% in recognizing sitting postures. Additionally, we performed a usability survey using two standard questionnaires, the System Usability Scale (SUS) and the Questionnaire for User Interface Satisfaction (QUIS). The analysis of survey results indicated that SPRS is user-friendly, easy to use, and responsive.

Hardware-Software Partitioning for Real-Time Object Detection Using Dynamic Parameter Optimization

Computer vision algorithms implementations, especially for real-time applications, are present in a variety of devices that we are currently using (from smartphones or automotive applications to monitoring/security applications) and pose specific challenges, memory bandwidth or energy consumption (e.g., for mobility) being the most notable ones. This paper aims at providing a solution to improve the overall quality of real-time object detection computer vision algorithms using a hybrid hardware-software implementation. To this end, we explore the methods for a proper allocation of algorithm components towards hardware (as IP Cores) and the interfacing between hardware and software. Addressing specific design constraints, the relationship between the above components allows embedded artificial intelligence to select the operating hardware blocks (IP cores)-in the configuration phase-and to dynamically change the parameters of the aggregated hardware resources-in the instantiation phase, similar to the concretization of a class into a software object. The conclusions show the benefits of using hybrid hardware-software implementations, as well as major gains from using IP Cores, managed by artificial intelligence, for an object detection use-case, implemented on a FPGA demonstrator built around a Xilinx Zynq-7000 SoC Mini-ITX sub-system.

Contactless Camera-Based Heart Rate and Respiratory Rate Monitoring Using AI on Hardware

Detecting vital signs by using a contactless camera-based approach can provide several advantages over traditional clinical methods, such as lower financial costs, reduced visit times, increased comfort, and enhanced safety for healthcare professionals. Specifically, Eulerian Video Magnification (EVM) or Remote Photoplethysmography (rPPG) methods can be utilised to remotely estimate heart rate and respiratory rate biomarkers. In this paper two contactless camera-based health monitoring architectures are developed using EVM and rPPG, respectively; to this end, two different CNNs, (Mediapipe's BlazeFace and FaceMesh) are used to extract suitable regions of interest from incoming video frames. These two methods are implemented and deployed on four off-the-shelf edge devices as well as on a PC and evaluated in terms of latency (in each stage of the application's pipeline), throughput (FPS), power consumption (Watt), efficiency (throughput/Watt), and value (throughput/cost). This work provides important insights about the computational costs and bottlenecks of each method on each hardware platform, as well as which platform to use depending on the target metric. One of our insights shows that the Jetson Xavier NX platform is the best platform in terms of throughput and efficiency, while Raspberry Pi 4 8 GB is the best platform in terms of value.

Real-Time Evaluation of Time-Domain Pulse Rate Variability Parameters in Different Postures and Breathing Patterns Using Wireless Photoplethysmography Sensor: Towards Remote Healthcare in Low-Resource Communities

Photoplethysmography (PPG) signals have been widely used in evaluating cardiovascular biomarkers, however, there is a lack of in-depth understanding of the remote usage of this technology and its viability for underdeveloped countries. This study aims to quantitatively evaluate the performance of a low-cost wireless PPG device in detecting ultra-short-term time-domain pulse rate variability (PRV) parameters in different postures and breathing patterns. A total of 30 healthy subjects were recruited. ECG and PPG signals were simultaneously recorded in 3 min using miniaturized wearable sensors. Four heart rate variability (HRV) and PRV parameters were extracted from ECG and PPG signals, respectively, and compared using analysis of variance (ANOVA) or Scheirer-Ray-Hare test with post hoc analysis. In addition, the data loss was calculated as the percentage of missing sampling points. Posture did not present statistical differences across the PRV parameters but a statistical difference between indicators was found. Strong variation was found for the RMSSD indicator in the standing posture. The sitting position in both breathing patterns demonstrated the lowest data loss (1.0 ± 0.6 and 1.0 ± 0.7) and the lowest percentage of different factors for all indicators. The usage of commercial PPG and BLE devices can allow the reliable extraction of the PPG signal and PRV indicators in real time.

Microbiological Quality Estimation of Meat Using Deep CNNs on Embedded Hardware Systems

Spectroscopic sensor imaging of food samples meta-processed by deep machine learning models can be used to assess the quality of the sample. This article presents an architecture for estimating microbial populations in meat samples using multispectral imaging and deep convolutional neural networks. The deep learning models operate on embedded platforms and not offline on a separate computer or a cloud server. Different storage conditions of the meat samples were used, and various deep learning models and embedded platforms were evaluated. In addition, the hardware boards were evaluated in terms of latency, throughput, efficiency and value on different data pre-processing and imaging-type setups. The experimental results showed the advantage of the XavierNX platform in terms of latency and throughput and the advantage of Nano and RP4 in terms of efficiency and value, respectively.

Dynamic Image Difficulty-Aware DNN Pruning

Deep Neural Networks (DNNs) have achieved impressive performance in various image recognition tasks, but their large model sizes make them challenging to deploy on resource-constrained devices. In this paper, we propose a dynamic DNN pruning approach that takes into account the difficulty of the incoming images during inference. To evaluate the effectiveness of our method, we conducted experiments on the ImageNet dataset on several state-of-art DNNs. Our results show that the proposed approach reduces the model size and amount of DNN operations without the need to retrain or fine-tune the pruned model. Overall, our method provides a promising direction for designing efficient frameworks for lightweight DNN models that can adapt to the varying complexity of input images.

On-Line Evaluation and Monitoring of Security Features of an RO-Based PUF/TRNG for IoT Devices

The proliferation of devices for the Internet of Things (IoT) and their implication in many activities of our lives have led to a considerable increase in concern about the security of these devices, posing a double challenge for designers and developers of products. On the one hand, the design of new security primitives, suitable for resource-limited devices, can facilitate the inclusion of mechanisms and protocols to ensure the integrity and privacy of the data exchanged over the Internet. On the other hand, the development of techniques and tools to evaluate the quality of the proposed solutions as a step prior to their deployment, as well as to monitor their behavior once in operation against possible changes in operating conditions arising naturally or as a consequence of a stress situation forced by an attacker. To address these challenges, this paper first describes the design of a security primitive that plays an important role as a component of a hardware-based root of trust, as it can act as a source of entropy for True Random Number Generation (TRNG) or as a Physical Unclonable Function (PUF) to facilitate the generation of identifiers linked to the device on which it is implemented. The work also illustrates different software components that allow carrying out a self-assessment strategy to characterize and validate the performance of this primitive in its dual functionality, as well as to monitor possible changes in security levels that may occur during operation as a result of device aging and variations in power supply or operating temperature. The designed PUF/TRNG is provided as a configurable IP module, which takes advantage of the internal architecture of the Xilinx Series-7 and Zynq-7000 programmable devices and incorporates an AXI4-based standard interface to facilitate its interaction with soft- and hard-core processing systems. Several test systems that contain different instances of the IP have been implemented and subjected to an exhaustive set of on-line tests to obtain the metrics that determine its quality in terms of uniqueness, reliability, and entropy characteristics. The results obtained prove that the proposed module is a suitable candidate for various security applications. As an example, an implementation that uses less than 5% of the resources of a low-cost programmable device is capable of obfuscating and recovering 512-bit cryptographic keys with virtually zero error rate.

An Integrated Framework for Data Quality Fusion in Embedded Sensor Systems

The advancement of embedded sensor systems allowed the monitoring of complex processes based on connected devices. As more and more data are produced by these sensor systems, and as the data are used in increasingly vital areas of applications, it is of growing importance to also track the data quality of these systems. We propose a framework to fuse sensor data streams and associated data quality attributes into a single meaningful and interpretable value that represents the current underlying data quality. Based on the definition of data quality attributes and metrics to determine real-valued figures representing the quality of the attributes, the fusion algorithms are engineered. Methods based on maximum likelihood estimation (MLE) and fuzzy logic are used to perform data quality fusion by utilizing domain knowledge and sensor measurements. Two data sets are used to verify the proposed fusion framework. First, the methods are applied to a proprietary data set targeting sample rate inaccuracies of a micro-electro-mechanical system (MEMS) accelerometer and second, to the publicly available Intel Lab Data set. The algorithms are verified against their expected behavior based on data exploration and correlation analysis. We prove that both fusion approaches are capable of detecting data quality issues and providing an interpretable data quality indicator.

DEMIX: Domain-Enforced Memory Isolation for Embedded System

Memory isolation is an essential technology for safeguarding the resources of lightweight embedded systems. This technique isolates system resources by constraining the scope of the processor's accessible memory into distinct units known as domains. Despite the security offered by this approach, the Memory Protection Unit (MPU), the most common memory isolation method provided in most lightweight systems, incurs overheads during domain switching due to the privilege level intervention. However, as IoT environments become increasingly interconnected and more resources become required for protection, the significant overhead associated with domain switching under this constraint is expected to be crucial, making it harder to operate with more granular domains. To mitigate these issues, we propose DEMIX, which supports efficient memory isolation for multiple domains. DEMIX comprises two mainelements-Domain-Enforced Memory Isolation and instruction-level domain isolation-with the primary idea of enabling granular access control for memory by validating the domain state of the processor and the executed instructions. By achieving fine-grained validation of memory regions, our technique safely extends the supported domain capabilities of existing technologies while eliminating the overhead associated with switching between domains. Our implementation of eight user domains shows that our approach yields a hardware overhead of a slight 8% in Ibex Core, a very lightweight RISC-V processor.

Direction of Arrival Method for L-Shaped Array with RF Switch: An Embedded Implementation Perspective

This paper addresses the challenge of implementing Direction of Arrival (DOA) methods for indoor localization using Internet of Things (IoT) devices, particularly with the recent direction-finding capability of Bluetooth. DOA methods are complex numerical methods that require significant computational resources and can quickly deplete the batteries of small embedded systems typically found in IoT networks. To address this challenge, the paper presents a novel Unitary R-D Root MUSIC for L-shaped arrays that is tailor-made for such devices utilizing a switching protocol defined by Bluetooth. The solution exploits the radio communication system design to speed up execution, and its root-finding method circumvents complex arithmetic despite being used for complex polynomials. The paper carries out experiments on energy consumption, memory footprint, accuracy, and execution time in a commercial constrained embedded IoT device series without operating systems and software layers to prove the viability of the implemented solution. The results demonstrate that the solution achieves good accuracy and attains an execution time of a few milliseconds, making it a viable solution for DOA implementation in IoT devices.

Medical Devices with Embedded Sensor Systems: Design and Development Methodology for Start-Ups

Embedded systems have become a key technology for the evolution of medical devices. However, the regulatory requirements that must be met make designing and developing these devices challenging. As a result, many start-ups attempting to develop medical devices fail. Therefore, this article presents a methodology to design and develop embedded medical devices while minimising the economic investment during the technical risk stages and encouraging customer feedback. The proposed methodology is based on the execution of three stages: Development Feasibility, Incremental and Iterative Prototyping, and Medical Product Consolidation. All this is completed in compliance with the applicable regulations. The methodology mentioned above is validated through practical use cases in which the development of a wearable device for monitoring vital signs is the most relevant. The presented use cases sustain the proposed methodology, for the devices were successfully CE marked. Moreover, ISO 13485 certification is obtained by following the proposed procedures.

A Review of Embedded Machine Learning Based on Hardware, Application, and Sensing Scheme

Machine learning is an expanding field with an ever-increasing role in everyday life, with its utility in the industrial, agricultural, and medical sectors being undeniable. Recently, this utility has come in the form of machine learning implementation on embedded system devices. While there have been steady advances in the performance, memory, and power consumption of embedded devices, most machine learning algorithms still have a very high power consumption and computational demand, making the implementation of embedded machine learning somewhat difficult. However, different devices can be implemented for different applications based on their overall processing power and performance. This paper presents an overview of several different implementations of machine learning on embedded systems divided by their specific device, application, specific machine learning algorithm, and sensors. We will mainly focus on NVIDIA Jetson and Raspberry Pi devices with a few different less utilized embedded computers, as well as which of these devices were more commonly used for specific applications in different fields. We will also briefly analyze the specific ML models most commonly implemented on the devices and the specific sensors that were used to gather input from the field. All of the papers included in this review were selected using Google Scholar and published papers in the IEEExplore database. The selection criterion for these papers was the usage of embedded computing systems in either a theoretical study or practical implementation of machine learning models. The papers needed to have provided either one or, preferably, all of the following results in their studies-the overall accuracy of the models on the system, the overall power consumption of the embedded machine learning system, and the inference time of their models on the embedded system. Embedded machine learning is experiencing an explosion in both scale and scope, both due to advances in system performance and machine learning models, as well as greater affordability and accessibility of both. Improvements are noted in quality, power usage, and effectiveness.

Reducing the Energy Consumption of sEMG-Based Gesture Recognition at the Edge Using Transformers and Dynamic Inference

Hand gesture recognition applications based on surface electromiographic (sEMG) signals can benefit from on-device execution to achieve faster and more predictable response times and higher energy efficiency. However, deploying state-of-the-art deep learning (DL) models for this task on memory-constrained and battery-operated edge devices, such as wearables, requires a careful optimization process, both at design time, with an appropriate tuning of the DL models' architectures, and at execution time, where the execution of large and computationally complex models should be avoided unless strictly needed. In this work, we pursue both optimization targets, proposing a novel gesture recognition system that improves upon the state-of-the-art models both in terms of accuracy and efficiency. At the level of DL model architecture, we apply for the first time tiny transformer models (which we call bioformers) to sEMG-based gesture recognition. Through an extensive architecture exploration, we show that our most accurate bioformer achieves a higher classification accuracy on the popular Non-Invasive Adaptive hand Prosthetics Database 6 (Ninapro DB6) dataset compared to the state-of-the-art convolutional neural network (CNN) TEMPONet (+3.1%). When deployed on the RISC-V-based low-power system-on-chip (SoC) GAP8, bioformers that outperform TEMPONet in accuracy consume 7.8×-44.5× less energy per inference. At runtime, we propose a three-level dynamic inference approach that combines a shallow classifier, i.e., a random forest (RF) implementing a simple "rest detector" with two bioformers of different accuracy and complexity, which are sequentially applied to each new input, stopping the classification early for "easy" data. With this mechanism, we obtain a flexible inference system, capable of working in many different operating points in terms of accuracy and average energy consumption. On GAP8, we obtain a further 1.03×-1.35× energy reduction compared to static bioformers at iso-accuracy.

Multi-Core Time-Triggered OCBP-Based Scheduling for Mixed Criticality Periodic Task Systems

Mixed criticality systems are one of the relatively new directions of development for the classical real-time systems. As the real-time embedded systems become more and more complex, incorporating different tasks with different criticality levels, the continuous development of mixed criticality systems is only natural. These systems have practically entered every field where embedded systems are present: avionics, automotive, medical systems, wearable devices, home automation, industry and even the Internet of Things. While scheduling techniques have already been proposed in the literature for different types of mixed criticality systems, the number of papers addressing multiprocessor platforms running in a time-triggered mixed criticality environment is relatively low. These algorithms are easier to certify due to their complete determinism and isolation between components of different criticalities. Our research has centered on the problem of real-time scheduling on multiprocessor platforms for periodic tasks in a time-triggered mixed criticality environment. A partitioned, non-preemptive, table-driven scheduling algorithm was proposed, called Partitioned Time-Triggered Own Criticality Based Priority, based on a uniprocessor mixed criticality method. Furthermore, an analysis of the scheduling algorithm is provided in terms of success ratio by comparing it against an event-driven and a time-triggered method.

On-Device Intelligence for Malfunction Detection of Water Pump Equipment in Agricultural Premises: Feasibility and Experimentation

The digital transformation of agriculture is a promising necessity for tackling the increasing nutritional needs on Earth and the degradation of natural resources. Toward this direction, the availability of innovative electronic components and of the accompanying software programs can be exploited to detect malfunctions in typical agricultural equipment, such as water pumps, thereby preventing potential failures and water and economic losses. In this context, this article highlights the steps for adding intelligence to sensors installed on pumps in order to intercept and deliver malfunction alerts, based on cheap in situ microcontrollers, sensors, and radios and easy-to-use software tools. This involves efficient data gathering, neural network model training, generation, optimization, and execution procedures, which are further facilitated by the deployment of an experimental platform for generating diverse disturbances of the water pump operation. The best-performing variant of the malfunction detection model can achieve an accuracy rate of about 93% based on the vibration data. The system being implemented follows the on-device intelligence approach that decentralizes processing and networking tasks, thereby aiming to simplify the installation process and reduce the overall costs. In addition to highlighting the necessary implementation variants and details, a characteristic set of evaluation results is also presented, as well as directions for future exploitation.

ROMI: A Real-Time Optical Digit Recognition Embedded System for Monitoring Patients in Intensive Care Units

With advances in the Internet of Things, patients in intensive care units are constantly monitored to expedite emergencies. Due to the COVID-19 pandemic, non-face-to-face monitoring has been required for the safety of patients and medical staff. A control center monitors the vital signs of patients in ICUs. However, some medical devices, such as ventilators and infusion pumps, operate in a standalone fashion without communication capabilities, requiring medical staff to check them manually. One promising solution is to use a robotic system with a camera. We propose a real-time optical digit recognition embedded system called ROMI. ROMI is a mobile robot that monitors patients by recognizing digits displayed on LCD screens of medical devices in real time. ROMI consists of three main functions for recognizing digits: digit localization, digit classification, and digit annotation. We developed ROMI by using Matlab Simulink, and the maximum digit recognition performance was 0.989 mAP on alexnet. The developed system was deployed on NVIDIA GPU embedded platforms: Jetson Nano, Jetson Xavier NX, and Jetson AGX Xavier. We also created a benchmark by evaluating the runtime performance by considering ten pre-trained CNN models and three NVIDIA GPU platforms. We expect that ROMI will support medical staff with non-face-to-face monitoring in ICUs, enabling more effective and prompt patient care.

A comparative analysis of near-infrared image colorization methods for low-power NVIDIA Jetson embedded systems

The near-infrared (NIR) image obtained by an NIR camera is a grayscale image that is inconsistent with the human visual spectrum. It can be difficult to perceive the details of a scene from an NIR scene; thus, a method is required to convert them to visible images, providing color and texture information. In addition, a camera produces so much video data that it increases the pressure on the cloud server. Image processing can be done on an edge device, but the computing resources of edge devices are limited, and their power consumption constraints need to be considered. Graphics Processing Unit (GPU)-based NVIDIA Jetson embedded systems offer a considerable advantage over Central Processing Unit (CPU)-based embedded devices in inference speed. For this study, we designed an evaluation system that uses image quality, resource occupancy, and energy consumption metrics to verify the performance of different NIR image colorization methods on low-power NVIDIA Jetson embedded systems for practical applications. The performance of 11 image colorization methods on NIR image datasets was tested on three different configurations of NVIDIA Jetson boards. The experimental results indicate that the Pix2Pix method performs best, with a rate of 27 frames per second on the Jetson Xavier NX. This performance is sufficient to meet the requirements of real-time NIR image colorization.

Implementation of an Embedded System into the Internet of Robotic Things

The article describes the use of embedded systems in the Industrial Internet of Things and its benefits for industrial robots. For this purpose, the article presents a case study, which deals with an embedded system using an advanced microcontroller designed to be placed directly on the robot. The proposed system is being used to collect information about industrial robot parameters that impact its behavior and its long-term condition. The device measures the robot's surroundings parameters and its vibrations while working. Besides that, it also has an enormous potential to collect other parameters such as air pollution or humidity. The collected data are stored on the cloud platform and processed and analysed. The embedded system proposed in this article is conceived to be small and mobile, as it is a wireless system that can be easily applied to any industrial robot.

Embedded Sensor Systems in Medical Devices: Requisites and Challenges Ahead

The evolution of technology enables the design of smarter medical devices. Embedded Sensor Systems play an important role, both in monitoring and diagnostic devices for healthcare. The design and development of Embedded Sensor Systems for medical devices are subjected to standards and regulations that will depend on the intended use of the device as well as the used technology. This article summarizes the challenges to be faced when designing Embedded Sensor Systems for the medical sector. With this aim, it presents the innovation context of the sector, the stages of new medical device development, the technological components that make up an Embedded Sensor System and the regulatory framework that applies to it. Finally, this article highlights the need to define new medical product design and development methodologies that help companies to successfully introduce new technologies in medical devices.

Feasibility of a Real-Time Embedded Hyperspectral Compressive Sensing Imaging System

Hyperspectral imaging has been attracting considerable interest as it provides spectrally rich acquisitions useful in several applications, such as remote sensing, agriculture, astronomy, geology and medicine. Hyperspectral devices based on compressive acquisitions have appeared recently as an alternative to conventional hyperspectral imaging systems and allow for data-sampling with fewer acquisitions than classical imaging techniques, even under the Nyquist rate. However, compressive hyperspectral imaging requires a reconstruction algorithm in order to recover all the data from the raw compressed acquisition. The reconstruction process is one of the limiting factors for the spread of these devices, as it is generally time-consuming and comes with a high computational burden. Algorithmic and material acceleration with embedded and parallel architectures (e.g., GPUs and FPGAs) can considerably speed up image reconstruction, making hyperspectral compressive systems suitable for real-time applications. This paper provides an in-depth analysis of the required performance in terms of computing power, data memory and bandwidth considering a compressive hyperspectral imaging system and a state-of-the-art reconstruction algorithm as an example. The results of the analysis show that real-time application is possible by combining several approaches, namely, exploitation of system matrix sparsity and bandwidth reduction by appropriately tuning data value encoding.

Velocity Prediction of a Pipeline Inspection Gauge (PIG) with Machine Learning

A device known as a pipeline inspection gauge (PIG) runs through oil and gas pipelines which performs various maintenance operations in the oil and gas industry. The PIG velocity, which plays a role in the efficiency of these operations, is usually determined indirectly from odometers installed in it. Although this is a relatively simple technique, the loss of contact between the odometer wheel and the pipeline results in measurement errors. To help reduce these errors, this investigation employed neural networks to estimate the speed of a prototype PIG, using the pressure difference that acts on the device inside the pipeline and its acceleration instead of using odometers. Static networks (e.g., multilayer perceptron) and recurrent networks (e.g., long short-term memory) were built, and in addition, a prototype PIG was developed with an embedded system based on Raspberry Pi 3 to collect speed, acceleration and pressure data for the model training. The implementation of the supervised neural networks used the Python library TensorFlow package. To train and evaluate the models, we used the PIG testing pipeline facilities available at the Petroleum Evaluation and Measurement Laboratory of the Federal University of Rio Grande do Norte (LAMP/UFRN). The results showed that the models were able to learn the relationship among the differential pressure, acceleration and speed of the PIG. The proposed approach can complement odometer-based systems, increasing the reliability of speed measurements.

A Comprehensive Survey on the Non-Invasive Passive Side-Channel Analysis

Side-channel analysis has become a widely recognized threat to the security of cryptographic implementations. Different side-channel attacks, as well as countermeasures, have been proposed in the literature. Such attacks pose a severe threat to both hardware and software cryptographic implementations, especially in the IoT environment where the attacker may easily gain physical access to a device, leaving it vulnerable to tampering. In this paper, we provide a comprehensive survey regarding the non-invasive passive side-channel analysis. We describe both non-profiled and profiled attacks, related security metrics, countermeasures against such attacks, and leakage-assessment methodologies, as available in the literature of more than twenty years of research.

Stateful Rotor for Continuity of Quaternion and Fast Sensor Fusion Algorithm Using 9-Axis Sensors

Advances in micro-electro-mechanical systems technology have led to the emergence of compact attitude measurement sensor products that integrate acceleration, magnetometer, and gyroscope sensors on a single chip, making them important devices in the field of three-dimensional (3D) attitude measurement for unmanned aerial vehicles, smartphones, and other devices. Sensor fusion algorithms for posture measurement have become an indispensable technology in cutting-edge research, such as human posture measurement using wearable sensors, and stabilization problems in robot position and posture measurement. We have also developed wearable sensors and powered suits in our previous research. We needed a technology for the real-time measurement of a 3D human body motion. It is known that quaternions can be used to algebraically handle 3D rotations; however, sensor fusion algorithms for three sensors are presently complex. This is because these algorithms deal with the post-rotation attitude (pure quaternions) rather than rotation information (the rotor) to avoid a double covering problem involving the rotor. If we are dealing with rotation, it may be possible to make the algorithm simpler and faster by dealing directly with the rotor. In this study, to solve the double covering problem involving the rotor, we propose a stateful rotor and develop a technique for uniquely determining the time-varying states of the rotor. The proposed stateful rotor guarantees the continuity of the rotor parameters with respect to angular changes, and this paper confirms its effectiveness by simulating two rotations around an arbitrary axis. In addition, we verify experimentally that a fast sensor fusion method using stateful rotor can be used for attitude calculation. Experiments also confirm that the calculated results converge to the desired rotation angle for two spatial rotations around an arbitrary axis. Since the proposed stateful rotor extends and stabilizes the definition of the rotor, it is applicable to any algorithm that deals with time-varying quaternionic rotors. In this research, an algorithm based on a multiply-add operation is designed to reduce computational complexity as a high-speed calculation for embedded systems. This method is theoretically equivalent to other methods, while contributing to power saving and the cost reduction of products.

A CNN-based image detector for plant leaf diseases classification

Identifying diseases from images of plant leaves is one of the most important research areas in precision agriculture. The aim of this paper is to propose an image detector embedding a resource constrained convolutional neural network (CNN) implemented in a low cost, low power platform, named OpenMV Cam H7 Plus, to perform a real-time classification of plant disease. The CNN network so obtained has been trained on two specific datasets for plant diseases detection, the ESCA-dataset and the PlantVillage-augmented dataset, and implemented in a low-power, low-cost Python programmable machine vision camera for real-time image acquisition and classification, equipped with a LCD display showing to the user the classification response in real-time. Experimental results show that this CNN-based image detector can be effectively implemented on the chosen constrained-resource system, achieving an accuracy of about 98.10%/95.24% with a very low memory cost (718.961 KB/735.727 KB) and inference time (122.969 ms/125.630 ms) tested on board for the ESCA and the PlantVillage-augmented datasets respectively, allowing the design of a portable embedded system for plant leaf diseases classification. Source files are available at https://doi.org/10.17605/OSF.IO/UCM8D.

A Low-Cost Multi-Purpose IoT Sensor for Biologging and Soundscape Activities

The rapid expansion in miniaturization, usability, energy efficiency, and affordability of Internet of Things (IoT) sensors, integrated with innovations in smart capability, is greatly increasing opportunities in ground-level monitoring of ecosystems at a specific scale using sensor grids. Surrounding sound is a powerful data source for investigating urban and non-urban ecosystem health, and researchers commonly use robust but expensive passive sensors as monitoring equipment to capture it. This paper comprehensively describes the hardware behind our low-cost, small multipurpose prototype, capable of monitoring different environments (e.g., remote locations) with onboard processing power. The device consists of a printed circuit board, microprocessor, local memory, environmental sensor, microphones, optical sensors and LoRa (Long Range) communication systems. The device was successfully used in different use cases, from monitoring mosquitoes enhanced with optical sensors to ocean activities using a hydrophone.

Composing Optimized Embedded Software Architectures for Physics-Based EKF-MPC Smart Sensor for Li-Ion Battery Cell Management

Efficient battery technology is imperative for the adoption of clean energy automotive solutions. In addition, efficient battery technology extends the useful life of the battery as well as provides improved performance to fossil fuel technology. Model predictive control (MPC) is an effective way to operate battery management systems (BMS) at their maximum capability, while maintaining the safety requirements. Using the physics-based model (PBM) of the battery allows the control system to operate on the chemical and physical process of the battery. Since these processes are internal to the battery and are physically unobservable, the extended Kalman filter (EKF) serves as a virtual observer that can monitor the physical and chemical properties that are otherwise unobservable. These three methods (i.e., PBM, EKF, and MPC) together can prolong the useful life of the battery, especially for Li-ion batteries. This capability is not limited to the automotive industry: any real-world smart application can benefit from a portable/mobile efficient BMS, compelling these systems to be executed on resource-constrained embedded devices. Furthermore, the intrinsic adaptive control process of the PBM is uniquely suited for smart systems and smart technology. However, the sheer computational complexity of PBM for MPC and EKF prevents it from being realized on highly constrained embedded devices. In this research work, we introduce a novel, unique, and efficient embedded software architecture for a PB-EKF-MPC smart sensor for BMS, specifically on embedded devices, by addressing the computational complexity of PBM. Our proposed embedded software architecture is created in such a way to be executed on a 32-bit embedded microprocessor running at 100 MHz with a limited memory of 128 KB, and still obtains an average execution time of 4.8 ms.

RESEKRA: Remote Enrollment Using SEaled Keys for Remote Attestation

This paper presents and implements a novel remote attestation method to ensure the integrity of a device applicable to decentralized infrastructures, such as those found in common edge computing scenarios. Edge computing can be considered as a framework where multiple unsupervised devices communicate with each other with lack of hierarchy, requesting and offering services without a central server to orchestrate them. Because of these characteristics, there are many security threats, and detecting attacks is essential. Many remote attestation systems have been developed to alleviate this problem, but none of them can satisfy the requirements of edge computing: accepting dynamic enrollment and removal of devices to the system, respecting the interrupted activity of devices, and last but not least, providing a decentralized architecture for not trusting in just one Verifier. This security flaw has a negative impact on the development and implementation of edge computing-based technologies because of the impossibility of secure implementation. In this work, we propose a remote attestation system that, through using a Trusted Platform Module (TPM), enables the dynamic enrollment and an efficient and decentralized attestation. We demonstrate and evaluate our work in two use cases, attaining acceptance of intermittent activity by IoT devices, deletion of the dependency of centralized verifiers, and the probation of continuous integrity between unknown devices just by one signature verification.

Validation of Embedded State Estimator Modules for Decentralized Monitoring of Power Distribution Systems Using IoT Components

Recent theoretical studies demonstrate the advantages of using decentralized architectures over traditional centralized architectures for real-time Power Distribution Systems (PDSs) operation. These advantages include the reduction of the amount of data to be transmitted and processed when performing state estimation in PDSs. The main contribution of this paper is to provide lab validation of the advantages and feasibility of decentralized monitoring of PDSs. Therefore, this paper presents an advanced trial emulating realistic conditions and hardware setup. More specifically, the paper proposes: (i) The laboratory development and implementation of an Advanced Measurement Infrastructure (AMI) prototype to enable the simulation of a smart grid. To emulate the information traffic between smart meters and distribution operation centers, communication modules, that enable the use of wireless networks for sending messages in real-time, are used, bridging concepts from both IoT and Edge Computing. (ii) The laboratory development and implementation of a decentralized architecture based on Embedded State Estimator Modules (ESEMs) are carried out. ESEMs manage information from smart meters at lower voltage networks, performing real-time state estimation in PDSs. Simulations performed on a real PDS with 208 buses (considering both medium and low voltage buses) have met the aims of this paper. The results show that by using ESEMs in a decentralized architecture, both the data transit through the communication network, as well as the computational requirements involved in monitoring PDSs in real-time, are reduced considerably without any loss of accuracy.

An FPGA-Embedded Brain-Computer Interface System to Support Individual Autonomy in Locked-In Individuals

Brain-computer interfaces (BCI) can detect specific EEG patterns and translate them into control signals for external devices by providing people suffering from severe motor disabilities with an alternative/additional channel to communicate and interact with the outer world. Many EEG-based BCIs rely on the P300 event-related potentials, mainly because they require training times for the user relatively short and provide higher selection speed. This paper proposes a P300-based portable embedded BCI system realized through an embedded hardware platform based on FPGA (field-programmable gate array), ensuring flexibility, reliability, and high-performance features. The system acquires EEG data during user visual stimulation and processes them in a real-time way to correctly detect and recognize the EEG features. The BCI system is designed to allow to user to perform communication and domotic controls.

Gesture-Based Human Machine Interaction Using RCNNs in Limited Computation Power Devices

The use of gestures is one of the main forms of human machine interaction (HMI) in many fields, from advanced robotics industrial setups, to multimedia devices at home. Almost every gesture detection system uses computer vision as the fundamental technology, with the already well-known problems of image processing: changes in lighting conditions, partial occlusions, variations in color, among others. To solve all these potential issues, deep learning techniques have been proven to be very effective. This research proposes a hand gesture recognition system based on convolutional neural networks and color images that is robust against environmental variations, has a real time performance in embedded systems, and solves the principal problems presented in the previous paragraph. A new CNN network has been specifically designed with a small architecture in terms of number of layers and total number of neurons to be used in computationally limited devices. The obtained results achieve a percentage of success of 96.92% on average, a better score than those obtained by previous algorithms discussed in the state of the art.

Correlating Time Series Signals and Event Logs in Embedded Systems

In many embedded systems, we face the problem of correlating signals characterising device operation (e.g., performance parameters, anomalies) with events describing internal device activities. This leads to the investigation of two types of data: time series, representing signal periodic samples in a background of noise, and sporadic event logs. The correlation process must take into account clock inconsistencies between the data acquisition and monitored devices, which provide time series signals and event logs, respectively. The idea of the presented solution is to classify event logs based on the introduced similarity metric and deriving their distribution in time. The identified event log sequences are matched with time intervals corresponding to specified sample patterns (objects) in the registered signal time series. The matching (correlation) process involves iterative time offset adjustment. The paper presents original algorithms to investigate correlation problems using the object-oriented data models corresponding to two monitoring sources. The effectiveness of this approach has been verified in power consumption analysis using real data collected from the developed Holter device. It is quite universal and can be easily adapted to other device optimisation problems.

A Review of Human Circulatory System Simulation: Bridging the Gap between Engineering and Medicine

(1) Background: Simulation-based training (SBT) is the practice of using hands-on training to immerse learners in a risk-free and high-fidelity environment. SBT is used in various fields due to its risk-free benefits from a safety and an economic perspective. In addition, SBT provides immersive training unmatched by traditional teaching the interactive visualization needed in particular scenarios. Medical SBT is a prevalent practice as it allows for a platform for learners to learn in a risk-free and cost-effective environment, especially in critical care, as mistakes could easily cause fatalities. An essential category of care is human circulatory system care (HCSC), which includes essential-to-simulate complications such as cardiac arrest. (2) Methods: In this paper, a deeper look onto existing human circulatory system medical SBT is presented to assess and highlight the important features that should be present with a focus on extracorporeal membrane oxygenation cannulation (ECMO) simulators and cardiac catheterization. (3) Results: A list of features is also suggested for an ideal simulator to bridge the gap between medical studies and simulator engineering, followed by a case study of an ECMO SBT system design. (4) Conclusions: a collection and discussion of existing work for HCSC SBT are portrayed as a guide for researchers and practitioners to compare existing SBT and recreating them effectively.

Wearable Edge AI Applications for Ecological Environments

Ecological environments research helps to assess the impacts on forests and managing forests. The usage of novel software and hardware technologies enforces the solution of tasks related to this problem. In addition, the lack of connectivity for large data throughput raises the demand for edge-computing-based solutions towards this goal. Therefore, in this work, we evaluate the opportunity of using a Wearable edge AI concept in a forest environment. For this matter, we propose a new approach to the hardware/software co-design process. We also address the possibility of creating wearable edge AI, where the wireless personal and body area networks are platforms for building applications using edge AI. Finally, we evaluate a case study to test the possibility of performing an edge AI task in a wearable-based environment. Thus, in this work, we evaluate the system to achieve the desired task, the hardware resource and performance, and the network latency associated with each part of the process. Through this work, we validated both the design pattern review and case study. In the case study, the developed algorithms could classify diseased leaves with a circa 90% accuracy with the proposed technique in the field. This results can be reviewed in the laboratory with more modern models that reached up to 96% global accuracy. The system could also perform the desired tasks with a quality factor of 0.95, considering the usage of three devices. Finally, it detected a disease epicenter with an offset of circa 0.5 m in a 6 m × 6 m × 12 m space. These results enforce the usage of the proposed methods in the targeted environment and the proposed changes in the co-design pattern.

Emotion Recognition on Edge Devices: Training and Deployment

Emotion recognition, among other natural language processing tasks, has greatly benefited from the use of large transformer models. Deploying these models on resource-constrained devices, however, is a major challenge due to their computational cost. In this paper, we show that the combination of large transformers, as high-quality feature extractors, and simple hardware-friendly classifiers based on linear separators can achieve competitive performance while allowing real-time inference and fast training. Various solutions including batch and Online Sequential Learning are analyzed. Additionally, our experiments show that latency and performance can be further improved via dimensionality reduction and pre-training, respectively. The resulting system is implemented on two types of edge device, namely an edge accelerator and two smartphones.

Addressing the Data Acquisition Paradigm in the Early Detection of Pediatric Foot Deformities

The analysis of plantar pressure through podometry has allowed analyzing and detecting different types of disorders and treatments in child patients. Early detection of an inadequate distribution of the patient's weight can prevent serious injuries to the knees and lower spine. In this paper, an embedded system capable of detecting the presence of normal, flat, or arched footprints using resistive pressure sensors was proposed. For this purpose, both hardware- and software-related criteria were studied for an improved data acquisition through signal coupling and filtering processes. Subsequently, learning algorithms allowed us to estimate the type of footprint biomechanics in preschool and school children volunteers. As a result, the proposed algorithm achieved an overall classification accuracy of 97.2%. A flat feet share of 60% was encountered in a sample of 1000 preschool children. Similarly, flat feet were observed in 52% of a sample of 600 school children.

Towards a Blockchain-Based Multi-UAV Surveillance System

This study describes a blockchain-based multi-unmanned aerial vehicle (multi-UAV) surveillance framework that enables UAV coordination and financial exchange between system users. The objective of the system is to allow a set of Points-Of-Interest (POI) to be surveyed by a set of autonomous UAVs that cooperate to minimize the time between successive visits while exhibiting unpredictable behavior to prevent external agents from learning their movements. The system can be seen as a marketplace where the UAVs are the service providers and the POIs are the service seekers. This concept is based on a blockchain embedded on the UAVs and on some nodes on the ground, which has two main functionalities. The first one is to plan the route of each UAV through an efficient and computationally cheap game-theoretic decision algorithm implemented into a smart contract. The second one is to allow financial transactions between the system and its users, where the POIs subscribe to surveillance services by buying tokens. Conversely, the system pays the UAVs in tokens for the provided services. The first benchmarking experiments show that the IOTA blockchain is a potential blockchain candidate to be integrated in the UAV embedded system and that the chosen decentralized decision-making coordination strategy is efficient enough to fill the mission requirements while being computationally light.

An Open-Source Scale Model Platform for Teaching Autonomous Vehicle Technologies

Emerging technologies in the context of Autonomous Vehicles (AV) have drastically evolved the industry’s qualification requirements. AVs incorporate complex perception and control systems. Teaching the associated skills that are necessary for the analysis of such systems becomes a very difficult process and existing solutions do not facilitate learning. In this study, our efforts are devoted to proposingan open-source scale model vehicle platform that is designed for teaching the fundamental concepts of autonomous vehicles technologies that are adapted to undergraduate and technical students. The proposed platform is as realistic as possible in order to present and address all of the fundamental concepts that are associated with AV. It includes all on-board components of a stand-alone system, including low and high level functions. Such functionalities are detailed and a proof of concept prototype is presented. A set of experiments is carried out, and the results obtained using this prototype validate the usability of the model for the analysis of time- and energy-constrained systems, as well as distributed embedded perception systems.

Smart Cities of the Future as Cyber Physical Systems: Challenges and Enabling Technologies

A smart city represents an improvement of today's cities, both functionally and structurally, that strategically utilizes several smart factors, capitalizing on Information and Communications Technology (ICT) to increase the city's sustainable growth and strengthen the city's functions, while ensuring the citizens' enhanced quality of life and health. Cities can be viewed as a microcosm of interconnected "objects" with which citizens interact daily, which represents an extremely interesting example of a cyber physical system (CPS), where the continuous monitoring of a city's status occurs through sensors and processors applied within the real-world infrastructure. Each object in a city can be both the collector and distributor of information regarding mobility, energy consumption, air pollution as well as potentially offering cultural and tourist information. As a consequence, the cyber and real worlds are strongly linked and interdependent in a smart city. New services can be deployed when needed, and evaluation mechanisms can be set up to assess the health and success of a smart city. In particular, the objectives of creating ICT-enabled smart city environments target (but are not limited to) improved city services; optimized decision-making; the creation of smart urban infrastructures; the orchestration of cyber and physical resources; addressing challenging urban issues, such as environmental pollution, transportation management, energy usage and public health; the optimization of the use and benefits of next generation (5G and beyond) communication; the capitalization of social networks and their analysis; support for tactile internet applications; and the inspiration of urban citizens to improve their quality of life. However, the large scale deployment of cyber-physical-social systems faces a series of challenges and issues (e.g., energy efficiency requirements, architecture, protocol stack design, implementation, and security), which requires more smart sensing and computing methods as well as advanced networking and communications technologies to provide more pervasive cyber-physical-social services. In this paper, we discuss the challenges, the state-of-the-art, and the solutions to a set of currently unresolved key questions related to CPSs and smart cities.

Smart Video Surveillance System Based on Edge Computing

New processing methods based on artificial intelligence (AI) and deep learning are replacing traditional computer vision algorithms. The more advanced systems can process huge amounts of data in large computing facilities. In contrast, this paper presents a smart video surveillance system executing AI algorithms in low power consumption embedded devices. The computer vision algorithm, typical for surveillance applications, aims to detect, count and track people’s movements in the area. This application requires a distributed smart camera system. The proposed AI application allows detecting people in the surveillance area using a MobileNet-SSD architecture. In addition, using a robust Kalman filter bank, the algorithm can keep track of people in the video also providing people counting information. The detection results are excellent considering the constraints imposed on the process. The selected architecture for the edge node is based on a UpSquared2 device that includes a vision processor unit (VPU) capable of accelerating the AI CNN inference. The results section provides information about the image processing time when multiple video cameras are connected to the same edge node, people detection precision and recall curves, and the energy consumption of the system. The discussion of results shows the usefulness of deploying this smart camera node throughout a distributed surveillance system.

Quantization and Deployment of Deep Neural Networks on Microcontrollers

Embedding Artificial Intelligence onto low-power devices is a challenging task that has been partly overcome with recent advances in machine learning and hardware design. Presently, deep neural networks can be deployed on embedded targets to perform different tasks such as speech recognition, object detection or Human Activity Recognition. However, there is still room for optimization of deep neural networks onto embedded devices. These optimizations mainly address power consumption, memory and real-time constraints, but also an easier deployment at the edge. Moreover, there is still a need for a better understanding of what can be achieved for different use cases. This work focuses on quantization and deployment of deep neural networks onto low-power 32-bit microcontrollers. The quantization methods, relevant in the context of an embedded execution onto a microcontroller, are first outlined. Then, a new framework for end-to-end deep neural networks training, quantization and deployment is presented. This framework, called MicroAI, is designed as an alternative to existing inference engines (TensorFlow Lite for Microcontrollers and STM32Cube.AI). Our framework can indeed be easily adjusted and/or extended for specific use cases. Execution using single precision 32-bit floating-point as well as fixed-point on 8- and 16 bits integers are supported. The proposed quantization method is evaluated with three different datasets (UCI-HAR, Spoken MNIST and GTSRB). Finally, a comparison study between MicroAI and both existing embedded inference engines is provided in terms of memory and power efficiency. On-device evaluation is done using ARM Cortex-M4F-based microcontrollers (Ambiq Apollo3 and STM32L452RE).

RcdMathLib: An Open Source Software Library for Computing on Resource-Limited Devices

We developped an open source library called RcdMathLib for solving multivariate linear and nonlinear systems. RcdMathLib supports on-the-fly computing on low-cost and resource-constrained devices, e.g., microcontrollers. The decentralized processing is a step towards ubiquitous computing enabling the implementation of Internet of Things (IoT) applications. RcdMathLib is modular- and layer-based, whereby different modules allow for algebraic operations such as vector and matrix operations or decompositions. RcdMathLib also comprises a utilities-module providing sorting and filtering algorithms as well as methods generating random variables. It enables solving linear and nonlinear equations based on efficient decomposition approaches such as the Singular Value Decomposition (SVD) algorithm. The open source library also provides optimization methods such as Gauss–Newton and Levenberg–Marquardt algorithms for solving problems of regression smoothing and curve fitting. Furthermore, a positioning module permits computing positions of IoT devices using algorithms for instance trilateration. This module also enables the optimization of the position by performing a method to reduce multipath errors on the mobile device. The library is implemented and tested on resource-limited IoT as well as on full-fledged operating systems. The open source software library is hosted on a GitLab repository.

Knowledge-Based Verification of Concatenative Programming Patterns Inspired by Natural Language for Resource-Constrained Embedded Devices

We propose a methodology to verify applications developed following programming patterns inspired by natural language that interact with physical environments and run on resource-constrained interconnected devices. Natural language patterns allow for the reduction of intermediate abstraction layers to map physical domain concepts into executable code avoiding the recourse to ontologies, which would need to be shared, kept up to date, and synchronized across a set of devices. Moreover, the computational paradigm we use for effective distributed execution of symbolic code on resource-constrained devices encourages the adoption of such patterns. The methodology is supported by a rule-based system that permits runtime verification of Software Under Test (SUT) on board the target devices through automated oracle and test case generation. Moreover, verification extends from syntactic and semantic checks to the evaluation of the effects of SUT execution on target hardware. Additionally, by exploiting rules tying sensors and actuators to physical quantities, the effects of code execution on the physical environment can be verified. The system is also able to build test code to highlight software issues that may arise during repeated SUT execution on the target hardware.

Survey on Deep Neural Networks in Speech and Vision Systems

This survey presents a review of state-of-the-art deep neural network architectures, algorithms, and systems in vision and speech applications. Recent advances in deep artificial neural network algorithms and architectures have spurred rapid innovation and development of intelligent vision and speech systems. With availability of vast amounts of sensor data and cloud computing for processing and training of deep neural networks, and with increased sophistication in mobile and embedded technology, the next-generation intelligent systems are poised to revolutionize personal and commercial computing. This survey begins by providing background and evolution of some of the most successful deep learning models for intelligent vision and speech systems to date. An overview of large-scale industrial research and development efforts is provided to emphasize future trends and prospects of intelligent vision and speech systems. Robust and efficient intelligent systems demand low-latency and high fidelity in resource-constrained hardware platforms such as mobile devices, robots, and automobiles. Therefore, this survey also provides a summary of key challenges and recent successes in running deep neural networks on hardware-restricted platforms, i.e. within limited memory, battery life, and processing capabilities. Finally, emerging applications of vision and speech across disciplines such as affective computing, intelligent transportation, and precision medicine are discussed. To our knowledge, this paper provides one of the most comprehensive surveys on the latest developments in intelligent vision and speech applications from the perspectives of both software and hardware systems. Many of these emerging technologies using deep neural networks show tremendous promise to revolutionize research and development for future vision and speech systems.

Deep Learning-Based Bearing Fault Diagnosis Method for Embedded Systems

Bearing elements are vital in induction motors; therefore, early fault detection of rolling-element bearings is essential in machine health monitoring. With the advantage of fault feature representation techniques of time–frequency domain for nonstationary signals and the advent of convolutional neural networks (CNNs), bearing fault diagnosis has achieved high accuracy, even at variable rotational speeds. However, the required computation and memory resources of CNN-based fault diagnosis methods render it difficult to be compatible with embedded systems, which are essential in real industrial platforms because of their portability and low costs. This paper proposes a novel approach for establishing a CNN-based process for bearing fault diagnosis on embedded devices using acoustic emission signals, which reduces the computation costs significantly in classifying the bearing faults. A light state-of-the-art CNN model, MobileNet-v2, is established via pruning to optimize the required system resources. The input image size, which significantly affects the consumption of system resources, is decreased by our proposed signal representation method based on the constant-Q nonstationary Gabor transform and signal decomposition adopting ensemble empirical mode decomposition with a CNN-based method for selecting intrinsic mode functions. According to our experimental results, our proposed method can provide the accuracy for bearing faults classification by up to 99.58% with less computation overhead compared to previous deep learning-based fault diagnosis methods.

Embedded Model Predictive Control for a Wearable Artificial Pancreas

While artificial pancreas (AP) systems are expected to improve the quality of life among people with type 1 diabetes mellitus (T1DM), the design of convenient systems that optimize the user experience, especially for those with active lifestyles, such as children and adolescents, still remains an open research question. In this work, we introduce an embeddable design and implementation of model predictive control (MPC) of AP systems for people with T1DM that significantly reduces the weight and on-body footprint of the AP system. The embeddable controller is based on a zone MPC that has been evaluated in multiple clinical studies. The proposed embedded zone MPC features a simpler design of the periodic safe zone in the cost function and the utilization of state-of-the-art alternating minimization algorithms for solving the convex programming problems inherent to MPC with linear models subject to convex constraints. Off-line closed-loop data generated by the FDA-accepted UVA/Padova simulator is used to select an optimization algorithm and corresponding tuning parameters. Through hardware-in-the-loop in silico results on a limited-resource Arduino Zero (Feather M0) platform, we demonstrate the potential of the proposed embedded MPC. In spite of resource limitations, our embedded zone MPC manages to achieve comparable performance of that of the full-version zone MPC implemented in a 64-bit desktop for scenarios with/without meal-disturbance compensations. Metrics for performance comparison included median percent time in the euglycemic ([70, 180] mg/dL range) of 84.3% vs. 83.1% for announced meals, with an equivalence test yielding p = 0.0013 and 66.2% vs. 66.0% for unannounced meals with p = 0.0028.