An adaptive embedding procedure for time series forecasting with deep neural networks

Nowadays, solving time series prediction problems is an open and challenging task. Many solutions are based on the implementation of deep neural architectures, which are able to analyze the structure of the time series and to carry out the prediction. In this work, we present a novel deep learning scheme based on an adaptive embedding mechanism. The latter is exploited to extract a compressed representation of the input time series that is used for the subsequent forecasting. The proposed model is based on a two-layer bidirectional Long Short-Term Memory network, where the first layer performs the adaptive embedding and the second layer acts as a predictor. The performances of the proposed forecasting scheme are compared with several models in two different scenarios, considering both well-known time series and real-life application cases. The experimental results show the accuracy and the flexibility of the proposed approach, which can be used as a prediction tool for any actual application.

A Class-Imbalance Aware and Explainable Spatio-Temporal Graph Attention Network for Neonatal Seizure Detection

Seizures are the most prevalent clinical indication of neurological disorders in neonates. In this study, a class-imbalance aware and explainable deep learning approach based on Convolutional Neural Networks (CNNs) and Graph Attention Networks (GATs) is proposed for the accurate automated detection of neonatal seizures. The proposed model integrates the temporal information of EEG signals with the spatial information on the EEG channels through the graph representation of the multi-channel EEG segments. One-dimensional CNNs are used to automatically develop a feature set that accurately represents the differences between seizure and nonseizure epochs in the time domain. By employing GAT, the attention mechanism is utilized to emphasize the critical channel pairs and information flow among brain regions. GAT coefficients were then used to empirically visualize the important regions during the seizure and nonseizure epochs, which can provide valuable insight into the location of seizures in the neonatal brain. Additionally, to tackle the severe class imbalance in the neonatal seizure dataset using under-sampling and focal loss techniques are used. Overall, the final Spatio-Temporal Graph Attention Network (ST-GAT) outperformed previous benchmarked methods with a mean AUC of 96.6% and Kappa of 0.88, demonstrating its high accuracy and potential for clinical applications.

Fast Seismic Assessment of Built Urban Areas with the Accuracy of Mechanical Methods Using a Feedforward Neural Network

Capacity curves obtained from nonlinear static analyses are widely used to perform seismic assessments of structures as an alternative to dynamic analysis. This paper presents a novel ‘en masse’ method to assess the seismic vulnerability of urban areas swiftly and with the accuracy of mechanical methods. At the core of this methodology is the calculation of the capacity curves of low-rise reinforced concrete buildings using neural networks, where no modeling of the building is required. The curves are predicted with minimal error, needing only basic geometric and material parameters of the structures to be specified. As a first implementation, a typology of prismatic buildings is defined and a training set of more than 7000 structures generated. The capacity curves are calculated through push-over analysis using SAP2000. The results feature the prediction of 100-point curves in a single run of the network while maintaining a very low mean absolute error. This paper proposes a method that improves current seismic assessment tools by providing a fast and accurate calculation of the vulnerability of large sets of buildings in urban environments.

Transformer-based Multi-target Regression on Electronic Health Records for Primordial Prevention of Cardiovascular Disease

Machine learning algorithms have been widely used to capture the static and temporal patterns within electronic health records (EHRs). While many studies focus on the (primary) prevention of diseases, primordial prevention (preventing the factors that are known to increase the risk of a disease occurring) is still widely under-investigated. In this study, we propose a multi-target regression model leveraging transformers to learn the bidirectional representations of EHR data and predict the future values of 11 major modifiable risk factors of cardiovascular disease (CVD). Inspired by the proven results of pre-training in natural language processing studies, we apply the same principles on EHR data, dividing the training of our model into two phases: pre-training and fine-tuning. We use the fine-tuned transformer model in a "multi-target regression" theme. Following this theme, we combine the 11 disjoint prediction tasks by adding shared and target-specific layers to the model and jointly train the entire model. We evaluate the performance of our proposed method on a large publicly available EHR dataset. Through various experiments, we demonstrate that the proposed method obtains a significant improvement (12.6% MAE on average across all 11 different outputs) over the baselines.

VPNET: Variable Projection Networks

In this paper, we introduce VPNet, a novel model-driven neural network architecture based on variable projection (VP). Applying VP operators to neural networks results in learnable features, interpretable parameters, and compact network structures. This paper discusses the motivation and mathematical background of VPNet and presents experiments. The VPNet approach was evaluated in the context of signal processing, where we classified a synthetic dataset and real electrocardiogram (ECG) signals. Compared to fully connected and one-dimensional convolutional networks, VPNet offers fast learning ability and good accuracy at a low computational cost of both training and inference. Based on these advantages and the promising results obtained, we anticipate a profound impact on the broader field of signal processing, in particular on classification, regression and clustering problems.

Hysteresis Modeling in Iron-Dominated Magnets Based on a Multi-Layered Narx Neural Network Approach

A full-fledged neural network modeling, based on a Multi-layered Nonlinear Autoregressive Exogenous Neural Network (NARX) architecture, is proposed for quasi-static and dynamic hysteresis loops, one of the most challenging topics for computational magnetism. This modeling approach overcomes drawbacks in attaining better than percent-level accuracy of classical and recent approaches for accelerator magnets, that combine hybridization of standard hysteretic models and neural network architectures. By means of an incremental procedure, different Deep Neural Network Architectures are selected, fine-tuned and tested in order to predict magnetic hysteresis in the context of electromagnets. Tests and results show that the proposed NARX architecture best fits the measured magnetic field behavior of a reference quadrupole at CERN. In particular, the proposed modeling framework leads to a percent error below 0.02% for the magnetic field prediction, thus outperforming state of the art approaches and paving a very promising way for future real time applications.

Supervised Detection of Connector Lock Events with Optical Microphone Data

In manufacturing industry, one of the main targets is to increase automation and ultimately to avoid failures under all circumstances. The plugging and locking of connectors is a class of tasks which is yet hard to be automatized with sufficiently high process stability. Due to the variation of plugging positions and external disturbances, e.g. occlusion due to cables, the quality assessment of plugging processes has emerged as a challenging task for image-based systems. For this reason, the proposed approach analyzes the inherent acoustic connector locking properties in combination with different neural network architectures in order to correctly identify connector locking signals and further to distinguish them from other machining events occurring in assembly plants. For this specific task, highly sensitive optical microphones have been applied for data acquisition. The proposed experiments are carried out under laboratory conditions as well as for the more complex situation in a real manufacturing environment. In this context, the usage of multimodal neural network architectures achieved highest levels in classification performance with accuracy levels close to 90%.

Individuality- and Commonality-Based Multiview Multilabel Learning

In multiview multilabel learning, each object is represented by several heterogeneous feature representations and is also annotated with a set of discrete nonexclusive labels. Previous studies typically focus on capturing the shared latent patterns among multiple views, while not sufficiently considering the diverse characteristics of individual views, which can cause performance degradation. In this article, we propose a novel approach [individuality- and commonality-based multiview multilabel learning (ICM2L)] to explicitly explore the individuality and commonality information of multilabel multiple view data in a unified model. Specifically, a common subspace is learned across different views to capture the shared patterns. Then, multiple individual classifiers are exploited to explore the characteristics of individual views. Next, an ensemble strategy is adopted to make a prediction. Finally, we develop an alternative solution to jointly optimize our model, which can enhance the robustness of the proposed model toward rare labels and reinforce the reciprocal effects of individuality and commonality among heterogeneous views, and thus further improve the performance. Experiments on various real-word datasets validate the effectiveness of ICM2L against the state-of-the-art solutions, and ICM2L can leverage the individuality and commonality information to achieve an improved performance as well as to enhance the robustness toward rare labels.

An Experimental Review on Deep Learning Architectures for Time Series Forecasting

In recent years, deep learning techniques have outperformed traditional models in many machine learning tasks. Deep neural networks have successfully been applied to address time series forecasting problems, which is a very important topic in data mining. They have proved to be an effective solution given their capacity to automatically learn the temporal dependencies present in time series. However, selecting the most convenient type of deep neural network and its parametrization is a complex task that requires considerable expertise. Therefore, there is a need for deeper studies on the suitability of all existing architectures for different forecasting tasks. In this work, we face two main challenges: a comprehensive review of the latest works using deep learning for time series forecasting and an experimental study comparing the performance of the most popular architectures. The comparison involves a thorough analysis of seven types of deep learning models in terms of accuracy and efficiency. We evaluate the rankings and distribution of results obtained with the proposed models under many different architecture configurations and training hyperparameters. The datasets used comprise more than 50,000 time series divided into 12 different forecasting problems. By training more than 38,000 models on these data, we provide the most extensive deep learning study for time series forecasting. Among all studied models, the results show that long short-term memory (LSTM) and convolutional networks (CNN) are the best alternatives, with LSTMs obtaining the most accurate forecasts. CNNs achieve comparable performance with less variability of results under different parameter configurations, while also being more efficient.

Neural Networks with Emotion Associations, Topic Modeling and Supervised Term Weighting for Sentiment Analysis

Automated sentiment analysis is becoming increasingly recognized due to the growing importance of social media and e-commerce platform review websites. Deep neural networks outperform traditional lexicon-based and machine learning methods by effectively exploiting contextual word embeddings to generate dense document representation. However, this representation model is not fully adequate to capture topical semantics and the sentiment polarity of words. To overcome these problems, a novel sentiment analysis model is proposed that utilizes richer document representations of word-emotion associations and topic models, which is the main computational novelty of this study. The sentiment analysis model integrates word embeddings with lexicon-based sentiment and emotion indicators, including negations and emoticons, and to further improve its performance, a topic modeling component is utilized together with a bag-of-words model based on a supervised term weighting scheme. The effectiveness of the proposed model is evaluated using large datasets of Amazon product reviews and hotel reviews. Experimental results prove that the proposed document representation is valid for the sentiment analysis of product and hotel reviews, irrespective of their class imbalance. The results also show that the proposed model improves on existing machine learning methods.

Convolutional neural networks for the automatic diagnosis of melanoma: An extensive experimental study

Melanoma is the type of skin cancer with the highest levels of mortality, and it is more dangerous because it can spread to other parts of the body if not caught and treated early. Melanoma diagnosis is a complex task, even for expert dermatologists, mainly due to the great variety of morphologies in moles of patients. Accordingly, the automatic diagnosis of melanoma is a task that poses the challenge of developing efficient computational methods that ease the diagnostic and, therefore, aid dermatologists in decision-making. In this work, an extensive analysis was conducted, aiming at assessing and illustrating the effectiveness of convolutional neural networks in coping with this complex task. To achieve this objective, twelve well-known convolutional network models were evaluated on eleven public image datasets. The experimental study comprised five phases, where first it was analyzed the sensitivity of the models regarding the optimization algorithm used for their training, and then it was analyzed the impact in performance when using different techniques such as cost-sensitive learning, data augmentation and transfer learning. The conducted study confirmed the usefulness, effectiveness and robustness of different convolutional architectures in solving melanoma diagnosis problem. Also, important guidelines to researchers working on this area were provided, easing the selection of both the proper convolutional model and technique according the characteristics of data.

Machine learning (ML) for the diagnosis of autism spectrum disorder (ASD) using brain imaging

Autism spectrum disorder (ASD) is a neurodevelopmental incurable disorder with a long diagnostic period encountered in the early years of life. If diagnosed early, the negative effects of this disease can be reduced by starting special education early. Machine learning (ML), an increasingly ubiquitous technology, can be applied for the early diagnosis of ASD. The aim of this study is to examine and provide a comprehensive state-of-the-art review of ML research for the diagnosis of ASD based on (a) structural magnetic resonance image (MRI), (b) functional MRI and (c) hybrid imaging techniques over the past decade. The accuracy of the studies with a large number of participants is in general lower than those with fewer participants leading to the conclusion that further large-scale studies are needed. An examination of the age of the participants shows that the accuracy of the automated diagnosis of ASD is higher at a younger age range. ML technology is expected to contribute significantly to the early and rapid diagnosis of ASD in the coming years and become available to clinicians in the near future. This review is aimed to facilitate that.