Survey on Multi-Output Learning

Robotic Kinesthesia: Estimating Object Geometry and Material With Robot's Haptic Senses

Humans excel at determining the shape and material of objects through touch. Drawing inspiration from this ability, we propose a robotic system that incorporates haptic sensing capability into its artificial recognition system to jointly learn the shape and material types of an object. To achieve this, we employ a serially connected robotic arm and develop a supervised learning task that learns and classifies target surface geometry and material types using multivariate time-series data from joint torque sensors. Additionally, we propose a joint torque-to-position generation task to derive a one-dimensional surface profile based on torque measurements. Experimental results successfully validate the proposed torque-based classification and regression tasks, suggesting that a robotic system can employ haptic sensing (i.e., perceived force) from each joint to recognize material types and geometry, akin to human abilities.

GRNN-based cascade ensemble model for non-destructive damage state identification: small data approach

Assessing the structural integrity of ageing structures that are affected by climate-induced stressors, challenges traditional engineering methods. The reason is that structural degradation often initiates and advances without any notable warning until visible severe damage or catastrophic failures occur. An example of this, is the conventional inspection methods for prestressed concrete bridges which fail to interpret large permanent deflections because the causes-typically tendon loss-are barely visible or measurable. In many occasions, traditional inspections fail to discern these latent defects and damage, leading to the need for expensive continuous structural health monitoring towards informed assessments to enable appropriate structural interventions. This is a capability gap that has led to fatalities and extensive losses because the operators have very little time to react. This study addresses this gap by proposing a novel machine learning approach to inform a rapid non-destructive assessment of bridge damage states based on measurable structural deflections. First, a comprehensive training dataset is assembled by simulating various plausible bridge damage scenarios associated with different degrees and patterns of tendon losses, the integrity of which is vital for the health of bridge decks. Second, a novel General Regression Neural Network (GRNN)-based cascade ensemble model, tailored for predicting three interdependent output attributes using limited datasets, is developed. The proposed cascade model is optimised by utilising the differential evolution method. Modelling and validation were conducted for a real long-span bridge. The results confirm the efficacy of the proposed model in accurately identifying bridge damage states when compared to existing methods. The model developed demonstrates exceptional prediction accuracy and reliability, underscoring its practical value in non-destructive bridge damage assessment, which can facilitate effective restoration planning.

Mineral oil emulsion species and concentration prediction using multi-output neural network based on fluorescence spectra in the solar-blind UV band

The accurate monitoring of oil spills is crucial for effective oil spill recovery, volume determination, and cleanup. Oil slicks become emulsified under the effects of wind and waves, which increases the consistency of the oil spills. This phenomenon makes oil spills more challenging to handle and exacerbates environmental pollution. In this study, the variation of the solar-blind ultraviolet (UV) fluorescence spectra obtained from simulated oil spills with different oil types and oil-water ratios was investigated. By designing and constructing a multi-angle excitation and detection system, an apparent fluorescence peak of the oil emulsions was observed at around 290 nm under 220 nm excitation. By utilizing competitive adaptive reweighted sampling (CARS) and multi-output neural network algorithms, both the types and concentrations of the emulsified oils were obtained simultaneously. The classification accuracy for identifying the oil type exceeds 98%, and the mean absolute percentage error (MAPE) for concentration regression is around 2%. The results indicate that active solar-blind UV fluorescence could become a supplementary method for on-site oil spill detection to achieve comprehensive monitoring of oil spills. This study provides potential applications for UV-induced fluorescence spectrometry in oil spill on-site monitoring during the daytime.

Rapid Determination of Wine Grape Maturity Level from pH, Titratable Acidity, and Sugar Content Using Non-Destructive In Situ Infrared Spectroscopy and Multi-Head Attention Convolutional Neural Networks

In the pursuit of enhancing the wine production process through the utilization of new technologies in viticulture, this study presents a novel approach for the rapid assessment of wine grape maturity levels using non-destructive, in situ infrared spectroscopy and artificial intelligence techniques. Building upon our previous work focused on estimating sugar content (∘Brix) from the visible and near-infrared (VNIR) and short-wave infrared (SWIR) regions, this research expands its scope to encompass pH and titratable acidity, critical parameters determining the grape maturity degree, and in turn, wine quality, offering a more representative estimation pathway. Data were collected from four grape varieties-Chardonnay, Malagouzia, Sauvignon Blanc, and Syrah-during the 2023 harvest and pre-harvest phenological stages in the vineyards of Ktima Gerovassiliou, northern Greece. A comprehensive spectral library was developed, covering the VNIR-SWIR spectrum (350-2500 nm), with measurements performed in situ. Ground truth data for pH, titratable acidity, and sugar content were obtained using conventional laboratory methods: total soluble solids (TSS) (∘Brix) by refractometry, titratable acidity by titration (expressed as mg tartaric acid per liter of must) and pH by a pH meter, analyzed at different maturation stages in the must samples. The maturity indicators were predicted from the point hyperspectral data by employing machine learning algorithms, including Partial Least Squares regression (PLS), Random Forest regression (RF), Support Vector Regression (SVR), and Convolutional Neural Networks (CNN), in conjunction with various pre-processing techniques. Multi-output models were also considered to simultaneously predict all three indicators to exploit their intercorrelations. A novel multi-input-multi-output CNN model was also proposed, incorporating a multi-head attention mechanism and enabling the identification of the spectral regions it focuses on, and thus having a higher interpretability degree. Our results indicate high accuracy in the estimation of sugar content, pH, and titratable acidity, with the best models yielding mean R2 values of 0.84, 0.76, and 0.79, respectively, across all properties. The multi-output models did not improve the prediction results compared to the best single-output models, and the proposed CNN model was on par with the next best model. The interpretability analysis highlighted that the CNN model focused on spectral regions associated with the presence of sugars (i.e., glucose and fructose) and of the carboxylic acid group. This study underscores the potential of portable spectrometry for real-time, non-destructive assessments of wine grape maturity, thereby providing valuable tools for informed decision making in the wine production industry. By integrating pH and titratable acidity into the analysis, our approach offers a holistic view of grape quality, facilitating more comprehensive and efficient viticultural practices.

Understanding the meanings of citations using sentiment, role, and citation function classifications

Traditional citation analyses use quantitative methods only, even though there is meaning in the sentences containing citations within the text. This article analyzes three citation meanings: sentiment, role, and function. We compare citation meanings patterns between fields of science and propose an appropriate deep learning model to classify the three meanings automatically at once. The data comes from Indonesian journal articles covering five different areas of science: food, energy, health, computer, and social science. The sentences in the article text were classified manually and used as training data for an automatic classification model. Several classic models were compared with the proposed multi-output convolutional neural network model. The manual classification revealed similar patterns in citation meaning across the science fields: (1) not many authors exhibit polarity when citing, (2) citations are still rarely used, and (3) citations are used mostly for introductions and establishing relations instead of for comparisons with and utilizing previous research. The proposed model’s automatic classification metric achieved a macro F1 score of 0.80 for citation sentiment, 0.84 for citation role, and 0.88 for citation function. The model can classify minority classes well concerning the unbalanced dataset. A machine model that can classify several citation meanings automatically is essential for analyzing big data of journal citations.

The Emerging Trends of Multi-Label Learning

Exabytes of data are generated daily by humans, leading to the growing needs for new efforts in dealing with the grand challenges for multi-label learning brought by big data. For example, extreme multi-label classification is an active and rapidly growing research area that deals with classification tasks with extremely large number of classes or labels; utilizing massive data with limited supervision to build a multi-label classification model becomes valuable for practical applications, etc. Besides these, there are tremendous efforts on how to harvest the strong learning capability of deep learning to better capture the label dependencies in multi-label learning, which is the key for deep learning to address real-world classification tasks. However, it is noted that there have been a lack of systemic studies that focus explicitly on analyzing the emerging trends and new challenges of multi-label learning in the era of big data. It is imperative to call for a comprehensive survey to fulfil this mission and delineate future research directions and new applications.

An Improved Tiered Head Pose Estimation Network with Self-Adjust Loss Function

As an important task in computer vision, head pose estimation has been widely applied in both academia and industry. However, there remains two challenges in the field of head pose estimation: (1) even given the same task (e.g., tiredness detection), the existing algorithms usually consider the estimation of the three angles (i.e., roll, yaw, and pitch) as separate facets, which disregard their interplay as well as differences and thus share the same parameters for all layers; and (2) the discontinuity in angle estimation definitely reduces the accuracy. To solve these two problems, a THESL-Net (tiered head pose estimation with self-adjust loss network) model is proposed in this study. Specifically, first, an idea of stepped estimation using distinct network layers is proposed, gaining a greater freedom during angle estimation. Furthermore, the reasons for the discontinuity in angle estimation are revealed, including not only labeling the dataset with quaternions or Euler angles, but also the loss function that simply adds the classification and regression losses. Subsequently, a self-adjustment constraint on the loss function is applied, making the angle estimation more consistent. Finally, to examine the influence of different angle ranges on the proposed model, experiments are conducted on three popular public benchmark datasets, BIWI, AFLW2000, and UPNA, demonstrating that the proposed model outperforms the state-of-the-art approaches.

Unsupervised Anomaly Detection for Time Series Data of Spacecraft Using Multi-Task Learning

Although in-orbit anomaly detection is extremely important to ensure spacecraft safety, the complex spatial-temporal correlation and sparsity of anomalies in the data pose significant challenges. This study proposes the new multi-task learning-based time series anomaly detection (MTAD) method, which captures the spatial-temporal correlation of the data to learn the generalized normal patterns and hence facilitates anomaly detection. First, four proxy tasks are implemented for feature extraction through joint learning: (1) Long short-term memory-based data prediction; (2) autoencoder-based latent representation learning and data reconstruction; (3) variational autoencoder-based latent representation learning and data reconstruction; and (4) joint latent representation-based data prediction. Proxy Tasks 1 and 4 capture the temporal correlation of the data by fusing the latent space, whereas Tasks 2 and 3 fully capture the spatial correlation of the data. The isolation forest algorithm then detects anomalies from the extracted features. Application to a real spacecraft dataset reveals the superiority of our method over existing techniques, and further ablation testing for each task proves the effectiveness of fusing multiple tasks. The proposed MTAD method demonstrates promising potential for effective in-orbit anomaly detection for spacecraft.

Geometric Metric Learning for Multi-Output Learning

Due to its wide applications, multi-output learning that predicts multiple output values for a single input at the same time is becoming more and more attractive. As one of the most popular frameworks for dealing with multi-output learning, the performance of the k-nearest neighbor (kNN) algorithm mainly depends on the metric used to compute the distance between different instances. In this paper, we propose a novel cost-weighted geometric mean metric learning method for multi-output learning. Specifically, this method learns a geometric mean metric which can make the distance between the input embedding and its correct output be smaller than the distance between the input embedding and the outputs of its nearest neighbors. The learned geometric mean metric can discover output dependencies and move the instances with different outputs far away in the embedding space. In addition, our objective function has a closed solution, and thus the calculation speed is very fast. Compared with state-of-the-art methods, it is easier to explain and also has a faster calculation speed. Experiments conducted on two multi-output learning tasks (i.e., multi-label classification and multi-objective regression) have confirmed that our method provides better results than state-of-the-art methods.

Attack Graph Generation with Machine Learning for Network Security

Recently, with the discovery of various security threats, diversification of hacking attacks, and changes in the network environment such as the Internet of Things, security threats on the network are increasing. Attack graph is being actively studied to cope with the recent increase in cyber threats. However, the conventional attack graph generation method is costly and time-consuming. In this paper, we propose a cheap and simple method for generating the attack graph. The proposed approach consists of learning and generating stages. First, it learns how to generate an attack path from the attack graph, which is created based on the vulnerability database, using machine learning and deep learning. Second, it generates the attack graph using network topology and system information with a machine learning model that is trained with the attack graph generated from the vulnerability database. We construct the dataset for attack graph generation with topological and system information. The attack graph generation problem is recast as a multi-output learning and binary classification problem. It shows attack path detection accuracy of 89.52% in the multi-output learning approach and 80.68% in the binary classification approach using the in-house dataset, respectively.

FORECASTING SPATIALLY CORRELATED TARGETS: SIMULTANEOUS PREDICTION OF HOUSING MARKET ACTIVITY ACROSS MULTIPLE AREAS

This study involved the development of an approach to forecast house prices and trading volumes across multiple areas simultaneously. Spatially correlated targets, such as house prices, can be predicted more accurately by leveraging the correlations across adjacent areas. A multi-output recurrent neural network, a deep learning algorithm specifically developed to analyze sequence data, was utilized to forecast the house prices and trading volumes in the four chosen study areas. The forecasting accuracy of future house prices in one of the four geographical areas clearly improved; this area was found to be a price-lagging area, and the forecasting accuracy of this area significantly increased by exploiting the information of a price-leading area. As for the prediction of trading volumes, the difference in performance between the multi-output recurrent neural network and conventional models was very small. The results of this study are expected to promote the use of deep learning to predict the housing market activity through a simultaneous forecasting framework.

3D Convolution Recurrent Neural Networks for Multi-Label Earthquake Magnitude Classification

We examine a classification task in which signals of naturally occurring earthquakes are categorized ranging from minor to major, based on their magnitude. Generalized to a single-label classification task, most prior investigations have focused on assessing whether an earthquake’s magnitude falls into the minor or large categories. This procedure is often not practical since the tremor it generates has a wide range of variation in the neighboring regions based on the distance, depth, type of surface, and several other factors. We present an integrated 3-dimensional convolutional recurrent neural network (3D-CNN-RNN) trained to classify the seismic waveforms into multiple categories based on the problem formulation. Recent studies demonstrate using artificial intelligence-based techniques in earthquake detection and location estimation tasks with progress in collecting seismic data. However, less work has been performed in classifying the seismic signals into single or multiple categories. We leverage the use of a benchmark dataset comprising of earthquake waveforms having different magnitude and present 3D-CNN-RNN, a highly scalable neural network for multi-label classification problems. End-to-end learning has become a conventional approach in audio and image-related classification studies. However, for seismic signals classification, it has yet to be established. In this study, we propose to deploy the trained model on personal seismometers to effectively categorize earthquakes and increase the response time by leveraging the data-centric approaches. For this purpose, firstly, we transform the existing benchmark dataset into a series of multi-label examples. Secondly, we develop a novel 3D-CNN-RNN model for multi-label seismic event classification. Finally, we validate and evaluate the learned model with unseen seismic waveforms instances and report whether a specific event is associated with a particular class or not. Experimental results demonstrate the superiority and effectiveness of the proposed approach on unseen data using the multi-label classifier.

Multi-dimensional Classification via Selective Feature Augmentation

In multi-dimensional classification (MDC), the semantics of objects are characterized by multiple class spaces from different dimensions. Most MDC approaches try to explicitly model the dependencies among class spaces in output space. In contrast, the recently proposed feature augmentation strategy, which aims at manipulating feature space, has also been shown to be an effective solution for MDC. However, existing feature augmentation approaches only focus on designing holistic augmented features to be appended with the original features, while better generalization performance could be achieved by exploiting multiple kinds of augmented features. In this paper, we propose the selective feature augmentation strategy that focuses on synergizing multiple kinds of augmented features. Specifically, by assuming that only part of the augmented features is pertinent and useful for each dimension’s model induction, we derive a classification model which can fully utilize the original features while conduct feature selection for the augmented features. To validate the effectiveness of the proposed strategy, we generate three kinds of simple augmented features based on standard kNN, weighted kNN, and maximum margin techniques, respectively. Comparative studies show that the proposed strategy achieves superior performance against both state-of-the-art MDC approaches and its degenerated versions with either kind of augmented features.

Learn class hierarchy using convolutional neural networks

A large amount of research on Convolutional Neural Networks (CNN) has focused on flat Classification in the multi-class domain. In the real world, many problems are naturally expressed as hierarchical classification problems, in which the classes to be predicted are organized in a hierarchy of classes. In this paper, we propose a new architecture for hierarchical classification, introducing a stack of deep linear layers using cross-entropy loss functions combined to a center loss function. The proposed architecture can extend any neural network model and simultaneously optimizes loss functions to discover local hierarchical class relationships and a loss function to discover global information from the whole class hierarchy while penalizing class hierarchy violations. We experimentally show that our hierarchical classifier presents advantages to the traditional classification approaches finding application in computer vision tasks. The same approach can also be applied to some CNN for text classification.

Dynamically Polarizable Force Fields for Surface Simulations via Multi-output Classification Neural Networks

We present a general procedure to introduce electronic polarization into classical Molecular Dynamics (MD) force fields using a Neural Network (NN) model. We apply this framework to the simulation of a solid-liquid interface where the polarization of the surface is essential to correctly capture the main features of the system. By introducing a multi-input, multi-output NN and treating the surface polarization as a discrete classification problem, we are able to obtain very good accuracy in terms of quality of predictions. Through the definition of a custom loss function we are able to impose a physically motivated constraint within the NN itself making this model extremely versatile, especially in the modeling of different surface charge states. The NN is validated considering the redistribution of electronic charge density within a graphene based electrode in contact with an aqueous electrolyte solution, a system highly relevant to the development of next generation low-cost supercapacitors. We compare the performances of our NN/MD model against Quantum Mechanics/Molecular Dynamics simulations where we obtain a most satisfactory agreement.

RMoR-Aion: Robust Multioutput Regression by Simultaneously Alleviating Input and Output Noises

Multioutput regression, referring to simultaneously predicting multiple continuous output variables with a single model, has drawn increasing attention in the machine learning community due to its strong ability to capture the correlations among multioutput variables. The methodology of output space embedding, built upon the low-rank assumption, is now the mainstream for multioutput regression since it can effectively reduce the parameter numbers while achieving effective performance. The existing low-rank methods, however, are sensitive to the noises of both inputs and outputs, referring to the noise problem. In this article, we develop a novel multioutput regression method by simultaneously alleviating input and output noises, namely, robust multioutput regression by alleviating input and output noises (RMoR-Aion), where both the noises of the input and output are exploited by leveraging auxiliary matrices. Furthermore, we propose a prediction output manifold constraint with the correlation information regarding the output variables to further reduce the adversarial effects of the noise. Our empirical studies demonstrate the effectiveness of RMoR-Aion compared with the state-of-the-art baseline methods, and RMoR-Aion is more stable in the settings with artificial noise.

When Gaussian Process Meets Big Data: A Review of Scalable GPs

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process regression (GPR), a well-known nonparametric, and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. However, they have not yet been comprehensively reviewed and analyzed to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this article is devoted to reviewing state-of-the-art scalable GPs involving two main categories: global approximations that distillate the entire data and local approximations that divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations that modify the prior but perform exact inference, posterior approximations that retain exact prior but perform approximate inference, and structured sparse approximations that exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

Unseen Land Cover Classification from High-Resolution Orthophotos Using Integration of Zero-Shot Learning and Convolutional Neural Networks

Zero-shot learning (ZSL) is an approach to classify objects unseen during the training phase and shown to be useful for real-world applications, especially when there is a lack of sufficient training data. Only a limited amount of works has been carried out on ZSL, especially in the field of remote sensing. This research investigates the use of a convolutional neural network (CNN) as a feature extraction and classification method for land cover mapping using high-resolution orthophotos. In the feature extraction phase, we used a CNN model with a single convolutional layer to extract discriminative features. In the second phase, we used class attributes learned from the Word2Vec model (pre-trained by Google News) to train a second CNN model that performed class signature prediction by using both the features extracted by the first CNN and class attributes during training and only the features during prediction. We trained and tested our models on datasets collected over two subareas in the Cameron Highlands (training dataset, first test dataset) and Ipoh (second test dataset) in Malaysia. Several experiments have been conducted on the feature extraction and classification models regarding the main parameters, such as the network’s layers and depth, number of filters, and the impact of Gaussian noise. As a result, the best models were selected using various accuracy metrics such as top-k categorical accuracy for k = [1,2,3], Recall, Precision, and F1-score. The best model for feature extraction achieved 0.953 F1-score, 0.941 precision, 0.882 recall for the training dataset and 0.904 F1-score, 0.869 precision, 0.949 recall for the first test dataset, and 0.898 F1-score, 0.870 precision, 0.838 recall for the second test dataset. The best model for classification achieved an average of 0.778 top-one, 0.890 top-two and 0.942 top-three accuracy, 0.798 F1-score, 0.766 recall and 0.838 precision for the first test dataset and 0.737 top-one, 0.906 top-two, 0.924 top-three, 0.729 F1-score, 0.676 recall and 0.790 precision for the second test dataset. The results demonstrated that the proposed ZSL is a promising tool for land cover mapping based on high-resolution photos.

On the Application of Machine Learning to the Design of UAV-Based 5G Radio Access Networks

A groundbreaking design of radio access networks (RANs) is needed to fulfill 5G traffic requirements. To this aim, a cost-effective and flexible strategy consists of complementing terrestrial RANs with unmanned aerial vehicles (UAVs). However, several problems must be solved in order to effectively deploy such UAV-based RANs (U-RANs). Indeed, due to the high complexity and heterogeneity of these networks, model-based design approaches, often relying on restrictive assumptions and constraints, exhibit severe limitation in real-world scenarios. Moreover, design of a set of appropriate protocols for such U-RANs is a highly sophisticated task. In this context, machine learning (ML) emerges as a useful tool to obtain practical and effective solutions. In this paper, we discuss why, how, and which types of ML methods are useful for designing U-RANs, by focusing in particular on supervised and reinforcement learning strategies.