A review of learning vector quantization classifiers

Low-Power Analog Integrated Architecture of the Voting Classification Algorithm for Diabetes Disease Prediction

A low-power ( 600nW), fully analog integrated architecture for a voting classification algorithm is introduced. It can effectively handle multiple-input features, maintaining exceptional levels of accuracy and with very low power consumption. The proposed architecture is based on a versatile Voting algorithm that selectively incorporates one of three key classification models: Bayes or Centroid, or, the Learning Vector Quantization model; all of which are implemented using Gaussian-likelihood and Euclidean distance function circuits, as well as a current comparison circuit. To evaluate the proposed architecture, a comprehensive comparison with popular analog classifiers is performed, using real-life diabetes dataset. All model architectures were trained using Python and compared with the software-based classifiers. The circuit implementations were performed using the TSMC nm CMOS process technology and the Cadence IC Suite was utilized for the design, schematic and post-layout simulations. The proposed classifiers achieved sensitivity of and specificity of .

Subspace corrected relevance learning with application in neuroimaging

In machine learning, data often comes from different sources, but combining them can introduce extraneous variation that affects both generalization and interpretability. For example, we investigate the classification of neurodegenerative diseases using FDG-PET data collected from multiple neuroimaging centers. However, data collected at different centers introduces unwanted variation due to differences in scanners, scanning protocols, and processing methods. To address this issue, we propose a two-step approach to limit the influence of center-dependent variation on the classification of healthy controls and early vs. late-stage Parkinson's disease patients. First, we train a Generalized Matrix Learning Vector Quantization (GMLVQ) model on healthy control data to identify a "relevance space" that distinguishes between centers. Second, we use this space to construct a correction matrix that restricts a second GMLVQ system's training on the diagnostic problem. We evaluate the effectiveness of this approach on the real-world multi-center datasets and simulated artificial dataset. Our results demonstrate that the approach produces machine learning systems with reduced bias - being more specific due to eliminating information related to center differences during the training process - and more informative relevance profiles that can be interpreted by medical experts. This method can be adapted to similar problems outside the neuroimaging domain, as long as an appropriate "relevance space" can be identified to construct the correction matrix.

An LVQ clustering algorithm based on neighborhood granules

Learning Vector Quantization (LVQ) is a clustering method with supervised information, simple structures, and powerful functions. LVQ assumes that the data samples are labeled, and the learning process uses labels to assist clustering. However, the LVQ is sensitive to initial values, resulting in a poor clustering effect. To overcome these shortcomings, a granular LVQ clustering algorithm is proposed by adopting the neighborhood granulation technology and the LVQ. Firstly, the neighborhood granulation is carried out on some features of a sample of the data set, then a neighborhood granular vector is formed. Furthermore, the size and operations of neighborhood granular vectors are defined, and the relative and absolute granular distances between granular vectors are proposed. Finally, these granular distances are proved to be metrics, and a granular LVQ clustering algorithm is designed. Some experiments are tested on several UCI data sets, and the results show that the granular LVQ clustering is better than the traditional LVQ clustering under suitable neighborhood parameters and distance measurement.

AI-Driven Aeronautical Ad Hoc Networks for 6G Wireless: Challenges, Opportunities, and the Road Ahead

Aeronautical ad hoc network (AANET) has been considered a promising candidate to complete the vision of "Internet in the sky" by supporting high-speed broadband connections on airplanes for 6G networks. However, the specific characteristics of AANET restrict the applicability of conventional topology and routing management algorithms. Here, these conventional methodologies reduce the packet delivery success of AANET with higher transfer delay. At that point, the artificial intelligence (AI)-driven solutions have been adapted to AANET to provide intelligent frameworks and architectures to cope with the high complexity. The AI-driven AANET can provide intelligent topology formation, sustainability, and routing management decisions in an automated fashion by considering its specific characteristics during the learning operations. More clearly, AI-driven AANETs support intelligent management architectures, overcoming conventional methodologies' drawbacks. Although AI-based management approaches are widely used in terrestrial networks, there is a lack of a comprehensive study that supports AI-driven solutions for AANETs. To this end, this article explores the possible utilization of primary AI methodologies on the road to AI-driven AANET. Specifically, the article addresses unsupervised, supervised, and reinforcement learning as primary AI methodologies to enable intelligent AANET topology formation, sustainability, and routing management. Here, we identify the challenges and opportunities of these primary AI methodologies during the execution of AANET management. Furthermore, we discuss the critical issue of security in AANET before providing open issues.

Supervised learning in the presence of concept drift: a modelling framework

We present a modelling framework for the investigation of supervised learning in non-stationary environments. Specifically, we model two example types of learning systems: prototype-based learning vector quantization (LVQ) for classification and shallow, layered neural networks for regression tasks. We investigate so-called student–teacher scenarios in which the systems are trained from a stream of high-dimensional, labeled data. Properties of the target task are considered to be non-stationary due to drift processes while the training is performed. Different types of concept drift are studied, which affect the density of example inputs only, the target rule itself, or both. By applying methods from statistical physics, we develop a modelling framework for the mathematical analysis of the training dynamics in non-stationary environments. Our results show that standard LVQ algorithms are already suitable for the training in non-stationary environments to a certain extent. However, the application of weight decay as an explicit mechanism of forgetting does not improve the performance under the considered drift processes. Furthermore, we investigate gradient-based training of layered neural networks with sigmoidal activation functions and compare with the use of rectified linear units. Our findings show that the sensitivity to concept drift and the effectiveness of weight decay differs significantly between the two types of activation function.

A novel cluster detection of COVID-19 patients and medical disease conditions using improved evolutionary clustering algorithm star

With the increasing number of samples, the manual clustering of COVID-19 and medical disease data samples becomes time-consuming and requires highly skilled labour. Recently, several algorithms have been used for clustering medical datasets deterministically; however, these definitions have not been effective in grouping and analysing medical diseases. The use of evolutionary clustering algorithms may help to effectively cluster these diseases. On this presumption, we improved the current evolutionary clustering algorithm star (ECA*), called iECA*, in three manners: (i) utilising the elbow method to find the correct number of clusters; (ii) cleaning and processing data as part of iECA* to apply it to multivariate and domain-theory datasets; (iii) using iECA* for real-world applications in clustering COVID-19 and medical disease datasets. Experiments were conducted to examine the performance of iECA* against state-of-the-art algorithms using performance and validation measures (validation measures, statistical benchmarking, and performance ranking framework). The results demonstrate three primary findings. First, iECA* was more effective than other algorithms in grouping the chosen medical disease datasets according to the cluster validation criteria. Second, iECA* exhibited the lower execution time and memory consumption for clustering all the datasets, compared to the current clustering methods analysed. Third, an operational framework was proposed to rate the effectiveness of iECA* against other algorithms in the datasets analysed, and the results indicated that iECA* exhibited the best performance in clustering all medical datasets. Further research is required on real-world multi-dimensional data containing complex knowledge fields for experimental verification of iECA* compared to evolutionary algorithms.

Probabilistic learning vector quantization on manifold of symmetric positive definite matrices

In this paper, we develop a new classification method for manifold-valued data in the framework of probabilistic learning vector quantization. In many classification scenarios, the data can be naturally represented by symmetric positive definite matrices, which are inherently points that live on a curved Riemannian manifold. Due to the non-Euclidean geometry of Riemannian manifolds, traditional Euclidean machine learning algorithms yield poor results on such data. In this paper, we generalize the probabilistic learning vector quantization algorithm for data points living on the manifold of symmetric positive definite matrices equipped with Riemannian natural metric (affine-invariant metric). By exploiting the induced Riemannian distance, we derive the probabilistic learning Riemannian space quantization algorithm, obtaining the learning rule through Riemannian gradient descent. Empirical investigations on synthetic data, image data , and motor imagery electroencephalogram (EEG) data demonstrate the superior performance of the proposed method.

A novel diagnostic and prognostic approach for unresponsive patients with anthroponotic cutaneous leishmaniasis using artificial neural networks

Cutaneous leishmaniasis (CL) imposes a major health burden throughout the tropical and subtropical regions of the globe. Unresponsive cases are common phenomena occurred upon exposure to the standard drugs. Therefore, rapid detection, prognosis and classification of the disease are crucial for selecting the proper treatment modality. Using machine learning (ML) techniques, this study aimed to detect unresponsive cases of ACL, caused by Leishmania tropica, which will consequently be used for a more effective treatment modality. This study was conducted as a case-control setting. Patients were selected in a major ACL focus from both unresponsive and responsive cases. Nine unique and relevant features of patients with ACL were selected. To categorize the patients, different classifier models such as k-nearest neighbors (KNN), support vector machines (SVM), multilayer perceptron (MLP), learning vector quantization (LVQ) and multipass LVQ were applied and compared for this supervised learning task. Comparison of the receiver operating characteristic graphs (ROC) and confusion plots for the above models represented that MLP was a fairly accurate prediction model to solve this problem. The overall accuracy in terms of sensitivity, specificity and area under ROC curve (AUC) of MLP classifier were 87.8%, 90.3%, 86% and 0.88%, respectively. Moreover, the duration of the skin lesion was the most influential feature in MLP classifier, while gender was the least. The present investigation demonstrated that MLP model could be utilized for rapid detection, accurate prognosis and effective treatment of unresponsive patients with ACL. The results showed that the major feature affecting the responsiveness to treatments is the duration of the lesion. This novel approach is unique and can be beneficial in developing diagnostic, prophylactic and therapeutic measures against the disease. This attempt could be a preliminary step towards the expansion of ML application in future directions.

Self-incremental learning vector quantization with human cognitive biases

Human beings have adaptively rational cognitive biases for efficiently acquiring concepts from small-sized datasets. With such inductive biases, humans can generalize concepts by learning a small number of samples. By incorporating human cognitive biases into learning vector quantization (LVQ), a prototype-based online machine learning method, we developed self-incremental LVQ (SILVQ) methods that can be easily interpreted. We first describe a method to automatically adjust the learning rate that incorporates human cognitive biases. Second, SILVQ, which self-increases the prototypes based on the method for automatically adjusting the learning rate, is described. The performance levels of the proposed methods are evaluated in experiments employing four real and two artificial datasets. Compared with the original learning vector quantization algorithms, our methods not only effectively remove the need for parameter tuning, but also achieve higher accuracy from learning small numbers of instances. In the cases of larger numbers of instances, SILVQ can still achieve an accuracy that is equal to or better than those of existing representative LVQ algorithms. Furthermore, SILVQ can learn linearly inseparable conceptual structures with the required and sufficient number of prototypes without overfitting.

Improving the Accuracy of Nearest-Neighbor Classification Using Principled Construction and Stochastic Sampling of Training-Set Centroids

A conceptually simple way to classify images is to directly compare test-set data and training-set data. The accuracy of this approach is limited by the method of comparison used, and by the extent to which the training-set data cover configuration space. Here we show that this coverage can be substantially increased using coarse-graining (replacing groups of images by their centroids) and stochastic sampling (using distinct sets of centroids in combination). We use the MNIST and Fashion-MNIST data sets to show that a principled coarse-graining algorithm can convert training images into fewer image centroids without loss of accuracy of classification of test-set images by nearest-neighbor classification. Distinct batches of centroids can be used in combination as a means of stochastically sampling configuration space, and can classify test-set data more accurately than can the unaltered training set. On the MNIST and Fashion-MNIST data sets this approach converts nearest-neighbor classification from a mid-ranking- to an upper-ranking member of the set of classical machine-learning techniques.

Generalized Learning Riemannian Space Quantization: A Case Study on Riemannian Manifold of SPD Matrices

Learning vector quantization (LVQ) is a simple and efficient classification method, enjoying great popularity. However, in many classification scenarios, such as electroencephalogram (EEG) classification, the input features are represented by symmetric positive-definite (SPD) matrices that live in a curved manifold rather than vectors that live in the flat Euclidean space. In this article, we propose a new classification method for data points that live in the curved Riemannian manifolds in the framework of LVQ. The proposed method alters generalized LVQ (GLVQ) with the Euclidean distance to the one operating under the appropriate Riemannian metric. We instantiate the proposed method for the Riemannian manifold of SPD matrices equipped with the Riemannian natural metric. Empirical investigations on synthetic data and real-world motor imagery EEG data demonstrate that the performance of the proposed generalized learning Riemannian space quantization can significantly outperform the Euclidean GLVQ, generalized relevance LVQ (GRLVQ), and generalized matrix LVQ (GMLVQ). The proposed method also shows competitive performance to the state-of-the-art methods on the EEG classification of motor imagery tasks.

Targeted Metabolic Profiling of Urine Highlights a Potential Biomarker Panel for the Diagnosis of Alzheimer's Disease and Mild Cognitive Impairment: A Pilot Study

The lack of sensitive and specific biomarkers for the early detection of mild cognitive impairment (MCI) and Alzheimer’s disease (AD) is a major hurdle to improving patient management. A targeted, quantitative metabolomics approach using both ¹H NMR and mass spectrometry was employed to investigate the performance of urine metabolites as potential biomarkers for MCI and AD. Correlation-based feature selection (CFS) and least absolute shrinkage and selection operator (LASSO) methods were used to develop biomarker panels tested using support vector machine (SVM) and logistic regression models for diagnosis of each disease state. Metabolic changes were investigated to identify which biochemical pathways were perturbed as a direct result of MCI and AD in urine. Using SVM, we developed a model with 94% sensitivity, 78% specificity, and 78% AUC to distinguish healthy controls from AD sufferers. Using logistic regression, we developed a model with 85% sensitivity, 86% specificity, and an AUC of 82% for AD diagnosis as compared to cognitively healthy controls. Further, we identified 11 urinary metabolites that were significantly altered to include glucose, guanidinoacetate, urocanate, hippuric acid, cytosine, 2- and 3-hydroxyisovalerate, 2-ketoisovalerate, tryptophan, trimethylamine N oxide, and malonate in AD patients, which are also capable of diagnosing MCI, with a sensitivity value of 76%, specificity of 75%, and accuracy of 81% as compared to healthy controls. This pilot study suggests that urine metabolomics may be useful for developing a test capable of diagnosing and distinguishing MCI and AD from cognitively healthy controls.

Comparison of Instance Selection and Construction Methods with Various Classifiers

Instance selection and construction methods were originally designed to improve the performance of the k-nearest neighbors classifier by increasing its speed and improving the classification accuracy. These goals were achieved by eliminating redundant and noisy samples, thus reducing the size of the training set. In this paper, the performance of instance selection methods is investigated in terms of classification accuracy and reduction of training set size. The classification accuracy of the following classifiers is evaluated: decision trees, random forest, Naive Bayes, linear model, support vector machine and k-nearest neighbors. The obtained results indicate that for the most of the classifiers compressing the training set affects prediction performance and only a small group of instance selection methods can be recommended as a general purpose preprocessing step. These are learning vector quantization based algorithms, along with the Drop2 and Drop3. Other methods are less efficient or provide low compression ratio.

Statistical Mechanics of On-Line Learning Under Concept Drift

We introduce a modeling framework for the investigation of on-line machine learning processes in non-stationary environments. We exemplify the approach in terms of two specific model situations: In the first, we consider the learning of a classification scheme from clustered data by means of prototype-based Learning Vector Quantization (LVQ). In the second, we study the training of layered neural networks with sigmoidal activations for the purpose of regression. In both cases, the target, i.e., the classification or regression scheme, is considered to change continuously while the system is trained from a stream of labeled data. We extend and apply methods borrowed from statistical physics which have been used frequently for the exact description of training dynamics in stationary environments. Extensions of the approach allow for the computation of typical learning curves in the presence of concept drift in a variety of model situations. First results are presented and discussed for stochastic drift processes in classification and regression problems. They indicate that LVQ is capable of tracking a classification scheme under drift to a non-trivial extent. Furthermore, we show that concept drift can cause the persistence of sub-optimal plateau states in gradient based training of layered neural networks for regression.

Study of Face Orientation Recognition Based on Neural Network

Learning vector quantization (LVQ) network and back-propagation (BP) network are constructed easily making use of MATLAB toolbox on the basis of maintaining the recognition rate. Face images are randomly selected from images set as training data of LVQ network and BP network. LVQ algorithm and BP algorithm are used to train network. The automatic recognition of face orientation is realized when the system obtains convergence network. First, all images are processed by edge detection. Then feature vectors representing position of the eye were extracted from edge detected images. Feature vectors of training set are sent to network to adjust the parameters which ensures the convergence speed and performance of the network. Experimental results show that the constructed LVQ network and BP network can judge face orientation according to feature vectors of input images. Generally, the recognition rate of LVQ network is higher than that of BP network. The LVQ network and BP network are both feasible and effective for face orientation recognition to some extent. The advantage of this work is that the recognition system is efficient and easy to promote. This paper focuses on how to use MATLAB easily to design identification network rather than the complexity of identification system. The future research will focus on the stability and robustness of recognition network.

Can Learning Vector Quantization be an Alternative to SVM and Deep Learning? - Recent Trends and Advanced Variants of Learning Vector Quantization for Classification Learning

Learning vector quantization (LVQ) is one of the most powerful approaches for prototype based classification of vector data, intuitively introduced by Kohonen. The prototype adaptation scheme relies on its attraction and repulsion during the learning providing an easy geometric interpretability of the learning as well as of the classification decision scheme. Although deep learning architectures and support vector classifiers frequently achieve comparable or even better results, LVQ models are smart alternatives with low complexity and computational costs making them attractive for many industrial applications like intelligent sensor systems or advanced driver assistance systems.

Nowadays, the mathematical theory developed for LVQ delivers sufficient justification of the algorithm making it an appealing alternative to other approaches like support vector machines and deep learning techniques.

This review article reports current developments and extensions of LVQ starting from the generalized LVQ (GLVQ), which is known as the most powerful cost function based realization of the original LVQ. The cost function minimized in GLVQ is an soft-approximation of the standard classification error allowing gradient descent learning techniques. The GLVQ variants considered in this contribution, cover many aspects like bordersensitive learning, application of non-Euclidean metrics like kernel distances or divergences, relevance learning as well as optimization of advanced statistical classification quality measures beyond the accuracy including sensitivity and specificity or area under the ROC-curve.

According to these topics, the paper highlights the basic motivation for these variants and extensions together with the mathematical prerequisites and treatments for integration into the standard GLVQ scheme and compares them to other machine learning approaches. For detailed description and mathematical theory behind all, the reader is referred to the respective original articles.

Thus, the intention of the paper is to provide a comprehensive overview of the stateof- the-art serving as a starting point to search for an appropriate LVQ variant in case of a given specific classification problem as well as a reference to recently developed variants and improvements of the basic GLVQ scheme.

Prototype-based models in machine learning

An overview is given of prototype-based models in machine learning. In this framework, observations, i.e., data, are stored in terms of typical representatives. Together with a suitable measure of similarity, the systems can be employed in the context of unsupervised and supervised analysis of potentially high-dimensional, complex datasets. We discuss basic schemes of competitive vector quantization as well as the so-called neural gas approach and Kohonen's topology-preserving self-organizing map. Supervised learning in prototype systems is exemplified in terms of learning vector quantization. Most frequently, the familiar Euclidean distance serves as a dissimilarity measure. We present extensions of the framework to nonstandard measures and give an introduction to the use of adaptive distances in relevance learning.