Model Behavior Preserving for Class-Incremental Learning

Deep models have shown to be vulnerable to catastrophic forgetting, a phenomenon that the recognition performance on old data degrades when a pre-trained model is fine-tuned on new data. Knowledge distillation (KD) is a popular incremental approach to alleviate catastrophic forgetting. However, it usually fixes the absolute values of neural responses for isolated historical instances, without considering the intrinsic structure of the responses by a convolutional neural network (CNN) model. To overcome this limitation, we recognize the importance of the global property of the whole instance set and treat it as a behavior characteristic of a CNN model relevant to model incremental learning. On this basis: 1) we design an instance neighborhood-preserving (INP) loss to maintain the order of pair-wise instance similarities of the old model in the feature space; 2) we devise a label priority-preserving (LPP) loss to preserve the label ranking lists within instance-wise label probability vectors in the output space; and 3) we introduce an efficient derivable ranking algorithm for calculating the two loss functions. Extensive experiments conducted on CIFAR100 and ImageNet show that our approach achieves the state-of-the-art performance.

A lightweight intrusion detection method for IoT based on deep learning and dynamic quantization

Intrusion detection ensures that IoT can protect itself against malicious intrusions in extensive and intricate network traffic data. In recent years, deep learning has been extensively and effectively employed in IoT intrusion detection. However, the limited computing power and storage space of IoT devices restrict the feasibility of deploying resource-intensive intrusion detection systems on them. This article introduces the DL-BiLSTM lightweight IoT intrusion detection model. By combining deep neural networks (DNNs) and bidirectional long short-term memory networks (BiLSTMs), the model enables nonlinear and bidirectional long-distance feature extraction of complex network information. This capability allows the system to capture complex patterns and behaviors related to cyber-attacks, thus enhancing detection performance. To address the resource constraints of IoT devices, the model utilizes the incremental principal component analysis (IPCA) algorithm for feature dimensionality reduction. Additionally, dynamic quantization is employed to trim the specified cell structure of the model, thereby reducing the computational burden on IoT devices while preserving accurate detection capability. The experimental results on the benchmark datasets CIC IDS2017, N-BaIoT, and CICIoT2023 demonstrate that DL-BiLSTM surpasses traditional deep learning models and cutting-edge detection techniques in terms of detection performance, while maintaining a lower model complexity.

An Approach for Streaming Data Feature Extraction Based on Discrete Cosine Transform and Particle Swarm Optimization

Incremental feature extraction algorithms are designed to analyze large-scale data streams. Many of them suffer from high computational cost, time complexity, and data dependency, which adversely affects the processing of the data stream. With this motivation, this paper presents a novel incremental feature extraction approach based on the Discrete Cosine Transform (DCT) for the data stream. The proposed approach is separated into initial and sequential phases, and each phase uses a fixed-size windowing technique for processing the current samples. The initial phase is performed only on the first window to construct the initial model as a baseline. In this phase, normalization and DCT are applied to each sample in the window. Subsequently, the efficient feature subset is determined by a particle swarm optimization-based method. With the construction of the initial model, the sequential phase begins. The normalization and DCT processes are likewise applied to each sample. Afterward, the feature subset is selected according to the initial model. Finally, the k-nearest neighbor classifier is employed for classification. The approach is tested on the well-known streaming data sets and compared with state-of-the-art incremental feature extraction algorithms. The experimental studies demonstrate the proposed approach’s success in terms of recognition accuracy and learning time.

Streaming chunk incremental learning for class-wise data stream classification with fast learning speed and low structural complexity

Due to the fast speed of data generation and collection from advanced equipment, the amount of data obviously overflows the limit of available memory space and causes difficulties achieving high learning accuracy. Several methods based on discard-after-learn concept have been proposed. Some methods were designed to cope with a single incoming datum but some were designed for a chunk of incoming data. Although the results of these approaches are rather impressive, most of them are based on temporally adding more neurons to learn new incoming data without any neuron merging process which can obviously increase the computational time and space complexities. Only online versatile elliptic basis function (VEBF) introduced neuron merging to reduce the space-time complexity of learning only a single incoming datum. This paper proposed a method for further enhancing the capability of discard-after-learn concept for streaming data-chunk environment in terms of low computational time and neural space complexities. A set of recursive functions for computing the relevant parameters of a new neuron, based on statistical confidence interval, was introduced. The newly proposed method, named streaming chunk incremental learning (SCIL), increases the plasticity and the adaptabilty of the network structure according to the distribution of incoming data and their classes. When being compared to the others in incremental-like manner, based on 11 benchmarked data sets of 150 to 581,012 samples with attributes ranging from 4 to 1,558 formed as streaming data, the proposed SCIL gave better accuracy and time in most data sets.

Semisupervised Feature Selection Based on Relevance and Redundancy Criteria

Feature selection aims to gain relevant features for improved classification performance and remove redundant features for reduced computational cost. How to balance these two factors is a problem especially when the categorical labels are costly to obtain. In this paper, we address this problem using semisupervised learning method and propose a max-relevance and min-redundancy criterion based on Pearson's correlation (RRPC) coefficient. This new method uses the incremental search technique to select optimal feature subsets. The new selected features have strong relevance to the labels in supervised manner, and avoid redundancy to the selected feature subsets under unsupervised constraints. Comparative studies are performed on binary data and multicategory data from benchmark data sets. The results show that the RRPC can achieve a good balance between relevance and redundancy in semisupervised feature selection. We also compare the RRPC with classic supervised feature selection criteria (such as mRMR and Fisher score), unsupervised feature selection criteria (such as Laplacian score), and semisupervised feature selection criteria (such as sSelect and locality sensitive). Experimental results demonstrate the effectiveness of our method.

A Self-Organizing Incremental Neural Network based on local distribution learning

In this paper, we propose an unsupervised incremental learning neural network based on local distribution learning, which is called Local Distribution Self-Organizing Incremental Neural Network (LD-SOINN). The LD-SOINN combines the advantages of incremental learning and matrix learning. It can automatically discover suitable nodes to fit the learning data in an incremental way without a priori knowledge such as the structure of the network. The nodes of the network store rich local information regarding the learning data. The adaptive vigilance parameter guarantees that LD-SOINN is able to add new nodes for new knowledge automatically and the number of nodes will not grow unlimitedly. While the learning process continues, nodes that are close to each other and have similar principal components are merged to obtain a concise local representation, which we call a relaxation data representation. A denoising process based on density is designed to reduce the influence of noise. Experiments show that the LD-SOINN performs well on both artificial and real-word data.

Iterative Refinement of Possibility Distributions by Learning for Pixel-Based Classification

This paper proposes an approach referred as: iterative refinement of possibility distributions by learning (IRPDL) for pixel-based image classification. The IRPDL approach is based on the use of possibilistic reasoning concepts exploiting expert knowledge sources as well as ground possibilistic seeds learning. The set of seeds is constructed by incrementally updating and refining the possibility distributions. Synthetic images as well as real images from the RIDER Breast MRI database are being used to evaluate the IRPDL performance. Its performance is compared with three relevant reference methods: region growing, semi-supervised fuzzy pattern matching, and Markov random fields. The IRDPL performance (in terms of recognition rate, 87.3%) is close to the Markovian method (88.8%) that is considered to be the reference in pixel-based image classification. IRPDL outperforms the other two methods, respectively, at the recognition rates of 83.9% and 84.7%. In addition, the proposed IRPDL requires fewer parameters for the mathematical representation and presents a reduced computational complexity.

Online Sequential Projection Vector Machine with Adaptive Data Mean Update

We propose a simple online learning algorithm especial for high-dimensional data. The algorithm is referred to as online sequential projection vector machine (OSPVM) which derives from projection vector machine and can learn from data in one-by-one or chunk-by-chunk mode. In OSPVM, data centering, dimension reduction, and neural network training are integrated seamlessly. In particular, the model parameters including (1) the projection vectors for dimension reduction, (2) the input weights, biases, and output weights, and (3) the number of hidden nodes can be updated simultaneously. Moreover, only one parameter, the number of hidden nodes, needs to be determined manually, and this makes it easy for use in real applications. Performance comparison was made on various high-dimensional classification problems for OSPVM against other fast online algorithms including budgeted stochastic gradient descent (BSGD) approach, adaptive multihyperplane machine (AMM), primal estimated subgradient solver (Pegasos), online sequential extreme learning machine (OSELM), and SVD + OSELM (feature selection based on SVD is performed before OSELM). The results obtained demonstrated the superior generalization performance and efficiency of the OSPVM.

Smart Colonography for Distributed Medical Databases with Group Kernel Feature Analysis

Computer-Aided Detection (CAD) of polyps in Computed Tomographic (CT) colonography is currently very limited since a single database at each hospital/institution doesn't provide sufficient data for training the CAD system's classification algorithm. To address this limitation, we propose to use multiple databases, (e.g., big data studies) to create multiple institution-wide databases using distributed computing technologies, which we call smart colonography. Smart colonography may be built by a larger colonography database networked through the participation of multiple institutions via distributed computing. The motivation herein is to create a distributed database that increases the detection accuracy of CAD diagnosis by covering many true-positive cases. Colonography data analysis is mutually accessible to increase the availability of resources so that the knowledge of radiologists is enhanced. In this article, we propose a scalable and efficient algorithm called Group Kernel Feature Analysis (GKFA), which can be applied to multiple cancer databases so that the overall performance of CAD is improved. The key idea behind the proposed GKFA method is to allow the feature space to be updated as the training proceeds with more data being fed from other institutions into the algorithm. Experimental results show that GKFA achieves very good classification accuracy.

Kernel association for classification and prediction: a survey

Kernel association (KA) in statistical pattern recognition used for classification and prediction have recently emerged in a machine learning and signal processing context. This survey outlines the latest trends and innovations of a kernel framework for big data analysis. KA topics include offline learning, distributed database, online learning, and its prediction. The structural presentation and the comprehensive list of references are geared to provide a useful overview of this evolving field for both specialists and relevant scholars.

Online motor fault detection and diagnosis using a hybrid FMM-CART model

In this brief, a hybrid model combining the fuzzy min-max (FMM) neural network and the classification and regression tree (CART) for online motor detection and diagnosis tasks is described. The hybrid model, known as FMM-CART, exploits the advantages of both FMM and CART for undertaking data classification and rule extraction problems. To evaluate the applicability of the proposed FMM-CART model, an evaluation with a benchmark data set pertaining to electrical motor bearing faults is first conducted. The results obtained are equivalent to those reported in the literature. Then, a laboratory experiment for detecting and diagnosing eccentricity faults in an induction motor is performed. In addition to producing accurate results, useful rules in the form of a decision tree are extracted to provide explanation and justification for the predictions from FMM-CART. The experimental outcome positively shows the potential of FMM-CART in undertaking online motor fault detection and diagnosis tasks.

Efficient online subspace learning with an indefinite kernel for visual tracking and recognition

We propose an exact framework for online learning with a family of indefinite (not positive) kernels. As we study the case of nonpositive kernels, we first show how to extend kernel principal component analysis (KPCA) from a reproducing kernel Hilbert space to Krein space. We then formulate an incremental KPCA in Krein space that does not require the calculation of preimages and therefore is both efficient and exact. Our approach has been motivated by the application of visual tracking for which we wish to employ a robust gradient-based kernel. We use the proposed nonlinear appearance model learned online via KPCA in Krein space for visual tracking in many popular and difficult tracking scenarios. We also show applications of our kernel framework for the problem of face recognition.

A SURVEY OF VISION-BASED ARCHITECTURES FOR ROBOT LEARNING BY IMITATION

Learning by imitation is a natural and intuitive way to teach social robots new behaviors. While these learning systems can use different sensory inputs, vision is often their main or even their only source of input data. However, while many vision-based robot learning by imitation (RLbI) architectures have been proposed in the last decade, they may be difficult to compare due to the absence of a common, structured description. The first contribution of this survey is the definition of a set of standard components that can be used to describe any RLbI architecture. Once these components have been defined, the second contribution of the survey is an analysis of how different vision-based architectures implement and connect them. This bottom–up, structural analysis of architectures allows to compare different solutions, highlighting their main advantages and drawbacks, from a more flexible perspective than the comparison of monolithic systems.

Comparison between analog and digital neural network implementations for range-finding applications

A neural network (NN) was developed in order to increase the distance range of a phase-shift laser range finder and to achieve surface recognition, by using two photoelectrical signals issued from the measurement system. The NN architecture consists of a multilayer perceptron (MLP) with two inputs, three neurons in the hidden layer, and one output. Depending on the application, the NN output has to resolve the ambiguity due to phase-shift measurement by linearizing the inverse of the square law, or to indicate an output voltage corresponding to the tested surface. This embedded system dedicated to optoelectronic measurements was successfully tested with an analog NN, implemented in 0.35- microm complimentary metal-oxide-semiconductor (CMOS) technology, resulting in a threefold increase in the distance range with respect to the one limited by the phase-shift measurement, and by discriminating four types of surfaces (a plastic surface, glossy paper, a painted wall, and a porous surface), at a remote distance between the range finder and the target varying from 0.5 m up to 1.25 m and with a laser beam angle varying between -pi/6 and pi/6 with respect to the target. In this type of application, NN analog implementation provides many advantages, notably use of a small silicon area, low power consumption and no analog-to-digital conversions (ADCs). Nevertheless, digital implementation allows ease of conception and reconfigurability and an embedded weight and bias update. This paper presents the complete measurement system and a comparison between both types of implementation, by developing the advantages and drawbacks relative to each method. An optimized mixed architecture, using both techniques, is then proposed and discussed at the end of the paper.