An incremental network for on-line unsupervised classification and topology learning

A Fast Multi-Scale of Distributed Batch-Learning Growing Neural Gas for Multi-Camera 3D Environmental Map Building

Biologically inspired intelligent methods have been applied to various sensing systems in order to extract features from a huge size of raw sensing data. For example, point cloud data can be applied to human activity recognition, multi-person tracking, and suspicious person detection, but a single RGB-D camera is not enough to perform the above tasks. Therefore, this study propose a 3D environmental map-building method integrating point cloud data measured via multiple RGB-D cameras. First, a fast multi-scale of distributed batch-learning growing neural gas (Fast MS-DBL-GNG) is proposed as a topological feature extraction method in order to reduce computational costs because a single RGB-D camera may output 1 million data. Next, random sample consensus (RANSAC) is applied to integrate two sets of point cloud data using topological features. In order to show the effectiveness of the proposed method, Fast MS-DBL-GNG is applied to perform topological mapping from several point cloud data sets measured in different directions with some overlapping areas included in two images. The experimental results show that the proposed method can extract topological features enough to integrate point cloud data sets, and it runs 14 times faster than the previous GNG method with a 23% reduction in the quantization error. Finally, this paper discuss the advantage and disadvantage of the proposed method through numerical comparison with other methods, and explain future works to improve the proposed method.

Automated Design of Multipass Heuristics for Resource-Constrained Job Scheduling With Self-Competitive Genetic Programming

Resource constraint job scheduling is an important combinatorial optimization problem with many practical applications. This problem aims at determining a schedule for executing jobs on machines satisfying several constraints (e.g., precedence and resource constraints) given a shared central resource while minimizing the tardiness of the jobs. Due to the complexity of the problem, several exact, heuristic, and hybrid methods have been attempted. Despite their success, scalability is still a major issue of the existing methods. In this study, we develop a new genetic programming algorithm for resource constraint job scheduling to overcome or alleviate the scalability issue. The goal of the proposed algorithm is to evolve effective and efficient multipass heuristics by a surrogate-assisted learning mechanism and self-competitive genetic operations. The experiments show that the evolved multipass heuristics are very effective when tested with a large dataset. Moreover, the algorithm scales very well as excellent solutions are found for even the largest problem instances, outperforming existing metaheuristic and hybrid methods.

Research Review for Broad Learning System: Algorithms, Theory, and Applications

In recent years, the appearance of the broad learning system (BLS) is poised to revolutionize conventional artificial intelligence methods. It represents a step toward building more efficient and effective machine-learning methods that can be extended to a broader range of necessary research fields. In this survey, we provide a comprehensive overview of the BLS in data mining and neural networks for the first time, focusing on summarizing various BLS methods from the aspects of its algorithms, theories, applications, and future open research questions. First, we introduce the basic pattern of BLS manifestation, the universal approximation capability, and essence from the theoretical perspective. Furthermore, we focus on BLS's various improvements based on the current state of the theoretical research, which further improves its flexibility, stability, and accuracy under general or specific conditions, including classification, regression, semisupervised, and unsupervised tasks. Due to its remarkable efficiency, impressive generalization performance, and easy extendibility, BLS has been applied in different domains. Next, we illustrate BLS's practical advances, such as computer vision, biomedical engineering, control, and natural language processing. Finally, the future open research problems and promising directions for BLSs are pointed out.

Non-Uniform Input-Based Adaptive Growing Neural Gas for Unstructured Environment Map Construction

The research and development of special robots such as excavation robots is an important way to achieve safe and efficient production in coal mines. Affected by the unstructured environment such as complex working conditions and unsteady factor disturbances, the real-time construction of section environment maps that can accurately describe the environment and facilitate trajectory planning and decision making has become a key scientific problem to be solved as soon as possible. Therefore, non-uniform input based adaptive growing neural gas for unstructured environment map construction has been proposed. Considering complex load identification, real-time location identification, and the types of unsteady disturbance factors and working conditions, a set of environment identification models has been established based on a large amount of underground measured data training. These models can express whether the section environment has changed, as well as the type and magnitude of the change, to realize the overall knowledge extraction and parametric representation of the unstructured environment. Then, in order to solve the problems of inaccurate topology, excessive aging of connecting edges, and excessive deletion of nodes in non-uniform input environment, an adaptive growing neural gas algorithm based on non-uniform input environment (AGNG-NU) is proposed. Featured by a dynamic response deletion mechanism and adaptive adjustment mechanism of neuron parameters, the generated nodes and their topology can be dynamically adjusted according to the density of regional sample points. Several sets of non-uniform input environments are set to test the algorithm. The experimental results show that the topological maps established by AGNG-NU express clearer environmental information and, at the same time, the accuracy and distribution are improved by 8% and 15%, respectively, compared with the basic GNG algorithm. The accuracy and the distribution have also been significantly improved compared with other common SOM and GCS algorithms.

Prototype Regularized Manifold Regularization Technique for Semi-Supervised Online Extreme Learning Machine

Data streaming applications such as the Internet of Things (IoT) require processing or predicting from sequential data from various sensors. However, most of the data are unlabeled, making applying fully supervised learning algorithms impossible. The online manifold regularization approach allows sequential learning from partially labeled data, which is useful for sequential learning in environments with scarcely labeled data. Unfortunately, the manifold regularization technique does not work out of the box as it requires determining the radial basis function (RBF) kernel width parameter. The RBF kernel width parameter directly impacts the performance as it is used to inform the model to which class each piece of data most likely belongs. The width parameter is often determined off-line via hyperparameter search, where a vast amount of labeled data is required. Therefore, it limits its utility in applications where it is difficult to collect a great deal of labeled data, such as data stream mining. To address this issue, we proposed eliminating the RBF kernel from the manifold regularization technique altogether by combining the manifold regularization technique with a prototype learning method, which uses a finite set of prototypes to approximate the entire data set. Compared to other manifold regularization approaches, this approach instead queries the prototype-based learner to find the most similar samples for each sample instead of relying on the RBF kernel. Thus, it no longer necessitates the RBF kernel, which improves its practicality. The proposed approach can learn faster and achieve a higher classification performance than other manifold regularization techniques based on experiments on benchmark data sets. Results showed that the proposed approach can perform well even without using the RBF kernel, which improves the practicality of manifold regularization techniques for semi-supervised learning.

SAGMAD—A Signature Agnostic Malware Detection System Based on Binary Visualisation and Fuzzy Sets

Image conversion of byte-level data, or binary visualisation, is a relevant approach to security applications interested in malicious activity detection. However, in practice, binary visualisation has always been seen to have great limitations when dealing with large volumes of data, and would be a reluctant candidate as the core building block of an intrusion detection system (IDS). This is due to the requirements of computational time when processing the flow of byte data into image format. Machine intelligence solutions based on colour tone variations that are intended for pattern recognition would overtax the process. In this paper, we aim to solve this issue by proposing a fast binary visualisation method that uses Fuzzy Set theory and the H-indexing space filling curve. Our model can assign different colour tones on a byte, allowing it to be influenced by neighbouring byte values while preserving optimal locality indexing. With this work, we wish to establish the first steps in pursuit of a signature-free IDS. For our experiment, we used 5000 malicious and benign files of different sizes. Our methodology was tested on various platforms, including GRNET’s High-Performance Computing services. Further improvements in computation time allowed larger files to convert in roughly 0.5 s on a desktop environment. Its performance was also compared with existing machine learning-based detection applications that used traditional binary visualisation. Despite lack of optimal tuning, SAGMAD was able to achieve 91.94% accuracy, 90.63% precision, 92.7% recall, and an F-score of 91.61% on average when tested within previous binary visualisation applications and following their parameterisation scheme. The results exceeded malware file-based experiments and were similar to network intrusion applications. Overall, the results demonstrated here prove our method to be a promising mechanism for a fast AI-based signature-agnostic IDS.

Pruning Growing Self-Organizing Map Network for Human Physical Activity Identification

Human physical activity identification based on wearable sensors is of great significance to human health analysis. A large number of machine learning models have been applied to human physical activity identification and achieved remarkable results. However, most human physical activity identification models can only be trained based on labeled data, and it is difficult to obtain enough labeled data, which leads to weak generalization ability of the model. A Pruning Growing SOM model is proposed in this paper to address the limitations of small-scale labeled dataset, which is unsupervised in the training stage, and then only a small amount of labeled data is used for labeling neurons to reduce dependency on labeled data. In training stage, the inactive neurons in network can be deleted by pruning mechanism, which makes the model more consistent with the data distribution and improves the identification accuracy even on unbalanced dataset, especially for the action categories with poor identification effect. In addition, the pruning mechanism can also speed up the inference of the model by controlling its scale.

Multilayer Batch Learning Growing Neural Gas for Learning Multiscale Topologies

Hierarchical topological structure learning methods are expected to be developed in the field of data mining for extracting multiscale topological structures from an unknown dataset. However, most methods require user-defined parameters, and it is difficult for users to determine these parameters and effectively utilize the method. In this paper, we propose a new parameter-less hierarchical topological structure learning method based on growing neural gas (GNG). First, we propose batch learning GNG (BL-GNG) to improve the learning convergence and reduce the user-designed parameters in GNG. BL-GNG uses an objective function based on fuzzy C-means to improve the learning convergence. Next, we propose multilayer BL-GNG (MBL-GNG), which is a parameter-less unsupervised learning algorithm based on hierarchical topological structure learning. In MBL-GNG, the input data of each layer uses parent nodes to learn more abstract topological structures from the dataset. Furthermore, MBL-GNG can automatically determine the number of nodes and layers according to the data distribution. Finally, we conducted several experiments to evaluate our proposed method by comparing it with other hierarchical approaches and discuss the effectiveness of our proposed method.

A Continuous Learning Approach for Real-Time Network Intrusion Detection

Network intrusion detection is becoming a challenging task with cyberattacks that are becoming more and more sophisticated. Failing the prevention or detection of such intrusions might have serious consequences. Machine learning approaches try to recognize network connection patterns to classify unseen and known intrusions but also require periodic re-training to keep the performances at a high level. In this paper, a novel continuous learning intrusion detection system, called Soft-Forgetting Self-Organizing Incremental Neural Network (SF-SOINN), is introduced. SF-SOINN, besides providing continuous learning capabilities, is able to perform fast classification, is robust to noise, and it obtains good performances with respect to the existing approaches. The main characteristic of SF-SOINN is the ability to remove nodes from the neural network based on their utility estimate. SF-SOINN has been validated on the well-known NSL-KDD and CIC-IDS-2017 intrusion detection datasets as well as on some artificial data to show the classification capability on more general tasks.

Continuous detection of concept drift in industrial cyber-physical systems using closed loop incremental machine learning

The embedded, computational and cloud elements of industrial cyber physical systems (CPS) generate large volumes of data at high velocity to support the operations and functions of corresponding time-critical and mission-critical physical entities. Given the non-deterministic nature of these entities, the generated data streams are susceptible to dynamic and abrupt changes. Such changes, which are formally defined as concept drifts, leads to a decline in the accuracy and robustness of predicted CPS behaviors. Most existing work in concept drift detection are classifier dependent and require labeled data. However, CPS data streams are unlabeled, unstructured and change over time. In this paper, we propose an unsupervised machine learning algorithm for continuous concept drift detection in industrial CPS. This algorithm demonstrates three types of unsupervised learning, online, incremental and decremental. Furthermore, it distinguishes between abrupt and reoccurring drifts. We conducted experiments on SEA, a widely cited synthetic dataset of concept drift detection, and two industrial applications of CPS, task tracking in factory settings and smart energy consumption. The results of these experiments successfully validate the key features of the proposed algorithm and its utility of detecting change in non-deterministic CPS environments.

Evaluating the impacts of major cyclonic catastrophes in coastal Bangladesh using geospatial techniques

Cyclonic catastrophes frequently devastate coastal regions of Bangladesh that host around 35 million people which represents two-thirds of the total population. They have caused many problems like agricultural crop loss, forest degradation, damage to built-up areas, river and shoreline changes that are linked to people’s livelihood and ecological biodiversity. There is an absence of a comprehensive assessment of the major cyclonic disasters of Bangladesh that integrates geospatial technologies in a single study. This study aims to integrate geospatial technologies with major disasters and compares them, which has not been tried before. This paper tried to identify impacts that occurred in the coastal region by major catastrophic events at a vast level using different geospatial technologies. It focuses to identify the impacts of major catastrophic events on livelihood and food production as well as compare the impacts and intensity of different disasters. Furthermore, it compared the losses among several districts and for that previous and post-satellite images of disasters that occurred in 1988, 1991, 2007, 2009, 2019 were used. Classification technique like machine learning algorithm was done in pre- to post-disaster images. For quantifying change in the indication of different factors, indices including NDVI, NDWI, NDBI were developed. “Change vector analysis” equation was performed in bands of the images of pre- and post-disaster to identify the magnitude of change. Also, crop production variance was analyzed to detect impacts on crop production. Furthermore, the changes in shallow to deep water were analyzed. There is a notable change in shallow to deep water bodies after each disaster in Satkhira and Bhola district but subtle changes in Khulna and Bagerhat districts. Change vector analysis revealed greater intensity in Bhola in 1988 and Satkhira in 1991. Furthermore, over the years 2007 and 2009 it showed medium and deep intense areas all over the region. A sharp decrease in Aus rice production is witnessed in Barishal in 2007 when cyclone “Sidr” was stricken. The declination of potato production is seen in Khulna district after the 1988 cyclone. A huge change in the land-use classes from classified images like water body, Pasture land in 1988 and water body, forest in 1991 is marked out. Besides, a clear variation in the settlement was observed from the classified images. This study explores the necessity of using more geospatial technologies in disastrous impacts assessment around the world in the context of Bangladesh and, also, emphasizes taking effective, proper and sustainable disaster management and mitigation measures to counter future disastrous impacts.

Artificial Evolution Network: A Computational Perspective on the Expansibility of the Nervous System

Neurobiologists recently found the brain can use sudden emerged channels to process information. Based on this finding, we put forward a question whether we can build a computation model that is able to integrate a sudden emerged new type of perceptual channel into itself in an online way. If such a computation model can be established, it will introduce a channel-free property to the computation model and meanwhile deepen our understanding about the extendibility of the brain. In this article, a biologically inspired neural network named artificial evolution (AE) network is proposed to handle the problem. When a new perceptual channel emerges, the neurons in the network can grow new connections to connect the emerged channel according to the Hebb rule. In this article, we design a sensory channel expansion experiment to test the AE network. The experimental results demonstrate that the AE network can handle the sudden emerged perceptual channels effectively.

People-Centric Evolutionary System for Dynamic Production Scheduling

Evolving production scheduling heuristics is a challenging task because of the dynamic and complex production environments and the interdependency of multiple scheduling decisions. Different genetic programming (GP) methods have been developed for this task and achieved very encouraging results. However, these methods usually have trouble in discovering powerful and compact heuristics, especially for difficult problems. Moreover, there is no systematic approach for the decision makers to intervene and embed their knowledge and preferences in the evolutionary process. This article develops a novel people-centric evolutionary system for dynamic production scheduling. The two key components of the system are a new mapping technique to incrementally monitor the evolutionary process and a new adaptive surrogate model to improve the efficiency of GP. The experimental results with dynamic flexible job shop scheduling show that the proposed system outperforms the existing algorithms for evolving scheduling heuristics in terms of scheduling performance and heuristic sizes. The new system also allows the decision makers to interact on the fly and guide the evolution toward the desired solutions.

Online Topology Learning by a Gaussian Membership-Based Self-Organizing Incremental Neural Network

In order to extract useful information from data streams, incremental learning has been introduced in more and more data mining algorithms. For instance, a self-organizing incremental neural network (SOINN) has been proposed to extract a topological structure that consists of one or more neural networks to closely reflect the data distribution of data streams. However, SOINN has the tradeoff between deleting previously learned nodes and inserting new nodes, i.e., the stability-plasticity dilemma. Therefore, it is not guaranteed that the topological structure obtained by the SOINN will closely represent data distribution. For solving the stability-plasticity dilemma, we propose a Gaussian membership-based SOINN (Gm-SOINN). Unlike other SOINN-based methods that allow only one node to be identified as a "winner" (the nearest node), the Gm-SOINN uses a Gaussian membership to indicate to which degree the node is a winner. Hence, the Gm-SOINN avoids the topological structure that cannot represent the data distribution because previously learned nodes overly deleted or noisy nodes inserted. In addition, an evolving Gaussian mixture model is integrated into the Gm-SOINN to estimate the density distribution of nodes, thereby avoiding the wrong connection between two nodes. Experiments involving both artificial and real-world data sets indicate that our proposed Gm-SOINN achieves better performance than other topology learning methods.

Distributed dual vigilance fuzzy adaptive resonance theory learns online, retrieves arbitrarily-shaped clusters, and mitigates order dependence

This paper presents a novel adaptive resonance theory (ART)-based modular architecture for unsupervised learning, namely the distributed dual vigilance fuzzy ART (DDVFA). DDVFA consists of a global ART system whose nodes are local fuzzy ART modules. It is equipped with distributed higher-order activation and match functions and a dual vigilance mechanism. Together, these allow DDVFA to perform unsupervised modularization, create multi-prototype cluster representations, retrieve arbitrarily-shaped clusters, and reduce category proliferation. Another important contribution is the reduction of order-dependence, an issue that affects any agglomerative clustering method. This paper demonstrates two approaches for mitigating order-dependence: pre-processing using visual assessment of cluster tendency (VAT) or post-processing using a novel Merge ART module. The former is suitable for batch processing, whereas the latter also works for online learning. Experimental results in online mode carried out on 30 benchmark data sets show that DDVFA cascaded with Merge ART statistically outperformed the best other ART-based systems when samples were randomly presented. Conversely, they were found to be statistically equivalent in offline mode when samples were pre-processed using VAT. Remarkably, performance comparisons to non-ART-based clustering algorithms show that DDVFA (which learns incrementally) was also statistically equivalent to the non-incremental (offline) methods of density-based spatial clustering of applications with noise (DBSCAN), single linkage hierarchical agglomerative clustering (SL-HAC), and k-means, while retaining the appealing properties of ART. Links to the source code and data are provided. Considering the algorithm's simplicity, online learning capability, and performance, it is an ideal choice for many agglomerative clustering applications.

A Kernel Bayesian Adaptive Resonance Theory with A Topological Structure

This paper attempts to solve the typical problems of self-organizing growing network models, i.e. (a) an influence of the order of input data on the self-organizing ability, (b) an instability to high-dimensional data and an excessive sensitivity to noise, and (c) an expensive computational cost by integrating Kernel Bayes Rule (KBR) and Correntropy-Induced Metric (CIM) into Adaptive Resonance Theory (ART) framework. KBR performs a covariance-free Bayesian computation which is able to maintain a fast and stable computation. CIM is a generalized similarity measurement which can maintain a high-noise reduction ability even in a high-dimensional space. In addition, a Growing Neural Gas (GNG)-based topology construction process is integrated into the ART framework to enhance its self-organizing ability. The simulation experiments with synthetic and real-world datasets show that the proposed model has an outstanding stable self-organizing ability for various test environments.

Perception Coordination Network: A Neuro Framework for Multimodal Concept Acquisition and Binding

To simulate the concept acquisition and binding of different senses in the brain, a biologically inspired neural network model named perception coordination network (PCN) is proposed. It is a hierarchical structure, which is functionally divided into the primary sensory area (PSA), the primary sensory association area (SAA), and the higher order association area (HAA). The PSA contains feature neurons which respond to many elementary features, e.g., colors, shapes, syllables, and basic flavors. The SAA contains primary concept neurons which combine the elementary features in the PSA to represent unimodal concept of objects, e.g., the image of an apple, the Chinese word "[píng guǒ]" which names the apple, and the taste of the apple. The HAA contains associated neurons which connect the primary concept neurons of several PSA, e.g., connects the image, the taste, and the name of an apple. It means that the associated neurons have a multimodal response mode. Therefore, this area executes multisensory integration. PCN is an online incremental learning system, it is able to continuously acquire and bind multimodality concepts in an online way. The experimental results suggest that PCN is able to handle the multimodal concept acquisition and binding effectively.

A density-based competitive data stream clustering network with self-adaptive distance metric

Data stream clustering is a branch of clustering where patterns are processed as an ordered sequence. In this paper, we propose an unsupervised learning neural network named Density Based Self Organizing Incremental Neural Network(DenSOINN) for data stream clustering tasks. DenSOINN is a self organizing competitive network that grows incrementally to learn suitable nodes to fit the distribution of learning data, combining online unsupervised learning and topology learning by means of competitive Hebbian learning rule. By adopting a density-based clustering mechanism, DenSOINN discovers arbitrarily shaped clusters and diminishes the negative effect of noise. In addition, we adopt a self-adaptive distance framework to obtain good performance for learning unnormalized input data. Experiments show that the DenSOINN can achieve high standard performance comparing to state-of-the-art methods.

Simultaneous Forecasting of Meteorological Data Based on a Self-Organizing Incremental Neural Network

In this study, we propose a simultaneous forecasting model for meteorological time-series data based on a self-organizing incremental neural network (SOINN). Meteorological parameters (i.e., temperature, wet bulb temperature, humidity, wind speed, atmospheric pressure, and total solar radiation on a horizontal surface) are considered as input data for the prediction of meteorological time-series information. Based on a SOINN within normalized-refined-meteorological data, proposed model succeeded forecasting temperature, humidity, wind speed and atmospheric pressure simultaneously. In addition, proposed model does not take more than 2 s in training half-year period and 15 s in testing half-year period. This paper also elucidates the SOINN and the algorithm of the learning process. The effectiveness of our model is established by comparison of our results with experimental results and with results obtained by another model. Three advantages of our model are also described. The obtained information can be effective in applications based on neural networks, and the proposed model for handling meteorological phenomena may be helpful for other studies worldwide including energy management system.

Time Series Prediction of Tropical Storm Trajectory Using Self-Organizing Incremental Neural Networks and Error Evaluation

This study proposes a route prediction method using a self-organizing incremental neural network. The route trajectory is predicted from two location parameters (the latitude and longitude of the middle of a tropical storm) and the meteorological information (the atmospheric pressure). The method accurately predicted the normalized atmospheric pressure data of East Asia in the topological space of latitude and longitude, with low calculation cost. This paper explains the algorithms for training the self-organizing incremental neural network, the procedure for refining the datasets and the method for predicting the storm trajectory. The effectiveness of the proposed method was confirmed in experiments. With the results of experiments, possibility of prediction model improvement is discussed. Additionally, this paper explains the limitations of proposed method and brief solution to resolve. Although the proposed method was applied only to typhoon phenomena in the present study, it is potentially applicable to a wide range of global problems.

On the Accuracy and Parallelism of GPGPU-Powered Incremental Clustering Algorithms

Incremental clustering algorithms play a vital role in various applications such as massive data analysis and real-time data processing. Typical application scenarios of incremental clustering raise high demand on computing power of the hardware platform. Parallel computing is a common solution to meet this demand. Moreover, General Purpose Graphic Processing Unit (GPGPU) is a promising parallel computing device. Nevertheless, the incremental clustering algorithm is facing a dilemma between clustering accuracy and parallelism when they are powered by GPGPU. We formally analyzed the cause of this dilemma. First, we formalized concepts relevant to incremental clustering like evolving granularity. Second, we formally proved two theorems. The first theorem proves the relation between clustering accuracy and evolving granularity. Additionally, this theorem analyzes the upper and lower bounds of different-to-same mis-affiliation. Fewer occurrences of such mis-affiliation mean higher accuracy. The second theorem reveals the relation between parallelism and evolving granularity. Smaller work-depth means superior parallelism. Through the proofs, we conclude that accuracy of an incremental clustering algorithm is negatively related to evolving granularity while parallelism is positively related to the granularity. Thus the contradictory relations cause the dilemma. Finally, we validated the relations through a demo algorithm. Experiment results verified theoretical conclusions.

Nonparametric Density Estimation Based on Self-Organizing Incremental Neural Network for Large Noisy Data

With the ongoing development and expansion of communication networks and sensors, massive amounts of data are continuously generated in real time from real environments. Beforehand, prediction of a distribution underlying such data is difficult; furthermore, the data include substantial amounts of noise. These factors make it difficult to estimate probability densities. To handle these issues and massive amounts of data, we propose a nonparametric density estimator that rapidly learns data online and has high robustness. Our approach is an extension of both kernel density estimation (KDE) and a self-organizing incremental neural network (SOINN); therefore, we call our approach KDESOINN. An SOINN provides a clustering method that learns about the given data as networks of prototype of data; more specifically, an SOINN can learn the distribution underlying the given data. Using this information, KDESOINN estimates the probability density function. The results of our experiments show that KDESOINN outperforms or achieves performance comparable to the current state-of-the-art approaches in terms of robustness, learning time, and accuracy.

A Self-Organizing Incremental Neural Network based on local distribution learning

In this paper, we propose an unsupervised incremental learning neural network based on local distribution learning, which is called Local Distribution Self-Organizing Incremental Neural Network (LD-SOINN). The LD-SOINN combines the advantages of incremental learning and matrix learning. It can automatically discover suitable nodes to fit the learning data in an incremental way without a priori knowledge such as the structure of the network. The nodes of the network store rich local information regarding the learning data. The adaptive vigilance parameter guarantees that LD-SOINN is able to add new nodes for new knowledge automatically and the number of nodes will not grow unlimitedly. While the learning process continues, nodes that are close to each other and have similar principal components are merged to obtain a concise local representation, which we call a relaxation data representation. A denoising process based on density is designed to reduce the influence of noise. Experiments show that the LD-SOINN performs well on both artificial and real-word data.

Perception Evolution Network Based on Cognition Deepening Model--Adapting to the Emergence of New Sensory Receptor

The proposed perception evolution network (PEN) is a biologically inspired neural network model for unsupervised learning and online incremental learning. It is able to automatically learn suitable prototypes from learning data in an incremental way, and it does not require the predefined prototype number or the predefined similarity threshold. Meanwhile, being more advanced than the existing unsupervised neural network model, PEN permits the emergence of a new dimension of perception in the perception field of the network. When a new dimension of perception is introduced, PEN is able to integrate the new dimensional sensory inputs with the learned prototypes, i.e., the prototypes are mapped to a high-dimensional space, which consists of both the original dimension and the new dimension of the sensory inputs. In the experiment, artificial data and real-world data are used to test the proposed PEN, and the results show that PEN can work effectively.

Hebbian Network of Self-Organizing Receptive Field Neurons as Associative Incremental Learner

Associative learning plays a major role in the formation of the internal dynamic engine of an adaptive system or a cognitive robot. Interaction with the environment can provide a sparse and discrete set of sample correlations of input–output incidences. These incidences of associative data points can provide useful hints for capturing underlying mechanisms that govern the system’s behavioral dynamics. In many approaches to solving this problem, of learning system’s input–output relation, a set of previously prepared data points need to be presented to the learning mechanism, as a training data, before a useful estimations can be obtained. Besides data-coding is usually based on symbolic or nonimplicit representation schemes. In this paper, we propose an incremental learning mechanism that can bootstrap from a state of complete ignorance of any representative sample associations. Besides, the proposed system provides a novel mechanism for data representation in nonlinear manner through the fusion of self-organizing maps and Gaussian receptive fields. Our architecture is based solely on cortically-inspired techniques of coding and learning as: Hebbian plasticity and adaptive populations of neural circuitry for stimuli representation.

We define a neural network that captures the problem’s data space components using emergent arrangement of receptive field neurons that self-organize incrementally in response to sparse experiences of system–environment interactions. These learned components are correlated using a process of Hebbian plasticity that relates major components of input space to those of the output space. The viability of the proposed mechanism is demonstrated through multiple experimental setups from real-world regression and robotic arm sensory-motor learning problems.

STRUCTURALLY ENHANCED INCREMENTAL NEURAL LEARNING FOR IMAGE CLASSIFICATION WITH SUBGRAPH EXTRACTION

In this paper, a structurally enhanced incremental neural learning technique is proposed to learn a discriminative codebook representation of images for effective image classification applications. In order to accommodate the relationships such as structures and distributions among visual words into the codebook learning process, we develop an online codebook graph learning method based on a novel structurally enhanced incremental learning technique, called as "visualization-induced self-organized incremental neural network (ViSOINN)". The hidden structural information in the images is embedded into the graph representation evolving dynamically with the adaptive and competitive learning mechanism. Afterwards, image features can be coded using a sub-graph extraction process based on the learned codebook graph, and a classifier is subsequently used to complete the image classification task. Compared with other codebook learning algorithms originated from the classical Bag-of-Features (BoF) model, ViSOINN holds the following advantages: (1) it learns codebook efficiently and effectively from a small training set; (2) it models the relationships among visual words in metric scaling fashion, so preserving high discriminative power; (3) it automatically learns the codebook without a fixed pre-defined size; and (4) it enhances and preserves better the structure of the data. These characteristics help to improve image classification performance and make it more suitable for handling large-scale image classification tasks. Experimental results on the widely used Caltech-101 and Caltech-256 benchmark datasets demonstrate that ViSOINN achieves markedly improved performance and reduces the computational cost considerably.