Feature extraction with deep neural networks by a generalized discriminant analysis

Extraction and spatiotemporal analysis of impervious surfaces in Chongqing based on enhanced DeepLabv3

In this study, Sentinel-2 time series satellite remote sensing imagery and an improved CA-DeepLabV3+ semantic segmentation network were utilized to construct a model for extracting urban impervious surfaces. The model was used to extract the distribution information of impervious surfaces in the central urban area in Chongqing from 2017 to 2022. The spatiotemporal evolution characteristics of the impervious surfaces were analyzed using the area change and standard deviational ellipse methods. The results indicate that the improved CA-DeepLabV3+ model performs exceptionally well in identifying impervious surfaces, with precision, recall, F1 score, and MIoU values of 90.78%, 90.85%, 90.82%, and 83.25%, respectively, which are significantly better than those of other classic semantic segmentation models, demonstrating its high reliability and generalization performance. The analysis shows that the impervious surface area in Chongqing's central urban area has grown rapidly over the past five years, with a clear expansion trend, especially in the core urban area and its surrounding areas. The standard deviational ellipse analysis revealed that significant directional expansion of the impervious surfaces has occurred, primarily along the north-south axis. Overall, this model can achieve large-scale, time-series monitoring of the impervious surface distribution, providing critical technical support for studying urban impervious surface expansion and fine urban management, presenting promising application prospects.

Predicting the Early Detection of Breast Cancer Using Hybrid Machine Learning Systems and Thermographic Imaging

Breast cancer is a leading cause of mortality among women, emphasizing the critical need for precise early detection and prognosis. However, conventional methods often struggle to differentiate precancerous lesions or tailor treatments effectively. Thermal imaging, capturing subtle temperature variations, presents a promising avenue for non‐invasive cancer detection. While some studies explore thermography for breast cancer detection, integrating it with advanced machine learning for early diagnosis and personalized prediction remains relatively unexplored. This study proposes a novel hybrid machine learning system (HMLS) incorporating deep autoencoder techniques for automated early detection and prognostic stratification of breast cancer patients. By exploiting the temporal dynamics of thermographic data, this approach offers a more comprehensive analysis than static single‐frame approaches. Data processing involves splitting the dataset for training and testing. A predominant infrared image was selected, and matrix factorization was applied to capture temperature changes over time. Integration of convex factor analysis and bell‐curve membership function embedding for dimensionality reduction and feature extraction. The autoencoder deep neural network further reduces dimensionality. HMLS model development included feature selection and optimization of survival prediction algorithms through cross‐validation. Model performance was assessed using accuracy and F‐measure metrics. HMLS, integrating clinical data, achieved 81.6% accuracy, surpassing 77.6% using only convex‐NMF. The best classifier attained 83.2% accuracy on test data. This study demonstrates the effectiveness of thermographic imaging and HMLS for accurate early detection and personalized prediction of breast cancer. The proposed framework holds promise for enhancing patient care and potentially reducing mortality rates.

Gaussian class-conditional simplex loss for accurate, adversarially robust deep classifier training

In this work, we present the Gaussian Class-Conditional Simplex (GCCS) loss: a novel approach for training deep robust multiclass classifiers that improves over the state-of-the-art in terms of classification accuracy and adversarial robustness, with little extra cost for network training. The proposed method learns a mapping of the input classes onto Gaussian target distributions in a latent space such that a hyperplane can be used as the optimal decision surface. Instead of maximizing the likelihood of target labels for individual samples, our loss function pushes the network to produce feature distributions yielding high inter-class separation and low intra-class separation. The mean values of the learned distributions are centered on the vertices of a simplex such that each class is at the same distance from every other class. We show that the regularization of the latent space based on our approach yields excellent classification accuracy. Moreover, GCCS provides improved robustness against adversarial perturbations, outperforming models trained with conventional adversarial training (AT). In particular, our model learns a decision space that minimizes the presence of short paths toward neighboring decision regions. We provide a comprehensive empirical evaluation that shows how GCCS outperforms state-of-the-art approaches over challenging datasets for targeted and untargeted gradient-based, as well as gradient-free adversarial attacks, both in terms of classification accuracy and adversarial robustness.

Discriminant Feature Extraction by Generalized Difference Subspace

In this paper, we reveal the discriminant capacity of orthogonal data projection onto the generalized difference subspace (GDS), both theoretically and experimentally. In our previous work, we demonstrated that the GDS projection works as a quasi-orthogonalization of class subspaces, which is an effective feature extraction for subspace based classifiers. Here, we further show that GDS projection also works as a discriminant feature extraction through a similar mechanism to the Fisher discriminant analysis (FDA). A direct proof of the connection between GDS projection and FDA is difficult due to the significant difference in their formulations. To circumvent the complication, we first introduce geometrical Fisher discriminant analysis (gFDA) based on a simplified Fisher criterion. It is derived from a heuristic yet practically plausible assumption: the direction of the sample mean vector of a class is largely aligned to the first principal component vector of the class, given that the principal component analysis (PCA) is applied without data centering. gFDA works stably even under few samples, bypassing the small sample size (SSS) problem of FDA. We then prove that gFDA is equivalent to GDS projection with a small correction term. This equivalence ensures GDS projection to inherit the discriminant ability from FDA via gFDA. Furthermore, we discuss two useful extensions of these methods, 1) a nonlinear extension by kernel trick, 2) a combination with CNN features. The equivalence and the effectiveness of the extensions have been verified through extensive experiments on the extended Yale B+, CMU face database, ALOI, ETH80, MNIST, and CIFAR10, mainly focusing on image recognition under small samples.

Deep Fusion Feature Extraction for Caries Detection on Dental Panoramic Radiographs

Caries is the most well-known disease and relates to the oral health of billions of people around the world. Despite the importance and necessity of a well-designed detection method, studies in caries detection are still limited and show a restriction in performance. In this paper, we proposed a computer-aided diagnosis (CAD) method to detect caries among normal patients using dental radiographs. The proposed method mainly consists of two processes: feature extraction and classification. In the feature extraction phase, the chosen 2D tooth image was employed to extract deep activated features using a deep pre-trained model and geometric features using mathematic formulas. Both feature sets were then combined, called fusion feature, to complement each other defects. Then, the optimal fusion feature set was fed into well-known classification models such as support vector machine (SVM), k-nearest neighbor (KNN), decision tree (DT), Naïve Bayes (NB), and random forest (RF) to determine the best classification model that fit the fusion features set and perform the most preeminent result. The results show 91.70%, 90.43%, and 92.67% for accuracy, sensitivity, and specificity, respectively. The proposed method has outperformed the previous state-of-the-art and shows promising results when none of the measured factors is less than 90%; therefore, the method is promising for dentists and capable of wide-scale implementation caries detection in hospitals.

Deep Least Squares Fisher Discriminant Analysis

While being one of the first and most elegant tools for dimensionality reduction, Fisher linear discriminant analysis (FLDA) is not currently considered among the top methods for feature extraction or classification. In this paper, we will review two recent approaches to FLDA, namely, least squares Fisher discriminant analysis (LSFDA) and regularized kernel FDA (RKFDA) and propose deep FDA (DFDA), a straightforward nonlinear extension of LSFDA that takes advantage of the recent advances on deep neural networks. We will compare the performance of RKFDA and DFDA on a large number of two-class and multiclass problems, many of them involving class-imbalanced data sets and some having quite large sample sizes; we will use, for this, the areas under the receiver operating characteristics (ROCs) curve of the classifiers considered. As we shall see, the classification performance of both methods is often very similar and particularly good on imbalanced problems, but building DFDA models is considerably much faster than doing so for RKFDA, particularly in problems with quite large sample sizes.

Multi-View Linear Discriminant Analysis Network

In many real-world applications, an object can be described from multiple views or styles, leading to the emerging multi-view analysis. To eliminate the complicated (usually highly nonlinear) view discrepancy for favorable cross-view recognition and retrieval, we propose a Multi-view Linear Discriminant Analysis Network (MvLDAN) by seeking a nonlinear discriminant and view-invariant representation shared among multiple views. Unlike existing multi-view methods which directly learn a common space to reduce the view gap, our MvLDAN employs multiple feedforward neural networks (one for each view) and a novel eigenvalue-based multi-view objective function to encapsulate as much discriminative variance as possible into all the available common feature dimensions. With the proposed objective function, the MvLDAN could produce representations possessing: 1) low variance within the same class regardless of view discrepancy, 2) high variance between different classes regardless of view discrepancy, and 3) high covariance between any two views. In brief, in the learned multi-view space, the obtained deep features can be projected into a latent common space in which the samples from the same class are as close to each other as possible (even though they are from different views), and the samples from different classes are as far from each other as possible (even though they are from the same view). The effectiveness of the proposed method is verified by extensive experiments carried out on five databases, in comparison with the 19 state-of-the-art approaches.

Object Detection With Deep Learning: A Review

Due to object detection's close relationship with video analysis and image understanding, it has attracted much research attention in recent years. Traditional object detection methods are built on handcrafted features and shallow trainable architectures. Their performance easily stagnates by constructing complex ensembles that combine multiple low-level image features with high-level context from object detectors and scene classifiers. With the rapid development in deep learning, more powerful tools, which are able to learn semantic, high-level, deeper features, are introduced to address the problems existing in traditional architectures. These models behave differently in network architecture, training strategy, and optimization function. In this paper, we provide a review of deep learning-based object detection frameworks. Our review begins with a brief introduction on the history of deep learning and its representative tool, namely, the convolutional neural network. Then, we focus on typical generic object detection architectures along with some modifications and useful tricks to improve detection performance further. As distinct specific detection tasks exhibit different characteristics, we also briefly survey several specific tasks, including salient object detection, face detection, and pedestrian detection. Experimental analyses are also provided to compare various methods and draw some meaningful conclusions. Finally, several promising directions and tasks are provided to serve as guidelines for future work in both object detection and relevant neural network-based learning systems.

Probabilistic Linear Discriminant Analysis With Vectorial Representation for Tensor Data

Linear discriminant analysis (LDA) has been a widely used supervised feature extraction and dimension reduction method in pattern recognition and data analysis. However, facing high-order tensor data, the traditional LDA-based methods take two strategies. One is vectorizing original data as the first step. The process of vectorization will destroy the structure of high-order data and result in high dimensionality issue. Another is tensor LDA-based algorithms that extract features from each mode of high-order data and the obtained representations are also high-order tensor. This paper proposes a new probabilistic LDA (PLDA) model for tensorial data, namely, tensor PLDA. In this model, each tensorial data are decomposed into three parts: the shared subspace component, the individual subspace component, and the noise part. Furthermore, the first two parts are modeled by a linear combination of latent tensor bases, and the noise component is assumed to follow a multivariate Gaussian distribution. Model learning is conducted through a Bayesian inference process. To further reduce the total number of model parameters, the tensor bases are assumed to have tensor CandeComp/PARAFAC (CP) decomposition. Two types of experiments, data reconstruction and classification, are conducted to evaluate the performance of the proposed model with the convincing result, which is superior or comparable against the existing LDA-based methods.

A Novel Neural Network for Remote Sensing Image Matching

Rapid development of remote sensing (RS) imaging technology makes the acquired images have larger size, higher resolution, and more complex structure, which goes beyond the reach of classical hand-crafted feature-based matching. In this paper, we propose a feature learning approach based on two-branch networks to transform the image matching task into a two-class classification problem. To match two key points, two image patches centered at the key points are entered into the proposed network. The network aims to learn discriminative feature representations for patch matching, so that more matching pairs can be obtained on the premise of maintaining higher subpixel matching accuracy. The proposed network adopts a two-stage training mode to deal with the complex characteristics of RS images. An adaptive sample selection strategy is proposed to determine the size of each patch by the scale of its central key point. Thus, each patch can preserve the texture structure around its key point rather than all patches have a predetermined size. In the matching prediction stage, two strategies, namely, superpixel-based sample graded strategy and superpixel-based ordered spatial matching, are designed to improve the matching efficiency and matching accuracy, respectively. The experimental results and theoretical analysis demonstrate the feasibility, robustness, and effectiveness of the proposed method.

A Cross-Domain Recommender System With Kernel-Induced Knowledge Transfer for Overlapping Entities

The aim of recommender systems is to automatically identify user preferences within collected data, then use those preferences to make recommendations that help with decisions. However, recommender systems suffer from data sparsity problem, which is particularly prevalent in newly launched systems that have not yet had enough time to amass sufficient data. As a solution, cross-domain recommender systems transfer knowledge from a source domain with relatively rich data to assist recommendations in the target domain. These systems usually assume that the entities either fully overlap or do not overlap at all. In practice, it is more common for the entities in the two domains to partially overlap. Moreover, overlapping entities may have different expressions in each domain. Neglecting these two issues reduces prediction accuracy of cross-domain recommender systems in the target domain. To fully exploit partially overlapping entities and improve the accuracy of predictions, this paper presents a cross-domain recommender system based on kernel-induced knowledge transfer, called KerKT. Domain adaptation is used to adjust the feature spaces of overlapping entities, while diffusion kernel completion is used to correlate the non-overlapping entities between the two domains. With this approach, knowledge is effectively transferred through the overlapping entities, thus alleviating data sparsity issues. Experiments conducted on four data sets, each with three sparsity ratios, show that KerKT has 1.13%-20% better prediction accuracy compared with six benchmarks. In addition, the results indicate that transferring knowledge from the source domain to the target domain is both possible and beneficial with even small overlaps.

Using molecular descriptors for assisted screening of heterologous competitive antigens to improve the sensitivity of ELISA for detection of enrofloxacin in raw milk

The use of the heterologous competitive strategy has become a vital method to improve the sensitivity of ELISA. In this work, we prepared an anti-enrofloxacin (ENR) mAb with ENR-bovine serum albumin (BSA) as immunogen. The molecular descriptors of quinolones were then used to screen heterologous coating antigens for the detection of ENR based on an ensemble learning method to improve the sensitivity of the ELISA. Results indicated that 6 of the 7 selected heterologous competitive antigens could enhance the sensitivity of ELISA. The ELISA sensitivity for the detection of ENR with sarafloxacin-BSA as heterologous coating antigen was improved 10-fold (in PBS) and 6-fold (in milk) compared with that with ENR-BSA as homologous antigen. The strategy can effectively screen suitable heterologous competitive antigens to improve the sensitivity of ELISA, followed by preparation of mAb with no additional modification to the corresponding immunogen.

Deep Learning in Microscopy Image Analysis: A Survey

Computerized microscopy image analysis plays an important role in computer aided diagnosis and prognosis. Machine learning techniques have powered many aspects of medical investigation and clinical practice. Recently, deep learning is emerging as a leading machine learning tool in computer vision and has attracted considerable attention in biomedical image analysis. In this paper, we provide a snapshot of this fast-growing field, specifically for microscopy image analysis. We briefly introduce the popular deep neural networks and summarize current deep learning achievements in various tasks, such as detection, segmentation, and classification in microscopy image analysis. In particular, we explain the architectures and the principles of convolutional neural networks, fully convolutional networks, recurrent neural networks, stacked autoencoders, and deep belief networks, and interpret their formulations or modelings for specific tasks on various microscopy images. In addition, we discuss the open challenges and the potential trends of future research in microscopy image analysis using deep learning.

Structure Learning for Deep Neural Networks Based on Multiobjective Optimization

This paper focuses on the connecting structure of deep neural networks and proposes a layerwise structure learning method based on multiobjective optimization. A model with better generalization can be obtained by reducing the connecting parameters in deep networks. The aim is to find the optimal structure with high representation ability and better generalization for each layer. Then, the visible data are modeled with respect to structure based on the products of experts. In order to mitigate the difficulty of estimating the denominator in PoE, the denominator is simplified and taken as another objective, i.e., the connecting sparsity. Moreover, for the consideration of the contradictory nature between the representation ability and the network connecting sparsity, the multiobjective model is established. An improved multiobjective evolutionary algorithm is used to solve this model. Two tricks are designed to decrease the computational cost according to the properties of input data. The experiments on single-layer level, hierarchical level, and application level demonstrate the effectiveness of the proposed algorithm, and the learned structures can improve the performance of deep neural networks.

Heteroscedastic Max-min Distance Analysis for Dimensionality Reduction

Max-min distance analysis (MMDA) performs dimensionality reduction by maximizing the minimum pairwise distance between classes in the latent subspace under the homoscedastic assumption, which can address the class separation problem caused by the Fisher criterion, but is incapable of tackling heteroscedastic data properly. In this paper, we propose two heteroscedastic MMDA (HMMDA) methods to employ the differences of class covariances. Whitened HMMDA (WHMMDA) extends MMDA by utilizing the Chernoff distance as the separability measure between classes in the whitened space. Orthogonal HMMDA (OHMMDA) incorporates the maximization of the minimal pairwise Chernoff distance and the minimization of class compactness into a trace quotient formulation with an orthogonal constraint of the transformation, which can be solved by bisection search. Two variants of OHMMDA further encode the margin information by using only neighboring samples to construct the intra-class and inter-class scatters. Experiments on several UCI datasets and two face databases demonstrate the effectiveness of the HMMDA methods.

Multicolumn RBF Network

This paper proposes the multicolumn RBF network (MCRN) as a method to improve the accuracy and speed of a traditional radial basis function network (RBFN). The RBFN, as a fully connected artificial neural network (ANN), suffers from costly kernel inner-product calculations due to the use of many instances as the centers of hidden units. This issue is not critical for small datasets, as adding more hidden units will not burden the computation time. However, for larger datasets, the RBFN requires many hidden units with several kernel computations to generalize the problem. The MCRN mechanism is constructed based on dividing a dataset into smaller subsets using the k-d tree algorithm. resultant subsets are considered as separate training datasets to train individual RBFNs. Those small RBFNs are stacked in parallel and bulged into the MCRN structure during testing. The MCRN is considered as a well-developed and easy-to-use parallel structure, because each individual ANN has been trained on its own subsets and is completely separate from the other ANNs. This parallelized structure reduces the testing time compared with that of a single but larger RBFN, which cannot be easily parallelized due to its fully connected structure. Small informative subsets provide the MCRN with a regional experience to specify the problem instead of generalizing it. The MCRN has been tested on many benchmark datasets and has shown better accuracy and great improvements in training and testing times compared with a single RBFN. The MCRN also shows good results compared with those of some machine learning techniques, such as the support vector machine and k-nearest neighbors.

Airline Passenger Profiling Based on Fuzzy Deep Machine Learning

Passenger profiling plays a vital part of commercial aviation security, but classical methods become very inefficient in handling the rapidly increasing amounts of electronic records. This paper proposes a deep learning approach to passenger profiling. The center of our approach is a Pythagorean fuzzy deep Boltzmann machine (PFDBM), whose parameters are expressed by Pythagorean fuzzy numbers such that each neuron can learn how a feature affects the production of the correct output from both the positive and negative sides. We propose a hybrid algorithm combining a gradient-based method and an evolutionary algorithm for training the PFDBM. Based on the novel learning model, we develop a deep neural network (DNN) for classifying normal passengers and potential attackers, and further develop an integrated DNN for identifying group attackers whose individual features are insufficient to reveal the abnormality. Experiments on data sets from Air China show that our approach provides much higher learning ability and classification accuracy than existing profilers. It is expected that the fuzzy deep learning approach can be adapted for a variety of complex pattern analysis tasks.

Class-Specific Kernel Discriminant Analysis Revisited: Further Analysis and Extensions

In this paper, we revisit class-specific kernel discriminant analysis (KDA) formulation, which has been applied in various problems, such as human face verification and human action recognition. We show that the original optimization problem solved for the determination of class-specific discriminant projections is equivalent to a low-rank kernel regression (LRKR) problem using training data-independent target vectors. In addition, we show that the regularized version of class-specific KDA is equivalent to a regularized LRKR problem, exploiting the same targets. This analysis allows us to devise a novel fast solution. Furthermore, we devise novel incremental, approximate and deep (hierarchical) variants. The proposed methods are tested in human facial image and action video verification problems, where their effectiveness and efficiency is shown.

On Deep Learning for Trust-Aware Recommendations in Social Networks

With the emergence of online social networks, the social network-based recommendation approach is popularly used. The major benefit of this approach is the ability of dealing with the problems with cold-start users. In addition to social networks, user trust information also plays an important role to obtain reliable recommendations. Although matrix factorization (MF) becomes dominant in recommender systems, the recommendation largely relies on the initialization of the user and item latent feature vectors. Aiming at addressing these challenges, we develop a novel trust-based approach for recommendation in social networks. In particular, we attempt to leverage deep learning to determinate the initialization in MF for trust-aware social recommendations and to differentiate the community effect in user's trusted friendships. A two-phase recommendation process is proposed to utilize deep learning in initialization and to synthesize the users' interests and their trusted friends' interests together with the impact of community effect for recommendations. We perform extensive experiments on real-world social network data to demonstrate the accuracy and effectiveness of our proposed approach in comparison with other state-of-the-art methods.

Deep Neural Network for Structural Prediction and Lane Detection in Traffic Scene

Hierarchical neural networks have been shown to be effective in learning representative image features and recognizing object classes. However, most existing networks combine the low/middle level cues for classification without accounting for any spatial structures. For applications such as understanding a scene, how the visual cues are spatially distributed in an image becomes essential for successful analysis. This paper extends the framework of deep neural networks by accounting for the structural cues in the visual signals. In particular, two kinds of neural networks have been proposed. First, we develop a multitask deep convolutional network, which simultaneously detects the presence of the target and the geometric attributes (location and orientation) of the target with respect to the region of interest. Second, a recurrent neuron layer is adopted for structured visual detection. The recurrent neurons can deal with the spatial distribution of visible cues belonging to an object whose shape or structure is difficult to explicitly define. Both the networks are demonstrated by the practical task of detecting lane boundaries in traffic scenes. The multitask convolutional neural network provides auxiliary geometric information to help the subsequent modeling of the given lane structures. The recurrent neural network automatically detects lane boundaries, including those areas containing no marks, without any explicit prior knowledge or secondary modeling.

Scalable Linear Visual Feature Learning via Online Parallel Nonnegative Matrix Factorization

Visual feature learning, which aims to construct an effective feature representation for visual data, has a wide range of applications in computer vision. It is often posed as a problem of nonnegative matrix factorization (NMF), which constructs a linear representation for the data. Although NMF is typically parallelized for efficiency, traditional parallelization methods suffer from either an expensive computation or a high runtime memory usage. To alleviate this problem, we propose a parallel NMF method called alternating least square block decomposition (ALSD), which efficiently solves a set of conditionally independent optimization subproblems based on a highly parallelized fine-grained grid-based blockwise matrix decomposition. By assigning each block optimization subproblem to an individual computing node, ALSD can be effectively implemented in a MapReduce-based Hadoop framework. In order to cope with dynamically varying visual data, we further present an incremental version of ALSD, which is able to incrementally update the NMF solution with a low computational cost. Experimental results demonstrate the efficiency and scalability of the proposed methods as well as their applications to image clustering and image retrieval.

Salient Band Selection for Hyperspectral Image Classification via Manifold Ranking

Saliency detection has been a hot topic in recent years, and many efforts have been devoted in this area. Unfortunately, the results of saliency detection can hardly be utilized in general applications. The primary reason, we think, is unspecific definition of salient objects, which makes that the previously published methods cannot extend to practical applications. To solve this problem, we claim that saliency should be defined in a context and the salient band selection in hyperspectral image (HSI) is introduced as an example. Unfortunately, the traditional salient band selection methods suffer from the problem of inappropriate measurement of band difference. To tackle this problem, we propose to eliminate the drawbacks of traditional salient band selection methods by manifold ranking. It puts the band vectors in the more accurate manifold space and treats the saliency problem from a novel ranking perspective, which is considered to be the main contributions of this paper. To justify the effectiveness of the proposed method, experiments are conducted on three HSIs, and our method is compared with the six existing competitors. Results show that the proposed method is very effective and can achieve the best performance among the competitors.

Change Detection in Synthetic Aperture Radar Images Based on Deep Neural Networks

This paper presents a novel change detection approach for synthetic aperture radar images based on deep learning. The approach accomplishes the detection of the changed and unchanged areas by designing a deep neural network. The main guideline is to produce a change detection map directly from two images with the trained deep neural network. The method can omit the process of generating a difference image (DI) that shows difference degrees between multitemporal synthetic aperture radar images. Thus, it can avoid the effect of the DI on the change detection results. The learning algorithm for deep architectures includes unsupervised feature learning and supervised fine-tuning to complete classification. The unsupervised feature learning aims at learning the representation of the relationships between the two images. In addition, the supervised fine-tuning aims at learning the concepts of the changed and unchanged pixels. Experiments on real data sets and theoretical analysis indicate the advantages, feasibility, and potential of the proposed method. Moreover, based on the results achieved by various traditional algorithms, respectively, deep learning can further improve the detection performance.

A Multiobjective Sparse Feature Learning Model for Deep Neural Networks

Hierarchical deep neural networks are currently popular learning models for imitating the hierarchical architecture of human brain. Single-layer feature extractors are the bricks to build deep networks. Sparse feature learning models are popular models that can learn useful representations. But most of those models need a user-defined constant to control the sparsity of representations. In this paper, we propose a multiobjective sparse feature learning model based on the autoencoder. The parameters of the model are learnt by optimizing two objectives, reconstruction error and the sparsity of hidden units simultaneously to find a reasonable compromise between them automatically. We design a multiobjective induced learning procedure for this model based on a multiobjective evolutionary algorithm. In the experiments, we demonstrate that the learning procedure is effective, and the proposed multiobjective model can learn useful sparse features.

Incremental Linear Discriminant Analysis: A Fast Algorithm and Comparisons

It has always been a challenging task to develop a fast and an efficient incremental linear discriminant analysis (ILDA) algorithm. For this purpose, we conduct a new study for linear discriminant analysis (LDA) in this paper and develop a new ILDA algorithm. We propose a new batch LDA algorithm called LDA/QR. LDA/QR is a simple and fast LDA algorithm, which is obtained by computing the economic QR factorization of the data matrix followed by solving a lower triangular linear system. The relationship between LDA/QR and uncorrelated LDA (ULDA) is also revealed. Based on LDA/QR, we develop a new incremental LDA algorithm called ILDA/QR. The main features of our ILDA/QR include that: 1) it can easily handle the update from one new sample or a chunk of new samples; 2) it has efficient computational complexity and space complexity; and 3) it is very fast and always achieves competitive classification accuracy compared with ULDA algorithm and existing ILDA algorithms. Numerical experiments based on some real-world data sets demonstrate that our ILDA/QR is very efficient and competitive with the state-of-the-art ILDA algorithms in terms of classification accuracy, computational complexity, and space complexity.

Real-time keypoint recognition using restricted Boltzmann machine

Feature point recognition is a key component in many vision-based applications, such as vision-based robot navigation, object recognition and classification, image-based modeling, and augmented reality. Real-time performance and high recognition rates are of crucial importance to these applications. In this brief, we propose a novel method for real-time keypoint recognition using restricted Boltzmann machine (RBM). RBMs are generative models that can learn probability distributions of many different types of data including labeled and unlabeled data sets. Due to the inherent noise of the training data sets, we use an RBM to model statistical distributions of the training data. Furthermore, the learned RBM can be used as a competitive classifier to recognize the keypoints in real-time during the tracking stage, thus making it advantageous to be employed in applications that require real-time performance. Experiments have been conducted under a variety of conditions to demonstrate the effectiveness and generalization of the proposed approach.

Distributed object detection with linear SVMs

In vision and learning, low computational complexity and high generalization are two important goals for video object detection. Low computational complexity here means not only fast speed but also less energy consumption. The sliding window object detection method with linear support vector machines (SVMs) is a general object detection framework. The computational cost is herein mainly paid in complex feature extraction and innerproduct-based classification. This paper first develops a distributed object detection framework (DOD) by making the best use of spatial-temporal correlation, where the process of feature extraction and classification is distributed in the current frame and several previous frames. In each framework, only subfeature vectors are extracted and the response of partial linear classifier (i.e., subdecision value) is computed. To reduce the dimension of traditional block-based histograms of oriented gradients (BHOG) feature vector, this paper proposes a cell-based HOG (CHOG) algorithm, where the features in one cell are not shared with overlapping blocks. Using CHOG as feature descriptor, we develop CHOG-DOD as an instance of DOD framework. Experimental results on detection of hand, face, and pedestrian in video show the superiority of the proposed method.

Dictionary learning-based subspace structure identification in spectral clustering

In this paper, we study dictionary learning (DL) approach to identify the representation of low-dimensional subspaces from high-dimensional and nonnegative data. Such representation can be used to provide an affinity matrix among different subspaces for data clustering. The main contribution of this paper is to consider both nonnegativity and sparsity constraints together in DL such that data can be represented effectively by nonnegative and sparse coding coefficients and nonnegative dictionary bases. In the algorithm, we employ the proximal point technique for the resulting DL and sparsity optimization problem. We make use of coding coefficients to perform spectral clustering (SC) for data partitioning. Extensive experiments on real-world high-dimensional and nonnegative data sets, including text, microarray, and image data demonstrate that the proposed method can discover their subspace structures. Experimental results also show that our algorithm is computationally efficient and effective for obtaining high SC performance and interpreting the clustering results compared with the other testing methods.

Nonstationary source separation using sequential and variational Bayesian learning

Independent component analysis (ICA) is a popular approach for blind source separation where the mixing process is assumed to be unchanged with a fixed set of stationary source signals. However, the mixing system and source signals are nonstationary in real-world applications, e.g., the source signals may abruptly appear or disappear, the sources may be replaced by new ones or even moving by time. This paper presents an online learning algorithm for the Gaussian process (GP) and establishes a separation procedure in the presence of nonstationary and temporally correlated mixing coefficients and source signals. In this procedure, we capture the evolved statistics from sequential signals according to online Bayesian learning. The activity of nonstationary sources is reflected by an automatic relevance determination, which is incrementally estimated at each frame and continuously propagated to the next frame. We employ the GP to characterize the temporal structures of time-varying mixing coefficients and source signals. A variational Bayesian inference is developed to approximate the true posterior for estimating the nonstationary ICA parameters and for characterizing the activity of latent sources. The differences between this ICA method and the sequential Monte Carlo ICA are illustrated. In the experiments, the proposed algorithm outperforms the other ICA methods for the separation of audio signals in the presence of different nonstationary scenarios.

Simple exponential family PCA

Principal component analysis (PCA) is a widely used model for dimensionality reduction. In this paper, we address the problem of determining the intrinsic dimensionality of a general type data population by selecting the number of principal components for a generalized PCA model. In particular, we propose a generalized Bayesian PCA model, which deals with general type data by employing exponential family distributions. Model selection is realized by empirical Bayesian inference of the model. We name the model as simple exponential family PCA (SePCA), since it embraces both the principal of using a simple model for data representation and the practice of using a simplified computational procedure for the inference. Our analysis shows that the empirical Bayesian inference in SePCA formally realizes an intuitive criterion for PCA model selection - a preserved principal component must sufficiently correlate to data variance that is uncorrelated to the other principal components. Experiments on synthetic and real data sets demonstrate effectiveness of SePCA and exemplify its characteristics for model selection.

Portfolio of automated trading systems: complexity and learning set size issues

In this paper, we consider using profit/loss histories of multiple automated trading systems (ATSs) as N input variables in portfolio management. By means of multivariate statistical analysis and simulation studies, we analyze the influences of sample size (L) and input dimensionality on the accuracy of determining the portfolio weights. We find that degradation in portfolio performance due to inexact estimation of N means and N(N - 1)/2 correlations is proportional to N/L; however, estimation of N variances does not worsen the result. To reduce unhelpful sample size/dimensionality effects, we perform a clustering of N time series and split them into a small number of blocks. Each block is composed of mutually correlated ATSs. It generates an expert trading agent based on a nontrainable 1/N portfolio rule. To increase the diversity of the expert agents, we use training sets of different lengths for clustering. In the output of the portfolio management system, the regularized mean-variance framework-based fusion agent is developed in each walk-forward step of an out-of-sample portfolio validation experiment. Experiments with the real financial data (2003-2012) confirm the effectiveness of the suggested approach.

Generalization performance of Fisher linear discriminant based on Markov sampling

Fisher linear discriminant (FLD) is a well-known method for dimensionality reduction and classification that projects high-dimensional data onto a low-dimensional space where the data achieves maximum class separability. The previous works describing the generalization ability of FLD have usually been based on the assumption of independent and identically distributed (i.i.d.) samples. In this paper, we go far beyond this classical framework by studying the generalization ability of FLD based on Markov sampling. We first establish the bounds on the generalization performance of FLD based on uniformly ergodic Markov chain (u.e.M.c.) samples, and prove that FLD based on u.e.M.c. samples is consistent. By following the enlightening idea from Markov chain Monto Carlo methods, we also introduce a Markov sampling algorithm for FLD to generate u.e.M.c. samples from a given data of finite size. Through simulation studies and numerical studies on benchmark repository using FLD, we find that FLD based on u.e.M.c. samples generated by Markov sampling can provide smaller misclassification rates compared to i.i.d. samples.

Noise-tuning-based hysteretic noisy chaotic neural network for broadcast scheduling problem in wireless multihop networks

Compared with noisy chaotic neural networks (NCNNs), hysteretic noisy chaotic neural networks (HNCNNs) are more likely to exhibit better optimization performance at higher noise levels, but behave worse at lower noise levels. In order to improve the optimization performance of HNCNNs, this paper presents a novel noise-tuning-based hysteretic noisy chaotic neural network (NHNCNN). Using a noise tuning factor to modulate the level of stochastic noises, the proposed NHNCNN not only balances stochastic wandering and chaotic searching, but also exhibits stronger hysteretic dynamics, thereby improving the optimization performance at both lower and higher noise levels. The aim of the broadcast scheduling problem (BSP) in wireless multihop networks (WMNs) is to design an optimal time-division multiple-access frame structure with minimal frame length and maximal channel utilization. A gradual NHNCNN (G-NHNCNN), which combines the NHNCNN with the gradual expansion scheme, is applied to solve BSP in WMNs to demonstrate the performance of the NHNCNN. Simulation results show that the proposed NHNCNN has a larger probability of finding better solutions compared to both the NCNN and the HNCNN regardless of whether noise amplitudes are lower or higher.