Supervised Feature Selection With Orthogonal Regression and Feature Weighting

Subspace learning using low-rank latent representation learning and perturbation theorem: Unsupervised gene selection

In recent years, gene expression data analysis has gained growing significance in the fields of machine learning and computational biology. Typically, microarray gene datasets exhibit a scenario where the number of features exceeds the number of samples, resulting in an ill-posed and underdetermined equation system. The presence of redundant features in high-dimensional data leads to suboptimal performance and increased computational time for learning algorithms. Although feature extraction and feature selection are two approaches that can be employed to deal with this challenge, feature selection has greater interpretability ability which causes it to receive more attention. In this study, we propose an unsupervised feature selection which is based on pseudo label latent representation learning and perturbation theory. In the first step, pseudo labels are extracted and constructed using latent representation learning. In the second step, the least square problem is solved for original data matrix and perturbed data matrix. Features are clustered based on the similarity between the original data matrix and the perturbed data matrix using k-means. In the last step, features in each subcluster are ranked based on information gain criterion. To showcase the efficacy of the proposed approach, numerical experiments were carried out on six benchmark microarray datasets and two RNA-Sequencing benchmark datasets. The outcomes indicate that the proposed technique surpasses eight state-of-the-art unsupervised feature selection methods in both clustering accuracy and normalized mutual information.

Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance

Feature selection is essential in the analysis of molecular systems and many other fields, but several uncertainties remain: What is the optimal number of features for a simplified, interpretable model that retains essential information? How should features with different units be aligned, and how should their relative importance be weighted? Here, we introduce the Differentiable Information Imbalance (DII), an automated method to rank information content between sets of features. Using distances in a ground truth feature space, DII identifies a low-dimensional subset of features that best preserves these relationships. Each feature is scaled by a weight, which is optimized by minimizing the DII through gradient descent. This allows simultaneously performing unit alignment and relative importance scaling, while preserving interpretability. DII can also produce sparse solutions and determine the optimal size of the reduced feature space. We demonstrate the usefulness of this approach on two benchmark molecular problems: (1) identifying collective variables that describe conformations of a biomolecule, and (2) selecting features for training a machine-learning force field. These results show the potential of DII in addressing feature selection challenges and optimizing dimensionality in various applications. The method is available in the Python library DADApy.

A general adaptive unsupervised feature selection with auto-weighting

Feature selection (FS) is essential in machine learning and data mining as it makes handling high-dimensional data more efficient and reliable. More attention has been paid to unsupervised feature selection (UFS) due to the extra resources required to obtain labels for data in the real world. Most of the existing embedded UFS utilize a sparse projection matrix for FS. However, this may introduce additional regularization terms, and it is difficult to control the sparsity of the projection matrix well. Moreover, such methods may seriously destroy the original feature structure in the embedding space. Instead, avoiding projecting the original data into the low-dimensional embedding space and identifying features directly from the raw features that perform well in the process of making the data show a distinct cluster structure is a feasible solution. Inspired by this, this paper proposes a model called A General Adaptive Unsupervised Feature Selection with Auto-weighting (GAWFS), which utilizes two techniques, non-negative matrix factorization, and adaptive graph learning, to simulate the process of dividing data into clusters, and identifies the features that are most discriminative in the clustering process by a feature weighting matrix Θ. Since the weighting matrix is sparse, it also plays the role of FS or a filter. Finally, experiments comparing GAWFS with several state-of-the-art UFS methods on synthetic datasets and real-world datasets are conducted, and the results demonstrate the superiority of the GAWFS.

Pseudo-Label Guided Structural Discriminative Subspace Learning for Unsupervised Feature Selection

In this article, we propose a new unsupervised feature selection method named pseudo-label guided structural discriminative subspace learning (PSDSL). Unlike the previous methods that perform the two stages independently, it introduces the construction of probability graph into the feature selection learning process as a unified general framework, and therefore the probability graph can be learned adaptively. Moreover, we design a pseudo-label guided learning mechanism, and combine the graph-based method and the idea of maximizing the between-class scatter matrix with the trace ratio to construct an objective function that can improve the discrimination of the selected features. Besides, the main existing strategies of selecting features are to employ -norm for feature selection, but this faces the challenges of sparsity limitations and parameter tuning. For addressing this issue, we employ the -norm constraint on the learned subspace to ensure the row sparsity of the model and make the selected feature more stable. Effective optimization strategy is given to solve such NP-hard problem with the determination of parameters and complexity analysis in theory. Ultimately, extensive experiments conducted on nine real-world datasets and three biological ScRNA-seq genes datasets verify the effectiveness of the proposed method on the data clustering downstream task.

Surrogate-Assisted and Filter-Based Multiobjective Evolutionary Feature Selection for Deep Learning

Feature selection (FS) for deep learning prediction models is a difficult topic for researchers to tackle. Most of the approaches proposed in the literature consist of embedded methods through the use of hidden layers added to the neural network architecture that modify the weights of the units associated with each input attribute so that the worst attributes have less weight in the learning process. Other approaches used for deep learning are filter methods, which are independent of the learning algorithm, which can limit the precision of the prediction model. Wrapper methods are impractical with deep learning due to their high computational cost. In this article, we propose new attribute subset evaluation FS methods for deep learning of the wrapper, filter and wrapper-filter hybrid types, where multiobjective and many-objective evolutionary algorithms are used as search strategies. A novel surrogate-assisted approach is used to reduce the high computational cost of the wrapper-type objective function, while the filter-type objective functions are based on correlation and an adaptation of the reliefF algorithm. The proposed techniques have been applied in a time series forecasting problem of air quality in the Spanish south-east and an indoor temperature forecasting problem in a domotic house, with promising results compared to other FS techniques used in the literature.

Supervised Feature Selection via Collaborative Neurodynamic Optimization

As a crucial part of machine learning and pattern recognition, feature selection aims at selecting a subset of the most informative features from the set of all available features. In this article, supervised feature selection is at first formulated as a mixed-integer optimization problem with an objective function of weighted feature redundancy and relevancy subject to a cardinality constraint on the number of selected features. It is equivalently reformulated as a bound-constrained mixed-integer optimization problem by augmenting the objective function with a penalty function for realizing the cardinality constraint. With additional bilinear and linear equality constraints for realizing the integrality constraints, it is further reformulated as a bound-constrained biconvex optimization problem with two more penalty terms. Two collaborative neurodynamic optimization (CNO) approaches are proposed for solving the formulated and reformulated feature selection problems. One of the proposed CNO approaches uses a population of discrete-time recurrent neural networks (RNNs), and the other use a pair of continuous-time projection networks operating concurrently on two timescales. Experimental results on 13 benchmark datasets are elaborated to substantiate the superiority of the CNO approaches to several mainstream methods in terms of average classification accuracy with three commonly used classifiers.

Embedded EEG Feature Selection for Multi-Dimension Emotion Recognition via Local and Global Label Relevance

Due to the problem of a small amount of EEG samples and relatively high dimensionality of electroencephalogram (EEG) features, feature selection plays an essential role in EEG-based emotion recognition. However, current EEG-based emotion recognition studies utilize a problem transformation approach to transform multi-dimension emotional labels into single-dimension labels, and then implement commonly used single-label feature selection methods to search feature subsets, which ignores the relations between different emotional dimensions. To tackle the problem, we propose an efficient EEG feature selection method for multi-dimension emotion recognition (EFSMDER) via local and global label relevance. First, to capture the local label correlations, EFSMDER implements orthogonal regression to map the original EEG feature space into a low-dimension space. Then, it employs the global label correlations in the original multi-dimension emotional label space to effectively construct the label information in the low-dimension space. With the aid of local and global relevance information, EFSMDER can conduct representational EEG feature subset selection. Three EEG emotional databases with multi-dimension emotional labels were used for performance comparison between EFSMDER and fourteen state-of-the-art methods, and the EFSMDER method achieves the best multi-dimension classification accuracies of 86.43, 84.80, and 97.86 percent on the DREAMER, DEAP, and HDED datasets, respectively.

Sparse Index Tracking With K-Sparsity or ϵ-Deviation Constraint via ℓ0-Norm Minimization

Sparse index tracking, as one of the passive investment strategies, is to track a benchmark financial index via constructing a portfolio with a few assets in a market index. It can be considered as parameter learning in an adaptive system, in which we periodically update the selected assets and their investment percentages based on the sliding window approach. However, many existing algorithms for sparse index tracking cannot explicitly and directly control the number of assets or the tracking error. This article formulates sparse index tracking as two constrained optimization problems and then proposes two algorithms, namely, nonnegative orthogonal matching pursuit with projected gradient descent (NNOMP-PGD) and alternating direction method of multipliers for l0 -norm (ADMM- l0 ). The NNOMP-PGD aims at minimizing the tracking error subject to the number of selected assets less than or equal to a predefined number. With the NNOMP-PGD, investors can directly and explicitly control the number of selected assets. The ADMM- l0 aims at minimizing the number of selected assets subject to the tracking error that is upper bounded by a preset threshold. It can directly and explicitly control the tracking error. The convergence of the two proposed algorithms is also presented. With our algorithms, investors can explicitly and directly control the number of selected assets or the tracking error of the resultant portfolio. In addition, numerical experiments demonstrate that the proposed algorithms outperform the existing approaches.

Robust Supervised and Semisupervised Least Squares Regression Using ℓ2,p-Norm Minimization

Least squares regression (LSR) is widely applied in statistics theory due to its theoretical solution, which can be used in supervised, semisupervised, and multiclass learning. However, LSR begins to fail and its discriminative ability cannot be guaranteed when the original data have been corrupted and noised. In reality, the noises are unavoidable and could greatly affect the error construction in LSR. To cope with this problem, a robust supervised LSR (RSLSR) is proposed to eliminate the effect of noises and outliers. The loss function adopts l2,p -norm ( ) instead of square loss. In addition, the probability weight is added to each sample to determine whether the sample is a normal point or not. Its physical meaning is very clear, in which if the point is normal, the probability value is 1; otherwise, the weight is 0. To effectively solve the concave problem, an iterative algorithm is introduced, in which additional weights are added to penalize normal samples with large errors. We also extend RSLSR to robust semisupervised LSR (RSSLSR) to fully utilize the limited labeled samples. A large number of classification performances on corrupted data illustrate the robustness of the proposed methods.

Ensemble Feature Selection With Block-Regularized m × 2 Cross-Validation

Ensemble feature selection (EFS) has attracted significant interest in the literature due to its great potential in reducing the discovery rate of noise features and stabilizing the feature selection results. In view of the superior performance of block-regularized m × 2 cross-validation on generalization performance and algorithm comparison, a novel EFS technology based on block-regularized m × 2 cross-validation is proposed in this study. Contrary to the traditional ensemble learning with a binomial distribution, the distribution of feature selection frequency in the proposed technique is approximated by a beta distribution more accurately. Furthermore, theoretical analysis of the proposed technique shows that it yields a higher selection probability for important features, lower selected risk for noise features, more true positives, and fewer false positives. Finally, the above conclusions are verified by the simulated and real data experiments.

Semi-supervised regression with adaptive graph learning for EEG-based emotion recognition

Electroencephalogram (EEG) signals are widely used in the field of emotion recognition since it is resistant to camouflage and contains abundant physiological information. However, EEG signals are non-stationary and have low signal-noise-ratio, making it more difficult to decode in comparison with data modalities such as facial expression and text. In this paper, we propose a model termed semi-supervised regression with adaptive graph learning (SRAGL) for cross-session EEG emotion recognition, which has two merits. On one hand, the emotional label information of unlabeled samples is jointly estimated with the other model variables by a semi-supervised regression in SRAGL. On the other hand, SRAGL adaptively learns a graph to depict the connections among EEG data samples which further facilitates the emotional label estimation process. From the experimental results on the SEED-IV data set, we have the following insights. 1) SRAGL achieves superior performance compared to some state-of-the-art algorithms. To be specific, the average accuracies are 78.18%, 80.55%, and 81.90% in the three cross-session emotion recognition tasks. 2) As the iteration number increases, SRAGL converges quickly and optimizes the emotion metric of EEG samples gradually, leading to a reliable similarity matrix finally. 3) Based on the learned regression projection matrix, we obtain the contribution of each EEG feature, which enables us to automatically identify critical frequency bands and brain regions in emotion recognition.

Online Causal Feature Selection for Streaming Features

Recently, causal feature selection (CFS) has attracted considerable attention due to its outstanding interpretability and predictability performance. Such a method primarily includes the Markov blanket (MB) discovery and feature selection based on Granger causality. Representatively, the max-min MB (MMMB) can mine an optimal feature subset, i.e., MB; however, it is unsuitable for streaming features. Online streaming feature selection (OSFS) via online process streaming features can determine parents and children (PC), a subset of MB; however, it cannot mine the MB of the target attribute ( T ), i.e., a given feature, thus resulting in insufficient prediction accuracy. The Granger selection method (GSM) establishes a causal matrix of all features by performing excessively time; however, it cannot achieve a high prediction accuracy and only forecasts fixed multivariate time series data. To address these issues, we proposed an online CFS for streaming features (OCFSSFs) that mine MB containing PC and spouse and adopt the interleaving PC and spouse learning method. Furthermore, it distinguishes between PC and spouse in real time and can identify children with parents online when identifying spouses. We experimentally evaluated the proposed algorithm on synthetic datasets using precision, recall, and distance. In addition, the algorithm was tested on real-world and time series datasets using classification precision, the number of selected features, and running time. The results validated the effectiveness of the proposed algorithm.

Unsupervised Feature Selection via Graph Regularized Nonnegative CP Decomposition

Unsupervised feature selection has attracted remarkable attention recently. With the development of data acquisition technology, multi-dimensional tensor data has been appeared in enormous real-world applications. However, most existing unsupervised feature selection methods are non-tensor-based which results the vectorization of tensor data as a preprocessing step. This seemingly ordinary operation has led to an unnecessary loss of the multi-dimensional structural information and eventually restricted the quality of the selected features. To overcome the limitation, in this paper, we propose a novel unsupervised feature selection model: Nonnegative tensor CP (CANDECOMP/PARAFAC) decomposition based unsupervised feature selection, CPUFS for short. In specific, we devise new tensor-oriented linear classifier and feature selection matrix for CPUFS. In addition, CPUFS simultaneously conducts graph regularized nonnegative CP decomposition and newly-designed tensor-oriented pseudo label regression and feature selection to fully preserve the multi-dimensional data structure. To solve the CPUFS model, we propose an efficient iterative optimization algorithm with theoretically guaranteed convergence, whose computational complexity scales linearly in the number of features. A variation of the CPUFS model by incorporating nonnegativity into the linear classifier, namely CPUFSnn, is also proposed and studied. Experimental results on ten real-world benchmark datasets demonstrate the effectiveness of both CPUFS and CPUFSnn over the state-of-the-arts.

Performance optimization of salp swarm algorithm for multi-threshold image segmentation: Comprehensive study of breast cancer microscopy

Multi-threshold image segmentation (MIS) is now a well known image segmentation technique, and many researchers have applied intelligent algorithms to it, but these methods suffer from local optimal drawbacks. This paper presented a novel approach to improve the Salp Swarm Algorithm (SSA), namely EHSSA, and applied it to MIS. Knowing the inaccuracies and discussions on implementation of this method, a new efficient mechanism is proposed to improve global search capability of the algorithm and avoid falling into a local optimum. Moreover, the excellence of the proposed algorithm was proved by comparative experiments at IEEE CEC2014. Afterward, the performance of EHSSA was demonstrated by testing a set of images selected from the Berkeley segmentation data set 500 (BSDS500), and the experimental results were analyzed by evaluating the parameters, which proved the efficiency of the proposed algorithm in MIS. Furthermore, EHSSA was applied to the microscopic image segmentation of breast cancer. Medical image segmentation is the study of how to quickly extract objects of interest (human organs) from various images to perform qualitative and quantitative analysis of diseased tissues and improve the accuracy of their diagnosis, which assists the physician in making more informed decisions and patient rehabilitation. The results of this set of experiments also proved its superior performance. For any info about this paper, readers can refer to https://aliasgharheidari.com.

Evolving fuzzy k-nearest neighbors using an enhanced sine cosine algorithm: Case study of lupus nephritis

Because of its simplicity and effectiveness, fuzzy K-nearest neighbors (FKNN) is widely used in literature. The parameters have an essential impact on the performance of FKNN. Hence, the parameters need to be attuned to suit different problems. Also, choosing more representative features can enhance the performance of FKNN. This research proposes an improved optimization technique based on the sine cosine algorithm (LSCA), which introduces a linear population size reduction mechanism for enhancing the original algorithm's performance. Moreover, we developed an FKNN model based on the LSCA, it simultaneously performs feature selection and parameter optimization. Firstly, the search performance of LSCA is verified on the IEEE CEC2017 benchmark test function compared to the classical and improved algorithms. Secondly, the validity of the LSCA-FKNN model is verified on three medical datasets. Finally, we used the proposed LSCA-FKNN to predict lupus nephritis classes, and the model showed competitive results. The paper will be supported by an online web service for any question at https://aliasgharheidari.com.