Feature selection with neural networks

Neural network input feature selection using structured l2 − norm penalization

Artificial neural networks are referred to as universal approximators due to their inherent ability to reconstruct complex linear and nonlinear output maps conceived as input-output relationships from data sets. This can be done by reducing large networks via regularization in order to establish compact models containing fewer parameters aimed at describing vital dependencies in data sets. In situations where the data sets contain non-informative input features, devising a continuous, optimal input feature selection technique can lead to improved prediction or classification. We propose a continuous input selection technique through a dimensional reduction mechanism using a ‘structured’ l2 − norm regularization. The implementation is done by identifying the most informative feature subsets from a given data set via an adaptive training mechanism. The adaptation involves introducing a novel, modified gradient approach during training to deal with the non-differentiability associated with the gradient of the structured norm penalty. When the method is applied to process data sets, results indicate that the most informative inputs of artificial neural networks can be selected using a structured l2 − norm penalization.

Combined Free-Energy Calculation and Machine Learning Methods for Understanding Ligand Unbinding Kinetics

The determination of drug residence times, which define the time an inhibitor is in complex with its target, is a fundamental part of the drug discovery process. Synthesis and experimental measurements of kinetic rate constants are, however, expensive and time consuming. In this work, we aimed to obtain drug residence times computationally. Furthermore, we propose a novel algorithm to identify molecular design objectives based on ligand unbinding kinetics. We designed an enhanced sampling technique to accurately predict the free-energy profiles of the ligand unbinding process, focusing on the free-energy barrier for unbinding. Our method first identifies unbinding paths determining a corresponding set of internal coordinates (ICs) that form contacts between the protein and the ligand; it then iteratively updates these interactions during a series of biased molecular dynamics (MD) simulations to reveal the ICs that are important for the whole of the unbinding process. Subsequently, we performed finite-temperature string simulations to obtain the free-energy barrier for unbinding using the set of ICs as a complex reaction coordinate. Importantly, we also aimed to enable the further design of drugs focusing on improved residence times. To this end, we developed a supervised machine learning (ML) approach with inputs from unbiased “downhill” trajectories initiated near the transition state (TS) ensemble of the string unbinding path. We demonstrate that our ML method can identify key ligand–protein interactions driving the system through the TS. Some of the most important drugs for cancer treatment are kinase inhibitors. One of these kinase targets is cyclin-dependent kinase 2 (CDK2), an appealing target for anticancer drug development. Here, we tested our method using two different CDK2 inhibitors for the potential further development of these compounds. We compared the free-energy barriers obtained from our calculations with those observed in available experimental data. We highlighted important interactions at the distal ends of the ligands that can be targeted for improved residence times. Our method provides a new tool to determine unbinding rates and to identify key structural features of the inhibitors that can be used as starting points for novel design strategies in drug discovery.

A Novel Breast Cancer Diagnosis Scheme With Intelligent Feature and Parameter Selections

BACKGROUND AND OBJECTIVE

Breast cancer is the most commonly occurring cancer among women, which contributes to the global death rate. The key to increasing the survival rate of affected patients is early diagnosis along with appropriate treatments. Manual methods for breast cancer diagnosis fail due to human errors, inaccurate diagnoses, and are time-consuming when demands are high. Intelligent systems based on Artificial Neural Network (ANN) for automated breast cancer diagnosis are powerful due to their strong decision-making capabilities in complicated cases. Artificial Bee Colony, Artificial Immune System, and Bacterial Foraging Optimization are swarm intelligence algorithms that solve combinatorial optimization problems. This paper proposes two novel hybrid Artificial Bee Colony (ABC) optimization algorithms that overcome the demerits of standard ABC algorithms. First, this paper proposes a hybrid ABC approach called HABC, in which the standard ABC optimization is hybridized with a modified clonal selection algorithm of the Artificial Immune System that eliminates the poor exploration capabilities of standard ABC optimization. Further, this paper proposes a novel hybrid Artificial Bee Colony (Hybrid ABC) optimization where the strong explorative capabilities of the chemotaxis phase of the bacterial foraging optimization are integrated with a spiral model-based exploitative phase of the ABC by which the proposed Hybrid ABC overcomes the demerits of poor exploration and exploitation of the standard ABC algorithm.

METHODS

In this work, the two proposed hybrid approaches were used in concurrent feature selection and parameter optimization of an ANN model. The proposed algorithm is implemented using various back-propagation algorithms, including resilient back-propagation (HABC-RP and Hybrid ABC-RP), Levenberg Marquart (HABC-LM and Hybrid ABC-LM), and momentum-based gradient descent (HABC-MGD and Hybrid ABC-GD) for parameter tuning of ANN. The Wisconsin breast cancer dataset was used to evaluate the performance of the proposed algorithms in terms of accuracy, complexity, and computational time.

RESULTS

The mean accuracy of the proposed HABC-RP was 99.14% and 99.54% for Hybrid ABC which is better than the results found in the existing literature. HABC-RP attained a sensitivity of 98.32%, a specificity of 99.63%, and a precision of 99.38% whereas Hybrid ABC attained sensitivity of 99.08% and Specificity of 99.81%.

CONCLUSIONS

HABC-RP and Hybrid ABC-RP yielded high accuracy with a low complexity ANN structure compared to other variants. After evaluation, interestingly it is found that the Hybrid ABC-RP has achieved the highest mean accuracy of 99.54% with low complexity of 10.25 mean connections when compared to other variants proposed in this paper. It can be concluded that the concurrent selection of input features and tuning of parameters of ANN plays a vital role in increasing the accuracy of a breast cancer diagnosis. The proposed HABC-RP and Hybrid ABC-RP showed better results when compared to the existing breast cancer diagnosis systems taken for comparison. In the future, the proposed two-hybrid approaches can be used to generate optimal thresholds for the segmentation of tumors in abnormal images. HABC and Hybrid ABC can be used for tuning the parameters of various classifiers.

Interpretable Classification of Bacterial Raman Spectra with Knockoff Wavelets

Deep neural networks and other machine learning models are widely applied to biomedical signal data because they can detect complex patterns and compute accurate predictions. However, the difficulty of interpreting such models is a limitation, especially for applications involving high-stakes decision, including the identification of bacterial infections. This paper considers fast Raman spectroscopy data and demonstrates that a logistic regression model with carefully selected features achieves accuracy comparable to that of neural networks, while being much simpler and more transparent. Our analysis leverages wavelet features with intuitive chemical interpretations, and performs controlled variable selection with knockoffs to ensure the predictors are relevant and non-redundant. Although we focus on a particular data set, the proposed approach is broadly applicable to other types of signal data for which interpretability may be important.

Machine Learning in Predictive Toxicology: Recent Applications and Future Directions for Classification Models

In recent times, machine learning has become increasingly prominent in predictive toxicology as it has shifted from in vivo studies toward in silico studies. Currently, in vitro methods together with other computational methods such as quantitative structure-activity relationship modeling and absorption, distribution, metabolism, and excretion calculations are being used. An overview of machine learning and its applications in predictive toxicology is presented here, including support vector machines (SVMs), random forest (RF) and decision trees (DTs), neural networks, regression models, naïve Bayes, k-nearest neighbors, and ensemble learning. The recent successes of these machine learning methods in predictive toxicology are summarized, and a comparison of some models used in predictive toxicology is presented. In predictive toxicology, SVMs, RF, and DTs are the dominant machine learning methods due to the characteristics of the data available. Lastly, this review describes the current challenges facing the use of machine learning in predictive toxicology and offers insights into the possible areas of improvement in the field.

An Effective Multi-Label Feature Selection Model Towards Eliminating Noisy Features

Feature selection has devoted a consistently great amount of effort to dimension reduction for various machine learning tasks. Existing feature selection models focus on selecting the most discriminative features for learning targets. However, this strategy is weak in handling two kinds of features, that is, the irrelevant and redundant ones, which are collectively referred to as noisy features. These features may hamper the construction of optimal low-dimensional subspaces and compromise the learning performance of downstream tasks. In this study, we propose a novel multi-label feature selection approach by embedding label correlations (dubbed ELC) to address these issues. Particularly, we extract label correlations for reliable label space structures and employ them to steer feature selection. In this way, label and feature spaces can be expected to be consistent and noisy features can be effectively eliminated. An extensive experimental evaluation on public benchmarks validated the superiority of ELC.

Binary Whale Optimization Algorithm for Dimensionality Reduction

Feature selection (FS) was regarded as a global combinatorial optimization problem. FS is used to simplify and enhance the quality of high-dimensional datasets by selecting prominent features and removing irrelevant and redundant data to provide good classification results. FS aims to reduce the dimensionality and improve the classification accuracy that is generally utilized with great importance in different fields such as pattern classification, data analysis, and data mining applications. The main problem is to find the best subset that contains the representative information of all the data. In order to overcome this problem, two binary variants of the whale optimization algorithm (WOA) are proposed, called bWOA-S and bWOA-V. They are used to decrease the complexity and increase the performance of a system by selecting significant features for classification purposes. The first bWOA-S version uses the Sigmoid transfer function to convert WOA values to binary ones, whereas the second bWOA-V version uses a hyperbolic tangent transfer function. Furthermore, the two binary variants introduced here were compared with three famous and well-known optimization algorithms in this domain, such as Particle Swarm Optimizer (PSO), three variants of binary ant lion (bALO1, bALO2, and bALO3), binary Dragonfly Algorithm (bDA) as well as the original WOA, over 24 benchmark datasets from the UCI repository. Eventually, a non-parametric test called Wilcoxon’s rank-sum was carried out at 5% significance to prove the powerfulness and effectiveness of the two proposed algorithms when compared with other algorithms statistically. The qualitative and quantitative results showed that the two introduced variants in the FS domain are able to minimize the selected feature number as well as maximize the accuracy of the classification within an appropriate time.

Decision Support for Breast Cancer Detection: Classification Improvement Through Feature Selection

Several statistical-based approaches have been developed to support medical personnel in early breast cancer detection. This article presents a method for feature selection aimed at classifying cases into categories based on patients' breast tissue measures and protein microarray. The effectiveness of this feature selection strategy was evaluated against the commonly used Wisconsin Breast Cancer Database-WBCD (with several patients and fewer features) and a new protein microarray data set (with several features and fewer patients). Features were ranked according to a feature importance index that combines parameters emerging from the unsupervised method of principal component analysis and the supervised method of Bhattacharyya distance. Observations of a training set were iteratively categorized into malignant and benign cases through 3 classification techniques: k-Nearest Neighbor, linear discriminant analysis, and probabilistic neural network. After each classification, the feature with the smallest importance index was removed, and a new categorization was carried out until there was only one feature left. The subset yielding maximum accuracy was used to classify observations in the testing set. Our method yielded average 99.17% accurate classifications in the testing set while retaining average 4.61 out of 9 features in the WBCD, which is comparable to the best results reported by the literature on that data set, with the advantage of relying on simple and widely available multivariate techniques. When applied to the microarray data, the method yielded average accuracy of 98.30% while retaining average 2.17% of the original features. Our results can aid health-care professionals during early diagnosis of breast cancer.

Combining handcrafted features with latent variables in machine learning for prediction of radiation-induced lung damage

PURPOSE

There has been burgeoning interest in applying machine learning methods for predicting radiotherapy outcomes. However, the imbalanced ratio of a large number of variables to a limited sample size in radiation oncology constitutes a major challenge. Therefore, dimensionality reduction methods can be a key to success. The study investigates and contrasts the application of traditional machine learning methods and deep learning approaches for outcome modeling in radiotherapy. In particular, new joint architectures based on variational autoencoder (VAE) for dimensionality reduction are presented and their application is demonstrated for the prediction of lung radiation pneumonitis (RP) from a large-scale heterogeneous dataset.

METHODS

A large-scale heterogeneous dataset containing a pool of 230 variables including clinical factors (e.g., dose, KPS, stage) and biomarkers (e.g., single nucleotide polymorphisms (SNPs), cytokines, and micro-RNAs) in a population of 106 nonsmall cell lung cancer (NSCLC) patients who received radiotherapy was used for modeling RP. Twenty-two patients had grade 2 or higher RP. Four methods were investigated, including feature selection (case A) and feature extraction (case B) with traditional machine learning methods, a VAE-MLP joint architecture (case C) with deep learning and lastly, the combination of feature selection and joint architecture (case D). For feature selection, Random forest (RF), Support Vector Machine (SVM), and multilayer perceptron (MLP) were implemented to select relevant features. Specifically, each method was run for multiple times to rank features within several cross-validated (CV) resampled sets. A collection of ranking lists were then aggregated by top 5% and Kemeny graph methods to identify the final ranking for prediction. A synthetic minority oversampling technique was applied to correct for class imbalance during this process. For deep learning, a VAE-MLP joint architecture where a VAE aimed for dimensionality reduction and an MLP aimed for classification was developed. In this architecture, reconstruction loss and prediction loss were combined into a single loss function to realize simultaneous training and weights were assigned to different classes to mitigate class imbalance. To evaluate the prediction performance and conduct comparisons, the area under receiver operating characteristic curves (AUCs) were performed for nested CVs for both handcrafted feature selections and the deep learning approach. The significance of differences in AUCs was assessed using the DeLong test of U-statistics.

RESULTS

An MLP-based method using weight pruning (WP) feature selection yielded the best performance among the different hand-crafted feature selection methods (case A), reaching an AUC of 0.804 (95% CI: 0.761-0.823) with 29 top features. A VAE-MLP joint architecture (case C) achieved a comparable but slightly lower AUC of 0.781 (95% CI: 0.737-0.808) with the size of latent dimension being 2. The combination of handcrafted features (case A) and latent representation (case D) achieved a significant AUC improvement of 0.831 (95% CI: 0.805-0.863) with 22 features (P-value = 0.000642 compared with handcrafted features only (Case A) and P-value = 0.000453 compared to VAE alone (Case C)) with an MLP classifier.

CONCLUSION

The potential for combination of traditional machine learning methods and deep learning VAE techniques has been demonstrated for dealing with limited datasets in modeling radiotherapy toxicities. Specifically, latent variables from a VAE-MLP joint architecture are able to complement handcrafted features for the prediction of RP and improve prediction over either method alone.

EEG-Based Brain-Computer Interfaces Using Motor-Imagery: Techniques and Challenges

Electroencephalography (EEG)-based brain-computer interfaces (BCIs), particularly those using motor-imagery (MI) data, have the potential to become groundbreaking technologies in both clinical and entertainment settings. MI data is generated when a subject imagines the movement of a limb. This paper reviews state-of-the-art signal processing techniques for MI EEG-based BCIs, with a particular focus on the feature extraction, feature selection and classification techniques used. It also summarizes the main applications of EEG-based BCIs, particularly those based on MI data, and finally presents a detailed discussion of the most prevalent challenges impeding the development and commercialization of EEG-based BCIs.

Explaining Support Vector Machines: A Color Based Nomogram

Problem setting

Support vector machines (SVMs) are very popular tools for classification, regression and other problems. Due to the large choice of kernels they can be applied with, a large variety of data can be analysed using these tools. Machine learning thanks its popularity to the good performance of the resulting models. However, interpreting the models is far from obvious, especially when non-linear kernels are used. Hence, the methods are used as black boxes. As a consequence, the use of SVMs is less supported in areas where interpretability is important and where people are held responsible for the decisions made by models.

Objective

In this work, we investigate whether SVMs using linear, polynomial and RBF kernels can be explained such that interpretations for model-based decisions can be provided. We further indicate when SVMs can be explained and in which situations interpretation of SVMs is (hitherto) not possible. Here, explainability is defined as the ability to produce the final decision based on a sum of contributions which depend on one single or at most two input variables.

Results

Our experiments on simulated and real-life data show that explainability of an SVM depends on the chosen parameter values (degree of polynomial kernel, width of RBF kernel and regularization constant). When several combinations of parameter values yield the same cross-validation performance, combinations with a lower polynomial degree or a larger kernel width have a higher chance of being explainable.

Conclusions

This work summarizes SVM classifiers obtained with linear, polynomial and RBF kernels in a single plot. Linear and polynomial kernels up to the second degree are represented exactly. For other kernels an indication of the reliability of the approximation is presented. The complete methodology is available as an R package and two apps and a movie are provided to illustrate the possibilities offered by the method.

[A data mining method for breast cancer identification based on a selection of variables]

In the majority of countries, breast cancer among women is highly prevalent. If diagnosed in the early stages, there is a high probability of a cure. Several statistical-based approaches have been developed to assist in early breast cancer detection. This paper presents a method for selection of variables for the classification of cases into two classes, benign or malignant, based on cytopathological analysis of breast cell samples of patients. The variables are ranked according to a new index of importance of variables that combines the weighting importance of Principal Component Analysis and the explained variance based on each retained component. Observations from the test sample are categorized into two classes using the k-Nearest Neighbor algorithm and Discriminant Analysis, followed by elimination of the variable with the index of lowest importance. The subset with the highest accuracy is used to classify observations in the test sample. When applied to the Wisconsin Breast Cancer Database, the proposed method led to average of 97.77% in classification accuracy while retaining an average of 5.8 variables.

ROUGH MORPHOLOGY HYBRID APPROACH FOR MAMMOGRAPHY IMAGE CLASSIFICATION AND PREDICTION

The objective of this research is to illustrate how rough sets can be successfully integrated with mathematical morphology and provide a more effective hybrid approach to resolve medical imaging problems. Hybridization of rough sets and mathematical morphology techniques has been applied to depict their ability to improve the classification of breast cancer images into two outcomes: malignant and benign cancer. Algorithms based on mathematical morphology are first applied to enhance the contrast of the whole original image; to extract the region of interest (ROI) and to enhance the edges surrounding that region. Then, features are extracted characterizing the underlying texture of the ROI by using the gray-level co-occurrence matrix. The rough set approach to attribute reduction and rule generation is further presented. Finally, rough morphology is designed for discrimination of different ROI to test whether they represent malignant cancer or benign cancer. To evaluate performance of the presented rough morphology approach, we tested different mammogram images. The experimental results illustrate that the overall performance in locating optimal orientation offered by the proposed approach is high compared with other hybrid systems such as rough-neural and rough-fuzzy systems.

Computerized three-class classification of MRI-based prognostic markers for breast cancer

The purpose of this study is to investigate whether computerized analysis using three-class Bayesian artificial neural network (BANN) feature selection and classification can characterize tumor grades (grade 1, grade 2 and grade 3) of breast lesions for prognostic classification on DCE-MRI. A database of 26 IDC grade 1 lesions, 86 IDC grade 2 lesions and 58 IDC grade 3 lesions was collected. The computer automatically segmented the lesions, and kinetic and morphological lesion features were automatically extracted. The discrimination tasks-grade 1 versus grade 3, grade 2 versus grade 3, and grade 1 versus grade 2 lesions-were investigated. Step-wise feature selection was conducted by three-class BANNs. Classification was performed with three-class BANNs using leave-one-lesion-out cross-validation to yield computer-estimated probabilities of being grade 3 lesion, grade 2 lesion and grade 1 lesion. Two-class ROC analysis was used to evaluate the performances. We achieved AUC values of 0.80 ± 0.05, 0.78 ± 0.05 and 0.62 ± 0.05 for grade 1 versus grade 3, grade 1 versus grade 2, and grade 2 versus grade 3, respectively. This study shows the potential for (1) applying three-class BANN feature selection and classification to CADx and (2) expanding the role of DCE-MRI CADx from diagnostic to prognostic classification in distinguishing tumor grades.

Ant Colony Optimization for Feature Selection Involving Effective Local Search

This paper proposes an effective algorithm for feature selection (ACOFS) that uses a global Ant Colony Optimization algorithm (ACO) search strategy. To make ACO effective in feature selection, our proposed algorithm uses an effective local search in selecting significant features. The novelty of ACOFS lies in its effective balance between ant exploration and exploitation using new pheromone update and heuristic information computation rules to generate a subset of a smaller number of significant features. We evaluate algorithm performance using seven real-world benchmark classification datasets. Results show that ACOFS generates smaller subsets of significant features with improved classification accuracy.

Recent advances in electronic tongues

This minireview describes the main developments of electronic tongues (e-tongues) and taste sensors in recent years, with a summary of the principles of detection and materials used in the sensing units. E-tongues are sensor arrays capable of distinguishing very similar liquids employing the concept of global selectivity, where the difference in the electrical response of different materials serves as a fingerprint for the analysed sample. They have been widely used for the analysis of wines, fruit juices, coffee, milk and beverages, in addition to the detection of trace amounts of impurities or pollutants in waters. Among the various principles of detection, electrochemical measurements and impedance spectroscopy are the most prominent. With regard to the materials for the sensing units, in most cases use is made of ultrathin films produced in a layer-by-layer fashion to yield higher sensitivity with the advantage of control of the film molecular architecture. The concept of e-tongues has been extended to biosensing by using sensing units capable of molecular recognition, as in films with immobilized antigens or enzymes with specific recognition for clinical diagnosis. Because the identification of samples is basically a classification task, there has been a trend to use artificial intelligence and information visualization methods to enhance the performance of e-tongues.

Fault classification on vibration data with wavelet based feature selection scheme

Fault classification based upon vibration measurements is an essential building block of a conditional based health usage monitoring system. Multiple sensors are incorporated to assure the redundancy and to achieve the desired reliability and accuracy. The shortcoming of using multiple sensors is the need to deal with a high dimensional feature set, a computationally expensive task in classification. It is vital to reduce the feature dimension via an effective feature extraction and feature selection algorithm. A simple wavelet based feature selection scheme is proposed herein, uniquely built by local discriminant bases and genetic optimization. This scheme overcomes the disadvantages faced by the existing feature selection methods by producing a generic feature set, reducing the dimensionality of features, and requiring no prior information of the problem domain. The proposed feature selection scheme is based upon the strategy of "divide and conquer" that significantly reduce the computation time without compromising the classification performance. The simulation results show the proposed feature selection scheme provides at least 65% reduction of the total number of features at no cost of the classification accuracy.

Genetic programming for simultaneous feature selection and classifier design

This paper presents an online feature selection algorithm using genetic programming (GP). The proposed GP methodology simultaneously selects a good subset of features and constructs a classifier using the selected features. For a c-class problem, it provides a classifier having c trees. In this context, we introduce two new crossover operations to suit the feature selection process. As a byproduct, our algorithm produces a feature ranking scheme. We tested our method on several data sets having dimensions varying from 4 to 7129. We compared the performance of our method with results available in the literature and found that the proposed method produces consistently good results. To demonstrate the robustness of the scheme, we studied its effectiveness on data sets with known (synthetically added) redundant/bad features.

GMDH-based feature ranking and selection for improved classification of medical data

Medical applications are often characterized by a large number of disease markers and a relatively small number of data records. We demonstrate that complete feature ranking followed by selection can lead to appreciable reductions in data dimensionality, with significant improvements in the implementation and performance of classifiers for medical diagnosis. We describe a novel approach for ranking all features according to their predictive quality using properties unique to learning algorithms based on the group method of data handling (GMDH). An abductive network training algorithm is repeatedly used to select groups of optimum predictors from the feature set at gradually increasing levels of model complexity specified by the user. Groups selected earlier are better predictors. The process is then repeated to rank features within individual groups. The resulting full feature ranking can be used to determine the optimum feature subset by starting at the top of the list and progressively including more features until the classification error rate on an out-of-sample evaluation set starts to increase due to overfitting. The approach is demonstrated on two medical diagnosis datasets (breast cancer and heart disease) and comparisons are made with other feature ranking and selection methods. Receiver operating characteristics (ROC) analysis is used to compare classifier performance. At default model complexity, dimensionality reduction of 22 and 54% could be achieved for the breast cancer and heart disease data, respectively, leading to improvements in the overall classification performance. For both datasets, considerable dimensionality reduction introduced no significant reduction in the area under the ROC curve. GMDH-based feature selection results have also proved effective with neural network classifiers.