Feature selection for MLP neural network: the use of random permutation of probabilistic outputs

A Novel Hybrid Approach for Classifying Osteosarcoma Using Deep Feature Extraction and Multilayer Perceptron

Osteosarcoma is the most common type of bone cancer that tends to occur in teenagers and young adults. Due to crowded context, inter-class similarity, inter-class variation, and noise in H&E-stained (hematoxylin and eosin stain) histology tissue, pathologists frequently face difficulty in osteosarcoma tumor classification. In this paper, we introduced a hybrid framework for improving the efficiency of three types of osteosarcoma tumor (nontumor, necrosis, and viable tumor) classification by merging different types of CNN-based architectures with a multilayer perceptron (MLP) algorithm on the WSI (whole slide images) dataset. We performed various kinds of preprocessing on the WSI images. Then, five pre-trained CNN models were trained with multiple parameter settings to extract insightful features via transfer learning, where convolution combined with pooling was utilized as a feature extractor. For feature selection, a decision tree-based RFE was designed to recursively eliminate less significant features to improve the model generalization performance for accurate prediction. Here, a decision tree was used as an estimator to select the different features. Finally, a modified MLP classifier was employed to classify binary and multiclass types of osteosarcoma under the five-fold CV to assess the robustness of our proposed hybrid model. Moreover, the feature selection criteria were analyzed to select the optimal one based on their execution time and accuracy. The proposed model achieved an accuracy of 95.2% for multiclass classification and 99.4% for binary classification. Experimental findings indicate that our proposed model significantly outperforms existing methods; therefore, this model could be applicable to support doctors in osteosarcoma diagnosis in clinics. In addition, our proposed model is integrated into a web application using the FastAPI web framework to provide a real-time prediction.

A two-stage hybrid gene selection algorithm combined with machine learning models to predict the rupture status in intracranial aneurysms

An IA is an abnormal swelling of cerebral vessels, and a subset of these IAs can rupture causing aneurysmal subarachnoid hemorrhage (aSAH), often resulting in death or severe disability. Few studies have used an appropriate method of feature selection combined with machine learning by analyzing transcriptomic sequencing data to identify new molecular biomarkers. Following gene ontology (GO) and enrichment analysis, we found that the distinct status of IAs could lead to differential innate immune responses using all 913 differentially expressed genes, and considering that there are numerous irrelevant and redundant genes, we propose a mixed filter- and wrapper-based feature selection. First, we used the Fast Correlation-Based Filter (FCBF) algorithm to filter a large number of irrelevant and redundant genes in the raw dataset, and then used the wrapper feature selection method based on the he Multi-layer Perceptron (MLP) neural network and the Particle Swarm Optimization (PSO), accuracy (ACC) and mean square error (MSE) were then used as the evaluation criteria. Finally, we constructed a novel 10-gene signature (YIPF1, RAB32, WDR62, ANPEP, LRRCC1, AADAC, GZMK, WBP2NL, PBX1, and TOR1B) by the proposed two-stage hybrid algorithm FCBF-MLP-PSO and used different machine learning models to predict the rupture status in IAs. The highest ACC value increased from 0.817 to 0.919 (12.5% increase), the highest area under ROC curve (AUC) value increased from 0.87 to 0.94 (8.0% increase), and all evaluation metrics improved by approximately 10% after being processed by our proposed gene selection algorithm. Therefore, these 10 informative genes used to predict rupture status of IAs can be used as complements to imaging examinations in the clinic, meanwhile, this selected gene signature also provides new targets and approaches for the treatment of ruptured IAs.

Analyzing the serum of hemodialysis patients with end-stage chronic kidney disease by means of the combination of SERS and machine learning

The aim of this paper is a multivariate analysis of SERS characteristics of serum in hemodialysis patients, which includes constructing classification models (PLS-DA, CNN) by the presence/absence of end-stage chronic kidney disease (CKD) with dialysis and determining the most informative spectral bands for identifying dialysis patients by variable importance distribution. We found the spectral bands that are informative for detecting the hemodialysis patients: the 641 cm^-1, 724 cm^-1, 1094 cm^-1 and 1393 cm^-1 bands are associated with the degree of kidney function inhibition; and the 1001 cm^-1 band is able to demonstrate the distinctive features of hemodialysis patients with end-stage CKD.

Classification of skin cancer using convolutional neural networks analysis of Raman spectra

BACKGROUND AND OBJECTIVE

Skin cancer is the most common malignancy in whites accounting for about one third of all cancers diagnosed per year. Portable Raman spectroscopy setups for skin cancer "optical biopsy" are utilized to detect tumors based on their spectral features caused by the comparative presence of different chemical components. However, low signal-to-noise ratio in such systems may prevent accurate tumors classification. Thus, there is a challenge to develop methods for efficient skin tumors classification.

METHODS

We compare the performance of convolutional neural networks and the projection on latent structures with discriminant analysis for discriminating skin cancer using the analysis of Raman spectra with a high autofluorescence background stimulated by a 785 nm laser. We have registered the spectra of 617 cases of skin neoplasms (615 patients, 70 melanomas, 122 basal cell carcinomas, 12 squamous cell carcinomas and 413 benign tumors) in vivo with a portable Raman setup and created classification models both for convolutional neural networks and projection on latent structures approaches. To check the classification models stability, a 10-fold cross-validation was performed for all created models. To avoid models overfitting, the data was divided into a training set (80% of spectral dataset) and a test set (20% of spectral dataset).

RESULTS

The results for different classification tasks demonstrate that the convolutional neural networks significantly (p<0.01) outperforms the projection on latent structures. For the convolutional neural networks implementation we obtained ROC AUCs of 0.96 (0.94 - 0.97; 95% CI), 0.90 (0.85-0.94; 95% CI), and 0.92 (0.87 - 0.97; 95% CI) for classifying a) malignant vs benign tumors, b) melanomas vs pigmented tumors and c) melanomas vs seborrheic keratosis respectively.

CONCLUSIONS

The performance of the convolutional neural networks classification of skin tumors based on Raman spectra analysis is higher or comparable to the accuracy provided by trained dermatologists. The increased accuracy with the convolutional neural networks implementation is due to a more precise accounting of low intensity Raman bands in the intense autofluorescence background. The achieved high performance of skin tumors classifications with convolutional neural networks analysis opens a possibility for wide implementation of Raman setups in clinical setting.

Development of a Machine Learning-Based Predictive Model for Lung Metastasis in Patients With Ewing Sarcoma

BACKGROUND

This study aimed to develop and validate machine learning (ML)-based prediction models for lung metastasis (LM) in patients with Ewing sarcoma (ES), and to deploy the best model as an open access web tool.

METHODS

We retrospectively analyzed data from the Surveillance Epidemiology and End Results (SEER) Database from 2010 to 2016 and from four medical institutions to develop and validate predictive models for LM in patients with ES. Patient data from the SEER database was used as the training group (n = 929). Using demographic and clinicopathologic variables six ML-based models for predicting LM were developed, and internally validated using 10-fold cross validation. All ML-based models were subsequently externally validated using multiple data from four medical institutions (the validation group, n = 51). The predictive power of the models was evaluated by the area under receiver operating characteristic curve (AUC). The best-performing model was used to produce an online tool for use by clinicians to identify ES patients at risk from lung metastasis, to improve decision making and optimize individual treatment.

RESULTS

The study cohort consisted of 929 patients from the SEER database and 51 patients from multiple medical centers, a total of 980 ES patients. Of these, 175 (18.8%) had lung metastasis. Multivariate logistic regression analysis was performed with survival time, T-stage, N-stage, surgery, and bone metastasis providing the independent predictive factors of LM. The AUC value of six predictive models ranged from 0.585 to 0.705. The Random Forest (RF) model (AUC = 0.705) using 4 variables was identified as the best predictive model of LM in ES patients and was employed to construct an online tool to assist clinicians in optimizing patient treatment. (https://share.streamlit.io/liuwencai123/es_lm/main/es_lm.py).

CONCLUSIONS

Machine learning were found to have utility for predicting LM in patients with Ewing sarcoma, and the RF model gave the best performance. The accessibility of the predictive model as a web-based tool offers clear opportunities for improving the personalized treatment of patients with ES.

Prediction of premature all-cause mortality in patients receiving peritoneal dialysis using modified artificial neural networks

Premature all-cause mortality is high in patients receiving peritoneal dialysis (PD). The accurate and early prediction of mortality is critical and difficult. Three prediction models, the logistic regression (LR) model, artificial neural network (ANN) classic model and a new structured ANN model (ANN mixed model), were constructed and evaluated using a receiver operating characteristic (ROC) curve analysis. The permutation feature importance was used to interpret the important features in the ANN models. Eight hundred fifty-nine patients were enrolled in the study. The LR model performed slightly better than the other two ANN models on the test dataset; however, in the total dataset, the ANN models fit much better. The ANN mixed model showed the best prediction performance, with area under the ROC curves (AUROCs) of 0.8 and 0.79 for the 6-month and 12-month datasets. Our study showed that age, diastolic blood pressure (DBP), and low-density lipoprotein cholesterol (LDL-c) levels were common risk factors for premature mortality in patients receiving PD. Our ANN mixed model had incomparable advantages in fitting the overall data characteristics, and age is a steady risk factor for premature mortality in patients undergoing PD. Otherwise, DBP and LDL-c levels should receive more attention for all-cause mortality during follow-up.

Algorithm Supported Induction for Building Theory: How Can We Use Prediction Models to Theorize?

Across many fields of social science, machine learning (ML) algorithms are rapidly advancing research as tools to support traditional hypothesis testing research (e.g., through data reduction and automation of data coding or for improving matching on observable features of a phenomenon or constructing instrumental variables). In this paper, we argue that researchers are yet to recognize the value of ML techniques for theory building from data. This may be in part because of scholars’ inherent distaste for predictions without explanations that ML algorithms are known to produce. However, precisely because of this property, we argue that ML techniques can be very useful in theory construction during a key step of inductive theorizing—pattern detection. ML can facilitate algorithm supported induction, yielding conclusions about patterns in data that are likely to be robustly replicable by other analysts and in other samples from the same population. These patterns can then be used as inputs to abductive reasoning for building or developing theories that explain them. We propose that algorithm-supported induction is valuable for researchers interested in using quantitative data to both develop and test theories in a transparent and reproducible manner, and we illustrate our arguments using simulations.

Predicting Long-Term Mortality in TAVI Patients Using Machine Learning Techniques

Background: Whereas transcatheter aortic valve implantation (TAVI) has become the gold standard for aortic valve stenosis treatment in high-risk patients, it has recently been extended to include intermediate risk patients. However, the mortality rate at 5 years is still elevated. The aim of the present study was to develop a novel machine learning (ML) approach able to identify the best predictors of 5-year mortality after TAVI among several clinical and echocardiographic variables, which may improve the long-term prognosis. Methods: We retrospectively enrolled 471 patients undergoing TAVI. More than 80 pre-TAVI variables were collected and analyzed through different feature selection processes, which allowed for the identification of several variables with the highest predictive value of mortality. Different ML models were compared. Results: Multilayer perceptron resulted in the best performance in predicting mortality at 5 years after TAVI, with an area under the curve, positive predictive value, and sensitivity of 0.79, 0.73, and 0.71, respectively. Conclusions: We presented an ML approach for the assessment of risk factors for long-term mortality after TAVI to improve clinical prognosis. Fourteen potential predictors were identified with the organic mitral regurgitation (myxomatous or calcific degeneration of the leaflets and/or annulus) which showed the highest impact on 5 years mortality.

A Two-Step Integrated MLP-GTWR Method to Estimate 1 km Land Surface Temperature with Complete Spatial Coverage in Humid, Cloudy Regions

There is an increasing demand for a land surface temperature (LST) dataset with both fine spatial and temporal resolutions due to the key role of LST in the Earth’s land–atmosphere system. Currently, the technique most commonly used to meet the demand is thermal infrared (TIR) remote sensing. However, cloud contamination interferes with TIR transmission through the atmosphere, limiting the potential of space-borne TIR sensors to provide the LST with complete spatio-temporal coverage. To solve this problem, we developed a two-step integrated method to: (i) estimate the 10-km LST with a high spatial coverage from passive microwave (PMW) data using the multilayer perceptron (MLP) model; and (ii) downscale the LST to 1 km and fill the gaps based on the geographically and temporally weighted regression (GTWR) model. Finally, the 1-km all-weather LST for cloudy pixels was fused with Aqua MODIS clear-sky LST via bias correction. This method was applied to produce the all-weather LST products for both daytime and nighttime during the years 2013–2018 in South China. The evaluations showed that the accuracy of the reproduced LST on cloudy days was comparable to that of the MODIS LST in terms of mean absolute error (2.29–2.65 K), root mean square error (2.92–3.25 K), and coefficients of determination (0.82–0.92) against the in situ measurements at four flux stations and ten automatic meteorological stations with various land cover types. The spatial and temporal analysis showed that the MLP-GTWR LST were highly consistent with the MODIS, in situ, and ERA5-Land LST, with the satisfactory ability to present the LST pattern under cloudy conditions. In addition, the MLP-GTWR method outperformed a gap-filling method and another TIR-PMW integrated method due to the local strategy in MLP and the consideration of temporal non-stationarity relationship in GTWR. Therefore, the test of the developed method in the frequently cloudy South China indicates the efficient potential for further application to other humid regions to generate the LST under cloudy condition.

Predicting Poor Outcome Before Endovascular Treatment in Patients With Acute Ischemic Stroke

Background: Although endovascular treatment (EVT) has greatly improved outcomes in acute ischemic stroke, still one third of patients die or remain severely disabled after stroke. If we could select patients with poor clinical outcome despite EVT, we could prevent futile treatment, avoid treatment complications, and further improve stroke care. We aimed to determine the accuracy of poor functional outcome prediction, defined as 90-day modified Rankin Scale (mRS) score ≥5, despite EVT treatment. Methods: We included 1,526 patients from the MR CLEAN Registry, a prospective, observational, multicenter registry of ischemic stroke patients treated with EVT. We developed machine learning prediction models using all variables available at baseline before treatment. We optimized the models for both maximizing the area under the curve (AUC), reducing the number of false positives. Results: From 1,526 patients included, 480 (31%) of patients showed poor outcome. The highest AUC was 0.81 for random forest. The highest area under the precision recall curve was 0.69 for the support vector machine. The highest achieved specificity was 95% with a sensitivity of 34% for neural networks, indicating that all models contained false positives in their predictions. From 921 mRS 0-4 patients, 27-61 (3-6%) were incorrectly classified as poor outcome. From 480 poor outcome patients in the registry, 99-163 (21-34%) were correctly identified by the models. Conclusions: All prediction models showed a high AUC. The best-performing models correctly identified 34% of the poor outcome patients at a cost of misclassifying 4% of non-poor outcome patients. Further studies are necessary to determine whether these accuracies are reproducible before implementation in clinical practice.

Statistical analysis and accident prediction models leading to pedestrian injuries and deaths on rural roads in Iran

The purpose of this study was to develop models to predict the severity of pedestrian accidents on rural roads of Guilan, Iran. Therefore, the probability of occurrence of any type of accidents was measured using the accident data from March 2014 to March 2019. Eleven independent variables affecting the severity of pedestrian accidents as well as statistical analysis such as the frequency analysis, Friedman test and factor analysis, and modeling including multiple logistic regression and artificial neural networks using multi-layer perceptron (MLP) and radius basis function (RBF) have been used. Results of modeling and analysis of pedestrian accidents in different methods showed each of the methods depending on their function investigated the severity of accidents with different point of view and had different results. As a result, putting the output results together, the best measures can be suggested to increase the safety of pedestrians on the rural roads of Guilan.

Combining handcrafted features with latent variables in machine learning for prediction of radiation-induced lung damage

PURPOSE

There has been burgeoning interest in applying machine learning methods for predicting radiotherapy outcomes. However, the imbalanced ratio of a large number of variables to a limited sample size in radiation oncology constitutes a major challenge. Therefore, dimensionality reduction methods can be a key to success. The study investigates and contrasts the application of traditional machine learning methods and deep learning approaches for outcome modeling in radiotherapy. In particular, new joint architectures based on variational autoencoder (VAE) for dimensionality reduction are presented and their application is demonstrated for the prediction of lung radiation pneumonitis (RP) from a large-scale heterogeneous dataset.

METHODS

A large-scale heterogeneous dataset containing a pool of 230 variables including clinical factors (e.g., dose, KPS, stage) and biomarkers (e.g., single nucleotide polymorphisms (SNPs), cytokines, and micro-RNAs) in a population of 106 nonsmall cell lung cancer (NSCLC) patients who received radiotherapy was used for modeling RP. Twenty-two patients had grade 2 or higher RP. Four methods were investigated, including feature selection (case A) and feature extraction (case B) with traditional machine learning methods, a VAE-MLP joint architecture (case C) with deep learning and lastly, the combination of feature selection and joint architecture (case D). For feature selection, Random forest (RF), Support Vector Machine (SVM), and multilayer perceptron (MLP) were implemented to select relevant features. Specifically, each method was run for multiple times to rank features within several cross-validated (CV) resampled sets. A collection of ranking lists were then aggregated by top 5% and Kemeny graph methods to identify the final ranking for prediction. A synthetic minority oversampling technique was applied to correct for class imbalance during this process. For deep learning, a VAE-MLP joint architecture where a VAE aimed for dimensionality reduction and an MLP aimed for classification was developed. In this architecture, reconstruction loss and prediction loss were combined into a single loss function to realize simultaneous training and weights were assigned to different classes to mitigate class imbalance. To evaluate the prediction performance and conduct comparisons, the area under receiver operating characteristic curves (AUCs) were performed for nested CVs for both handcrafted feature selections and the deep learning approach. The significance of differences in AUCs was assessed using the DeLong test of U-statistics.

RESULTS

An MLP-based method using weight pruning (WP) feature selection yielded the best performance among the different hand-crafted feature selection methods (case A), reaching an AUC of 0.804 (95% CI: 0.761-0.823) with 29 top features. A VAE-MLP joint architecture (case C) achieved a comparable but slightly lower AUC of 0.781 (95% CI: 0.737-0.808) with the size of latent dimension being 2. The combination of handcrafted features (case A) and latent representation (case D) achieved a significant AUC improvement of 0.831 (95% CI: 0.805-0.863) with 22 features (P-value = 0.000642 compared with handcrafted features only (Case A) and P-value = 0.000453 compared to VAE alone (Case C)) with an MLP classifier.

CONCLUSION

The potential for combination of traditional machine learning methods and deep learning VAE techniques has been demonstrated for dealing with limited datasets in modeling radiotherapy toxicities. Specifically, latent variables from a VAE-MLP joint architecture are able to complement handcrafted features for the prediction of RP and improve prediction over either method alone.

A Scoring Scheme for Online Feature Selection: Simulating Model Performance Without Retraining

Increasing the number of features increases the complexity of a model even if the additional feature does not improve its decision-making capacity. Irrelevant features may also cause overfitting and reduce interpretability of the concerned model. It is, therefore, important that the features are optimally selected before a model is built. In the case of online learning, new instances are periodically discovered, and the respective model is tactically retrained as required. Similarly, there are many real-life situations where hundreds of new features are discovered periodically, and the existing model needs to be retrained or tested for its performance improvement. Supervised selection of feature subset usually requires creation of multiple suboptimal models, thus incurring time-intensive computations. Unsupervised selections, although faster, largely rely on some subjective definition of feature relevance. In this paper, we introduce a score that accurately determines the importance of the features. The proposed score is appropriate for online feature selection scenarios for its low time complexity and ability to interpret performance improvement of the current model after the addition of a new feature, without invoking a retraining.

Airport take-off noise assessment aimed at identify responsible aircraft classes

Assessment of aircraft noise is an important task of nowadays airports in order to fight environmental noise pollution given the recent discoveries on the exposure negative effects on human health. Noise monitoring and estimation around airports mostly use aircraft noise signals only for computing statistical indicators and depends on additional data sources so as to determine required inputs such as the aircraft class responsible for noise pollution. In this sense, the noise monitoring and estimation systems have been tried to improve by creating methods for obtaining more information from aircraft noise signals, especially real-time aircraft class recognition. Consequently, this paper proposes a multilayer neural-fuzzy model for aircraft class recognition based on take-off noise signal segmentation. It uses a fuzzy inference system to build a final response for each class p based on the aggregation of K parallel neural networks outputs Op(k) with respect to Linear Predictive Coding (LPC) features extracted from K adjacent signal segments. Based on extensive experiments over two databases with real-time take-off noise measurements, the proposed model performs better than other methods in literature, particularly when aircraft classes are strongly correlated to each other. A new strictly cross-checked database is introduced including more complex classes and real-time take-off noise measurements from modern aircrafts. The new model is at least 5% more accurate with respect to previous database and successfully classifies 87% of measurements in the new database.

Learning Subspace-Based RBFNN Using Coevolutionary Algorithm for Complex Classification Tasks

Many real-world classification problems are characterized by samples of a complex distribution in the input space. The classification accuracy is determined by intrinsic properties of all samples in subspaces of features. This paper proposes a novel algorithm for the construction of radial basis function neural network (RBFNN) classifier based on subspace learning. In this paper, feature subspaces are obtained for every hidden node of the RBFNN during the learning process. The connection weights between the input layer and the hidden layer are adjusted to produce various subspaces with dominative features for different hidden nodes. The network structure and dominative features are encoded in two subpopulations that are cooperatively coevolved using the coevolutionary algorithm to achieve a better global optimality for the estimated RBFNN. Experimental results illustrate that the proposed algorithm is able to obtain RBFNN models with both better classification accuracy and simpler network structure when compared with other learning algorithms. Thus, the proposed model provides a more flexible and efficient approach to complex classification tasks by employing the local characteristics of samples in subspaces.

Global sensitivity analysis approach for input selection and system identification purposes--a new framework for feedforward neural networks

A new algorithm for the selection of input variables of neural network is proposed. This new method, applied after the training stage, ranks the inputs according to their importance in the variance of the model output. The use of a global sensitivity analysis technique, extended Fourier amplitude sensitivity test, gives the total sensitivity index for each variable, which allows for the ranking and the removal of the less relevant inputs. Applied to some benchmarking problems in the field of features selection, the proposed approach shows good agreement in keeping the relevant variables. This new method is a useful tool for removing superfluous inputs and for system identification.