Feature Selection Using a Neural Network With Group Lasso Regularization and Controlled Redundancy

Convergence Analysis of Online Gradient Method for High-Order Neural Networks and Their Sparse Optimization

In this article, we investigate the boundedness and convergence of the online gradient method with the smoothing group regularization for the sigma-pi-sigma neural network (SPSNN). This enhances the sparseness of the network and improves its generalization ability. For the original group regularization, the error function is nonconvex and nonsmooth, which can cause oscillation of the error function. To ameliorate this drawback, we propose a simple and effective smoothing technique, which can effectively eliminate the deficiency of the original group regularization. The group regularization effectively optimizes the network structure from two aspects redundant hidden nodes tending to zero and redundant weights of surviving hidden nodes in the network tending to zero. This article shows the strong and weak convergence results for the proposed method and proves the boundedness of weights. Experiment results clearly demonstrate the capability of the proposed method and the effectiveness of redundancy control. The simulation results are observed to support the theoretical results.

Convergence analysis of sparse TSK fuzzy systems based on spectral Dai-Yuan conjugate gradient and application to high-dimensional feature selection

Dealing with high-dimensional problems has always been a key and challenging issue in the field of fuzzy systems. Traditional Takagi-Sugeno-Kang (TSK) fuzzy systems face the challenges of the curse of dimensionality and computational complexity when applied to high-dimensional data. To overcome these challenges, this paper proposes a novel approach for optimizing TSK fuzzy systems by integrating the spectral Dai-Yuan conjugate gradient (SDYCG) algorithm and the smoothing group L0 regularization technique. This method aims to address the challenges faced by TSK fuzzy systems in handling high-dimensional problems. The smoothing group L0 regularization technique is employed to introduce sparsity, select relevant features, and improve the generalization ability of the model. The SDYCG algorithm effectively accelerates convergence and enhances the learning performance of the network. Furthermore, we prove the weak convergence and strong convergence of the new algorithm under the strong Wolfe criterion, which means that the gradient norm of the error function with respect to the weight vector converges to zero, and the weight sequence approaches a fixed point.

Learning Accurate Label-Specific Features From Partially Multilabeled Data

Feature selection is an effective dimensionality reduction technique, which can speed up an algorithm and improve model performance such as predictive accuracy and result comprehensibility. The study of selecting label-specific features for each class label has attracted considerable attention since each class label might be determined by some inherent characteristics, where precise label information is required to guide label-specific feature selection. However, obtaining noise-free labels is quite difficult and impractical. In reality, each instance is often annotated by a candidate label set that comprises multiple ground-truth labels and other false-positive labels, termed partial multilabel (PML) learning scenario. Here, false-positive labels concealed in a candidate label set might induce the selection of false label-specific features while masking the intrinsic label correlations, which misleads the selection of relevant features and compromises the selection performance. To address this issue, a novel two-stage partial multilabel feature selection (PMLFS) approach is proposed, which elicits credible labels to guide accurate label-specific feature selection. First, the label confidence matrix is learned to help elicit ground-truth labels from the candidate label set via the label structure reconstruction strategy, each element of which indicates how likely a class label is ground truth. After that, based on distilled credible labels, a joint selection model, including label-specific feature learner and common feature learner, is designed to learn accurate label-specific features to each class label and common features for all class labels. Besides, label correlations are fused into the features selection process to facilitate the generation of an optimal feature subset. Extensive experimental results clearly validate the superiority of the proposed approach.

Surrogate-Assisted and Filter-Based Multiobjective Evolutionary Feature Selection for Deep Learning

Feature selection (FS) for deep learning prediction models is a difficult topic for researchers to tackle. Most of the approaches proposed in the literature consist of embedded methods through the use of hidden layers added to the neural network architecture that modify the weights of the units associated with each input attribute so that the worst attributes have less weight in the learning process. Other approaches used for deep learning are filter methods, which are independent of the learning algorithm, which can limit the precision of the prediction model. Wrapper methods are impractical with deep learning due to their high computational cost. In this article, we propose new attribute subset evaluation FS methods for deep learning of the wrapper, filter and wrapper-filter hybrid types, where multiobjective and many-objective evolutionary algorithms are used as search strategies. A novel surrogate-assisted approach is used to reduce the high computational cost of the wrapper-type objective function, while the filter-type objective functions are based on correlation and an adaptation of the reliefF algorithm. The proposed techniques have been applied in a time series forecasting problem of air quality in the Spanish south-east and an indoor temperature forecasting problem in a domotic house, with promising results compared to other FS techniques used in the literature.

A Machine Learning Model for the Prediction of COVID-19 Severity Using RNA-Seq, Clinical, and Co-Morbidity Data

The premise for this study emanated from the need to understand SARS-CoV-2 infections at the molecular level and to develop predictive tools for managing COVID-19 severity. With the varied clinical outcomes observed among infected individuals, creating a reliable machine learning (ML) model for predicting the severity of COVID-19 became paramount. Despite the availability of large-scale genomic and clinical data, previous studies have not effectively utilized multi-modality data for disease severity prediction using data-driven approaches. Our primary goal is to predict COVID-19 severity using a machine-learning model trained on a combination of patients' gene expression, clinical features, and co-morbidity data. Employing various ML algorithms, including Logistic Regression (LR), XGBoost (XG), Naïve Bayes (NB), and Support Vector Machine (SVM), alongside feature selection methods, we sought to identify the best-performing model for disease severity prediction. The results highlighted XG as the superior classifier, with 95% accuracy and a 0.99 AUC (Area Under the Curve), for distinguishing severity groups. Additionally, the SHAP analysis revealed vital features contributing to prediction, including several genes such as COX14, LAMB2, DOLK, SDCBP2, RHBDL1, and IER3-AS1. Notably, two clinical features, the absolute neutrophil count and Viremia Categories, emerged as top contributors. Integrating multiple data modalities has significantly improved the accuracy of disease severity prediction compared to using any single modality. The identified features could serve as biomarkers for COVID-19 prognosis and patient care, allowing clinicians to optimize treatment strategies and refine clinical decision-making processes for enhanced patient outcomes.

The use of longitudinal CT-based radiomics and clinicopathological features predicts the pathological complete response of metastasized axillary lymph nodes in breast cancer

BACKGROUND

Accurate assessment of axillary status after neoadjuvant therapy for breast cancer patients with axillary lymph node metastasis is important for the selection of appropriate subsequent axillary treatment decisions. Our objectives were to accurately predict whether the breast cancer patients with axillary lymph node metastases could achieve axillary pathological complete response (pCR).

METHODS

We collected imaging data to extract longitudinal CT image features before and after neoadjuvant chemotherapy (NAC), analyzed the correlation between radiomics and clinicopathological features, and developed models to predict whether patients with axillary lymph node metastasis can achieve axillary pCR after NAC. The clinical utility of the models was determined via decision curve analysis (DCA). Subgroup analyses were also performed. Then, a nomogram was developed based on the model with the best predictive efficiency and clinical utility and was validated using the calibration plots.

RESULTS

A total of 549 breast cancer patients with metastasized axillary lymph nodes were enrolled in this study. 42 independent radiomics features were selected from LASSO regression to construct a logistic regression model with clinicopathological features (LR radiomics-clinical combined model). The AUC of the LR radiomics-clinical combined model prediction performance was 0.861 in the training set and 0.891 in the testing set. For the HR + /HER2 - , HER2 + , and Triple negative subtype, the LR radiomics-clinical combined model yields the best prediction AUCs of 0.756, 0.812, and 0.928 in training sets, and AUCs of 0.757, 0.777 and 0.838 in testing sets, respectively.

CONCLUSIONS

The combination of radiomics features and clinicopathological characteristics can effectively predict axillary pCR status in NAC breast cancer patients.

Prediction of Cytochrome P450 Inhibition Using a Deep Learning Approach and Substructure Pattern Recognition

Cytochrome P450 (CYP) is a family of enzymes that are responsible for about 75% of all metabolic reactions. Among them, CYP1A2, CYP2C9, CYP2C19, CYP2D6, and CYP3A4 participate in the metabolism of most drugs and mediate many adverse drug reactions. Therefore, it is necessary to estimate the chemical inhibition of Cytochrome P450 enzymes in drug discovery and the food industry. In the past few decades, many computational models have been reported, and some provided good performance. However, there are still several issues that should be resolved for these models, such as single isoform, models with unbalanced performance, lack of structural characteristics analysis, and poor availability. In the present study, the deep learning models based on python using the Keras framework and TensorFlow were developed for the chemical inhibition of each CYP isoform. These models were established based on a large data set containing 85715 compounds extracted from the PubChem bioassay database. On external validation, the models provided good AUC values with 0.97, 0.94, 0.94, 0.96, and 0.94 for CYP1A2, CYP2C9, CYP2C19, CYP2D6, and CYP3A4, respectively. The models can be freely accessed on the Web server named CYPi-DNNpredictor (cypi.sapredictor.cn), and the codes for the model were made open source in the Supporting Information. In addition, we also analyzed the structural characteristics of chemicals with CYP450 inhibition and detected the structural alerts (SAs), which should be responsible for the inhibition. The SAs were also made available online, named CYPi-SAdetector (cypisa.sapredictor.cn). The models can be used as a powerful tool for the prediction of CYP450 inhibitors, and the SAs should provide useful information for the mechanisms of Cytochrome P450 inhibition.

Exploring relevant factors of cognitive impairment in the elderly Chinese population using Lasso regression and Bayesian networks

Older adults are highly susceptible to developing cognitive impairment(CI). Various factors contribute to the prevalence of CI, but the potential relationships among these factors remain unclear. This study aims to explore the relevant factors associated with CI in Chinese older adults and analyze the potential relationships between CI and these factors.We analyzed the data on 6886 older adults aged≥60 from the China Health and Retirement Longitudinal Study (CHARLS) 2018. Lasso regression was initially used to screening variables. Bayesian Networks(BNs) were used to identify the correlates of CI and potential associations between factors. After screening with Lasso regression, 11 variables were finally included in the BNs. The BNs, by establishing a complex network relationship, revealed that age, education, and indoor air pollution were the direct correlates affecting the occurrence of CI in older adults. It also indicated that marital status indirectly influenced CI through age, and residence indirectly linked to CI through two pathways: indoor air pollution and education.Our findings underscore the effectiveness of BNs in unveiling the intricate network linkages among CI and its associated factors, holding promising applications. It can serve as a reference for public health departments to address the prevention of CI in the elderly.

Establishment and validation of a heart failure risk prediction model for elderly patients after coronary rotational atherectomy based on machine learning

Objective

To develop and validate a heart failure risk prediction model for elderly patients after coronary rotational atherectomy based on machine learning methods.

Methods

A retrospective cohort study was conducted to select 303 elderly patients with severe coronary calcification as the study subjects. According to the occurrence of postoperative heart failure, the study subjects were divided into the heart failure group (n = 53) and the non-heart failure group (n = 250). Retrospective collection of clinical data from the study subjects during hospitalization. After processing the missing values in the original data and addressing sample imbalance using Adaptive Synthetic Sampling (ADASYN) method, the final dataset consists of 502 samples: 250 negative samples (i.e., patients not suffering from heart failure) and 252 positive samples (i.e., patients with heart failure). According to a 7:3 ratio, the datasets of 502 patients were randomly divided into a training set (n = 351) and a validation set (n = 151). On the training set, logistic regression (LR), extreme gradient boosting (XGBoost), support vector machine (SVM), and lightweight gradient boosting machine (LightGBM) algorithms were used to construct heart failure risk prediction models; Evaluate model performance on the validation set by calculating the area under the receiver operating characteristic curve (ROC) curve (AUC), sensitivity, specificity, positive predictive value, negative predictive value, F1-score, and prediction accuracy.

Result

A total of 17.49% of 303 patients occured postoperative heart failure. The AUC of LR, XGBoost, SVM, and LightGBM models in the training set were 0.872, 1.000, 0.699, and 1.000, respectively. After 10 fold cross validation, the AUC was 0.863, 0.972, 0.696, and 0.963 in the training set, respectively. Among them, XGBoost had the highest AUC and better predictive performance, while SVM models had the worst performance. The XGBoost model also showed good predictive performance in the validation set (AUC = 0.972, 95% CI [0.951-0.994]). The Shapley additive explanation (SHAP) method suggested that the six characteristic variables of blood cholesterol, serum creatinine, fasting blood glucose, age, triglyceride and NT-proBNP were important positive factors for the occurrence of heart failure, and LVEF was important negative factors for the occurrence of heart failure.

Conclusion

The seven characteristic variables of blood cholesterol, blood creatinine, fasting blood glucose, NT-proBNP, age, triglyceride and LVEF are all important factors affecting the occurrence of heart failure. The prediction model of heart failure risk for elderly patients after CRA based on the XGBoost algorithm is superior to SVM, LightGBM and the traditional LR model. This model could be used to assist clinical decision-making and improve the adverse outcomes of patients after CRA.

Nonlinear Feature Selection Neural Network via Structured Sparse Regularization

Feature selection is an important and effective data preprocessing method, which can remove the noise and redundant features while retaining the relevant and discriminative features in high-dimensional data. In real-world applications, the relationships between data samples and their labels are usually nonlinear. However, most of the existing feature selection models focus on learning a linear transformation matrix, which cannot capture such a nonlinear structure in practice and will degrade the performance of downstream tasks. To address the issue, we propose a novel nonlinear feature selection method to select those most relevant and discriminative features in high-dimensional dataset. Specifically, our method learns the nonlinear structure of high-dimensional data by a neural network with cross entropy loss function, and then using the structured sparsity norm such as l2,p -norm to regularize the weights matrix connecting the input layer and the first hidden layer of the neural network model to learn weight of each feature. Therefore, a structural sparse weights matrix is obtained by conducting nonlinear learning based on a neural network with structured sparsity regularization. Then, we use the gradient descent method to achieve the optimal solution of the proposed model. Evaluating the experimental results on several synthetic datasets and real-world datasets shows the effectiveness and superiority of the proposed nonlinear feature selection model.

Machine learning-based warning model for chronic kidney disease in individuals over 40 years old in underprivileged areas, Shanxi Province

Introduction

Chronic kidney disease (CKD) is a progressive disease with high incidence but early imperceptible symptoms. Since China's rural areas are subject to inadequate medical check-ups and single disease screening programme, it could easily translate into end-stage renal failure. This study aimed to construct an early warning model for CKD tailored to impoverished areas by employing machine learning (ML) algorithms with easily accessible parameters from ten rural areas in Shanxi Province, thereby, promoting a forward shift of treatment time and improving patients' quality of life.

Methods

From April to November 2019, CKD opportunistic screening was carried out in 10 rural areas in Shanxi Province. First, general information, physical examination data, blood and urine specimens were collected from 13,550 subjects. Afterward, feature selection of explanatory variables was performed using LASSO regression, and target datasets were balanced using the SMOTE (synthetic minority over-sampling technique) algorithm, i.e., albuminuria-to-creatinine ratio (ACR) and α1-microglobulin-to-creatinine ratio (MCR). Next, Bagging, Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) were employed for classification of ACR outcomes and MCR outcomes, respectively.

Results

12,330 rural residents were included in this study, with 20 explanatory variables. The cases with increased ACR and increased MCR represented 1,587 (12.8%) and 1,456 (11.8%), respectively. After conducting LASSO, 14 and 15 explanatory variables remained in these two datasets, respectively. Bagging, RF, and XGBoost performed well in classification, with the AUC reaching 0.74, 0.87, 0.87, 0.89 for ACR outcomes and 0.75, 0.88, 0.89, 0.90 for MCR outcomes. The five variables contributing most to the classification of ACR outcomes and MCR outcomes constituted SBP, TG, TC, and Hcy, DBP and age, TG, SBP, Hcy and FPG, respectively. Overall, the machine learning algorithms could emerge as a warning model for CKD.

Conclusion

ML algorithms in conjunction with rural accessible indexes boast good performance in classification, which allows for an early warning model for CKD. This model could help achieve large-scale population screening for CKD in poverty-stricken areas and should be promoted to improve the quality of life and reduce the mortality rate.

Using random forest algorithm for glomerular and tubular injury diagnosis

Objectives

Chronic kidney disease (CKD) is a common chronic condition with high incidence and insidious onset. Glomerular injury (GI) and tubular injury (TI) represent early manifestations of CKD and could indicate the risk of its development. In this study, we aimed to classify GI and TI using three machine learning algorithms to promote their early diagnosis and slow the progression of CKD.

Methods

Demographic information, physical examination, blood, and morning urine samples were first collected from 13,550 subjects in 10 counties in Shanxi province for classification of GI and TI. Besides, LASSO regression was employed for feature selection of explanatory variables, and the SMOTE (synthetic minority over-sampling technique) algorithm was used to balance target datasets, i.e., GI and TI. Afterward, Random Forest (RF), Naive Bayes (NB), and logistic regression (LR) were constructed to achieve classification of GI and TI, respectively.

Results

A total of 12,330 participants enrolled in this study, with 20 explanatory variables. The number of patients with GI, and TI were 1,587 (12.8%) and 1,456 (11.8%), respectively. After feature selection by LASSO, 14 and 15 explanatory variables remained in these two datasets. Besides, after SMOTE, the number of patients and normal ones were 6,165, 6,165 for GI, and 6,165, 6,164 for TI, respectively. RF outperformed NB and LR in terms of accuracy (78.14, 80.49%), sensitivity (82.00, 84.60%), specificity (74.29, 76.09%), and AUC (0.868, 0.885) for both GI and TI; the four variables contributing most to the classification of GI and TI represented SBP, DBP, sex, age and age, SBP, FPG, and GHb, respectively.

Conclusion

RF boasts good performance in classifying GI and TI, which allows for early auxiliary diagnosis of GI and TI, thus facilitating to help alleviate the progression of CKD, and enjoying great prospects in clinical practice.

Learned Block Iterative Shrinkage Thresholding Algorithm for Photothermal Super Resolution Imaging

Block-sparse regularization is already well known in active thermal imaging and is used for multiple-measurement-based inverse problems. The main bottleneck of this method is the choice of regularization parameters which differs for each experiment. We show the benefits of using a learned block iterative shrinkage thresholding algorithm (LBISTA) that is able to learn the choice of regularization parameters, without the need to manually select them. In addition, LBISTA enables the determination of a suitable weight matrix to solve the underlying inverse problem. Therefore, in this paper we present LBISTA and compare it with state-of-the-art block iterative shrinkage thresholding using synthetically generated and experimental test data from active thermography for defect reconstruction. Our results show that the use of the learned block-sparse optimization approach provides smaller normalized mean square errors for a small fixed number of iterations. Thus, this allows us to improve the convergence speed and only needs a few iterations to generate accurate defect reconstruction in photothermal super-resolution imaging.

A Disentangled Representation Based Brain Image Fusion via Group Lasso Penalty

Complementary and redundant relationships inherently exist between multi-modal medical images captured from the same brain. Fusion processes conducted on intermingled representations can cause information distortion and the loss of discriminative modality information. To fully exploit the interdependency between source images for better feature representation and improve the fusion accuracy, we present the multi-modal brain medical image fusion method in a disentangled pipeline under the deep learning framework. A three-branch auto-encoder with two complementary branches and a redundant branch is designed to extract the exclusive modality features and common structure features from input images. Especially, to promote the disentanglement of complement and redundancy, a complementary group lasso penalty is proposed to constrain the extracted feature maps. Then, based on the disentangled representations, different fusion strategies are adopted for complementary features and redundant features, respectively. The experiments demonstrate the superior performance of the proposed fusion method in terms of structure preservation, visual quality, and running efficiency.

Neural network input feature selection using structured l2 − norm penalization

Artificial neural networks are referred to as universal approximators due to their inherent ability to reconstruct complex linear and nonlinear output maps conceived as input-output relationships from data sets. This can be done by reducing large networks via regularization in order to establish compact models containing fewer parameters aimed at describing vital dependencies in data sets. In situations where the data sets contain non-informative input features, devising a continuous, optimal input feature selection technique can lead to improved prediction or classification. We propose a continuous input selection technique through a dimensional reduction mechanism using a ‘structured’ l2 − norm regularization. The implementation is done by identifying the most informative feature subsets from a given data set via an adaptive training mechanism. The adaptation involves introducing a novel, modified gradient approach during training to deal with the non-differentiability associated with the gradient of the structured norm penalty. When the method is applied to process data sets, results indicate that the most informative inputs of artificial neural networks can be selected using a structured l2 − norm penalization.

Two-timescale neurodynamic approaches to supervised feature selection based on alternative problem formulations

Feature selection is a crucial step in data processing and machine learning. While many greedy and sequential feature selection approaches are available, a holistic neurodynamics approach to supervised feature selection is recently developed via fractional programming by minimizing feature redundancy and maximizing relevance simultaneously. In view that the gradient of the fractional objective function is also fractional, alternative problem formulations are desirable to obviate the fractional complexity. In this paper, the fractional programming problem formulation is equivalently reformulated as bilevel and bilinear programming problems without using any fractional function. Two two-timescale projection neural networks are adapted for solving the reformulated problems. Experimental results on six benchmark datasets are elaborated to demonstrate the global convergence and high classification performance of the proposed neurodynamic approaches in comparison with six mainstream feature selection approaches.