Online Nonlinear AUC Maximization for Imbalanced Data Sets

Spherical model for Minimalist Machine Learning paradigm in handling complex databases

This paper presents the development of the N-Spherical Minimalist Machine Learning (MML) classifier, an innovative model within the Minimalist Machine Learning paradigm. Using N-spherical coordinates and concepts from metaheuristics and associative models, this classifier effectively addresses challenges such as data dimensionality and class imbalance in complex datasets. Performance evaluations using the F1 measure and balanced accuracy demonstrate its superior efficiency and robustness compared to state-of-the-art classifiers. Statistical validation is conducted using the Friedman and Holm tests. Although currently limited to binary classification, this work highlights the potential of minimalist approaches in machine learning for classification of highly dimensional and imbalanced data. Future extensions aim to include multi-class problems and mechanisms for handling categorical data.

New Scalable and Efficient Online Pairwise Learning Algorithm

Pairwise learning is an important machine-learning topic with many practical applications. An online algorithm is the first choice for processing streaming data and is preferred for handling large-scale pairwise learning problems. However, existing online pairwise learning algorithms are not scalable and efficient enough for large-scale high-dimensional data, because they were designed based on singly stochastic gradients. To address this challenging problem, in this article, we propose a dynamic doubly stochastic gradient algorithm (D2SG) for online pairwise learning. Especially, only the time and space complexities of are needed for incorporating a new sample, where is the dimensionality of data. This means that our D2SG is much faster and more scalable than the existing online pairwise learning algorithms while the statistical accuracy can be guaranteed through our rigorous theoretical analysis under standard assumptions. The experimental results on a variety of real-world datasets not only confirm the theoretical result of our new D2SG algorithm, but also show that D2SG has better efficiency and scalability than the existing online pairwise learning algorithms.

Class-Imbalanced-Aware Distantly Supervised Named Entity Recognition

Distantly supervised named entity recognition (NER), which automatically learns NER models without manually labeling data, has gained much attention recently. In distantly supervised NER, positive unlabeled (PU) learning methods have achieved notable success. However, existing PU learning-based NER methods are unable to automatically handle the class imbalance and further depend on the estimation of the unknown class prior; thus, the class imbalance and imperfect estimation of the class prior degenerate the NER performance. To address these issues, this article proposes a novel PU learning method for distantly supervised NER. The proposed method can automatically handle the class imbalance and does not need to engage in class prior estimation, which enables the proposed methods to achieve the state-of-the-art performance. Extensive experiments support our theoretical analysis and validate the superiority of our method.

AUCReshaping: improved sensitivity at high-specificity

The evaluation of deep-learning (DL) systems typically relies on the Area under the Receiver-Operating-Curve (AU-ROC) as a performance metric. However, AU-ROC, in its holistic form, does not sufficiently consider performance within specific ranges of sensitivity and specificity, which are critical for the intended operational context of the system. Consequently, two systems with identical AU-ROC values can exhibit significantly divergent real-world performance. This issue is particularly pronounced in the context of anomaly detection tasks, a commonly employed application of DL systems across various research domains, including medical imaging, industrial automation, manufacturing, cyber security, fraud detection, and drug research, among others. The challenge arises from the heavy class imbalance in training datasets, with the abnormality class often incurring a considerably higher misclassification cost compared to the normal class. Traditional DL systems address this by adjusting the weighting of the cost function or optimizing for specific points along the ROC curve. While these approaches yield reasonable results in many cases, they do not actively seek to maximize performance for the desired operating point. In this study, we introduce a novel technique known as AUCReshaping, designed to reshape the ROC curve exclusively within the specified sensitivity and specificity range, by optimizing sensitivity at a predetermined specificity level. This reshaping is achieved through an adaptive and iterative boosting mechanism that allows the network to focus on pertinent samples during the learning process. We primarily investigated the impact of AUCReshaping in the context of abnormality detection tasks, specifically in Chest X-Ray (CXR) analysis, followed by breast mammogram and credit card fraud detection tasks. The results reveal a substantial improvement, ranging from 2 to 40%, in sensitivity at high-specificity levels for binary classification tasks.

Fast AUC Maximization Learning Machine With Simultaneous Outlier Detection

While AUC maximizing support vector machine (AUCSVM) has been developed to solve imbalanced classification tasks, its huge computational burden will make AUCSVM become impracticable and even computationally forbidden for medium or large-scale imbalanced data. In addition, minority class sometimes means extremely important information for users or is corrupted by noises and/or outliers in practical application scenarios such as medical diagnosis, which actually inspires us to generalize the AUC concept to reflect such importance or upper bound of noises or outliers. In order to address these issues, by means of both the generalized AUC metric and the core vector machine (CVM) technique, a fast AUC maximizing learning machine, called ρ -AUCCVM, with simultaneous outlier detection is proposed in this study. ρ -AUCCVM has its notorious merits: 1) it indeed shares the CVM's advantage, that is, asymptotically linear time complexity with respect to the total number of sample pairs, together with space complexity independent on the total number of sample pairs and 2) it can automatically determine the importance of the minority class (assuming no noise) or the upper bound of noises or outliers. Extensive experimental results about benchmarking imbalanced datasets verify the above advantages of ρ -AUCCVM.

A Minimax Probability Machine for Nondecomposable Performance Measures

Imbalanced classification tasks are widespread in many real-world applications. For such classification tasks, in comparison with the accuracy rate (AR), it is usually much more appropriate to use nondecomposable performance measures such as the area under the receiver operating characteristic curve (AUC) and the F_β measure as the classification criterion since the label class is imbalanced. On the other hand, the minimax probability machine is a popular method for binary classification problems and aims at learning a linear classifier by maximizing the AR, which makes it unsuitable to deal with imbalanced classification tasks. The purpose of this article is to develop a new minimax probability machine for the F_β measure, called minimax probability machine for the F_β -measures (MPMF), which can be used to deal with imbalanced classification tasks. A brief discussion is also given on how to extend the MPMF model for several other nondecomposable performance measures listed in the article. To solve the MPMF model effectively, we derive its equivalent form which can then be solved by an alternating descent method to learn a linear classifier. Further, the kernel trick is employed to derive a nonlinear MPMF model to learn a nonlinear classifier. Several experiments on real-world benchmark datasets demonstrate the effectiveness of our new model.

A classification for complex imbalanced data in disease screening and early diagnosis

Imbalanced classification has drawn considerable attention in the statistics and machine learning literature. Typically, traditional classification methods often perform poorly when a severely skewed class distribution is observed, not to mention under a high-dimensional longitudinal data structure. Given the ubiquity of big data in modern health research, it is expected that imbalanced classification in disease diagnosis may encounter an additional level of difficulty that is imposed by such a complex data structure. In this article, we propose a nonparametric classification approach for imbalanced data in longitudinal and high-dimensional settings. Technically, the functional principal component analysis is first applied for feature extraction under the longitudinal structure. The univariate exponential loss function coupled with group LASSO penalty is then adopted into the classification procedure in high-dimensional settings. Along with a good improvement in imbalanced classification, our approach provides a meaningful feature selection for interpretation while enjoying a remarkably lower computational complexity. The proposed method is illustrated on the real data application of Alzheimer's disease early detection and its empirical performance in finite sample size is extensively evaluated by simulations.

Augmentation Method for High Intra-Class Variation Data in Apple Detection

Deep learning is widely used in modern orchard production for various inspection missions, which helps improve the efficiency of orchard operations. In the mission of visual detection during fruit picking, most current lightweight detection models are not yet effective enough to detect multi-type occlusion targets, severely affecting automated fruit-picking efficiency. This study addresses this problem by proposing the pioneering design of a multi-type occlusion apple dataset and an augmentation method of data balance. We divided apple occlusion into eight types and used the proposed method to balance the number of annotation boxes for multi-type occlusion apple targets. Finally, a validation experiment was carried out using five popular lightweight object detection models: yolox-s, yolov5-s, yolov4-s, yolov3-tiny, and efficidentdet-d0. The results show that, using the proposed augmentation method, the average detection precision of the five popular lightweight object detection models improved significantly. Specifically, the precision increased from 0.894 to 0.974, recall increased from 0.845 to 0.972, and mAP0.5 increased from 0.982 to 0.919 for yolox-s. This implies that the proposed augmentation method shows great potential for different fruit detection missions in future orchard applications.

Imbalanced Data Classification via Cooperative Interaction Between Classifier and Generator

Learning classifiers with imbalanced data can be strongly biased toward the majority class. To address this issue, several methods have been proposed using generative adversarial networks (GANs). Existing GAN-based methods, however, do not effectively utilize the relationship between a classifier and a generator. This article proposes a novel three-player structure consisting of a discriminator, a generator, and a classifier, along with decision boundary regularization. Our method is distinctive in which the generator is trained in cooperation with the classifier to provide minority samples that gradually expand the minority decision region, improving performance for imbalanced data classification. The proposed method outperforms the existing methods on real data sets as well as synthetic imbalanced data sets.

Large-Scale Nonlinear AUC Maximization via Triply Stochastic Gradients

Learning to improve AUC performance for imbalanced data is an important machine learning research problem. Most methods of AUC maximization assume that the model function is linear in the original feature space. However, this assumption is not suitable for nonlinear separable problems. Although there have been some nonlinear methods of AUC maximization, scaling up nonlinear AUC maximization is still an open question. To address this challenging problem, in this paper, we propose a novel large-scale nonlinear AUC maximization method (named as TSAM) based on the triply stochastic gradient descents. Specifically, we first use the random Fourier feature to approximate the kernel function. After that, we use the triply stochastic gradients w.r.t. the pairwise loss and random feature to iteratively update the solution. Finally, we prove that TSAM converges to the optimal solution with the rate of O(1/t) after t iterations. Experimental results on a variety of benchmark datasets not only confirm the scalability of TSAM, but also show a significant reduction of computational time compared with existing batch learning algorithms, while retaining the similar generalization performance.

AI Models for Predicting Readmission of Pneumonia Patients within 30 Days after Discharge

A model with capability for precisely predicting readmission is a target being pursued worldwide. The objective of this study is to design predictive models using artificial intelligence methods and data retrieved from the National Health Insurance Research Database of Taiwan for identifying high-risk pneumonia patients with 30-day all-cause readmissions. An integrated genetic algorithm (GA) and support vector machine (SVM), namely IGS, were used to design predictive models optimized with three objective functions. In IGS, GA was used for selecting salient features and optimal SVM parameters, while SVM was used for constructing the models. For comparison, logistic regression (LR) and deep neural network (DNN) were also applied for model construction. The IGS model with AUC used as the objective function achieved an accuracy, sensitivity, specificity, and area under ROC curve (AUC) of 70.11%, 73.46%, 69.26%, and 0.7758, respectively, outperforming the models designed with LR (65.77%, 78.44%, 62.54%, and 0.7689, respectively) and DNN (61.50%, 79.34%, 56.95%, and 0.7547, respectively), as well as previously reported models constructed using thedata of electronic health records with an AUC of 0.71–0.74. It can be used for automatically detecting pneumonia patients with a risk of all-cause readmissions within 30 days after discharge so as to administer suitable interventions to reduce readmission and healthcare costs.

MBA: Mini-Batch AUC Optimization

Area under the receiver operating characteristics curve (AUC) is an important metric for a wide range of machine-learning problems, and scalable methods for optimizing AUC have recently been proposed. However, handling very large data sets remains an open challenge for this problem. This article proposes a novel approach to AUC maximization based on sampling mini-batches of positive/negative instance pairs and computing U-statistics to approximate a global risk minimization problem. The resulting algorithm is simple, fast, and learning-rate free. We show that the number of samples required for good performance is independent of the number of pairs available, which is a quadratic function of the positive and negative instances. Extensive experiments show the practical utility of the proposed method.

Incorporating Particle Swarm Optimization into Improved Bacterial Foraging Optimization Algorithm Applied to Classify Imbalanced Data

In this paper, particle swarm optimization is incorporated into an improved bacterial foraging optimization algorithm, which is applied to classifying imbalanced data to solve the problem of how original bacterial foraging optimization easily falls into local optimization. In this study, the borderline synthetic minority oversampling technique (Borderline-SMOTE) and Tomek link are used to pre-process imbalanced data. Then, the proposed algorithm is used to classify the imbalanced data. In the proposed algorithm, firstly, the chemotaxis process is improved. The particle swarm optimization (PSO) algorithm is used to search first and then treat the result as bacteria, improving the global searching ability of bacterial foraging optimization (BFO). Secondly, the reproduction operation is improved and the selection standard of survival of the cost is improved. Finally, we improve elimination and dispersal operation, and the population evolution factor is introduced to prevent the population from stagnating and falling into a local optimum. In this paper, three data sets are used to test the performance of the proposed algorithm. The simulation results show that the classification accuracy of the proposed algorithm is better than the existing approaches.

Design of a Clinical Decision Support System for Predicting Erectile Dysfunction in Men Using NHIRD Dataset

Erectile dysfunction (ED) affects millions of men worldwide. Men with ED generally complain failure to attain or maintain an adequate erection during sexual activity. The prevalence of ED is strongly correlated with age, affecting about 40% of men at age 40 and nearly 70% at age 70. A variety of chronic diseases, including diabetes, ischemic heart disease, congestive heart failure, hypertension, depression, chronic renal failure, obstructive sleep apnea, prostate disease, gout, and sleep disorder, were reported to be associated with ED. In this study, data retrieved from a subset of the National Health Insurance Research Database of Taiwan were used for designing the clinical decision support system (CDSS) for predicting ED incidences in men. The positive cases were male patients aged 20-65 who were diagnosed with ED between January 2000 and December 2010 confirmed by at least three outpatient visits or at least one inpatient visit, while the negative cases were randomly selected from the database without a history of ED and were frequency (1:1), age, and index year matched with the ED patients. Data of a total of 2832 ED patients and 2832 non-ED patients, each consisting of 41 features including index age, 10 comorbidities, and 30 other comorbidity-related variables, were retrieved for designing the predictive models. Integrated genetic algorithm and support vector machine was adopted to design the CDSSs with two experiments of independent training and testing (ITT) conducted to verify their effectiveness. In the 1st ITT experiment, data extracted from January 2000 till December 2005 (61.51%, 1742 positive cases and 1742 negative cases) were used for training and validating and the data retrieved from January 2006 till December 2010 were used for testing (38.49%), whereas in the 2nd ITT experiment, data in the training set (77.78%) were extracted from January 2000 till Deceber 2007 and those in the testing set (22.22%) were retrieved afterward. Tenfold cross validation and three different objective functions were adopted for obtaining the optimal models with best predictive performance in the training phase. The testing results show that the CDSSs achieved a predictive performance with accuracy, sensitivity, specificity, g-mean, and area under ROC curve of 74.72%-76.65%, 72.33%-83.76%, 69.54%-77.10%, 0.7468-0.7632, and 0.766-0.817, respectively. In conclusion, the CDSSs designed based on cost-sensitive objective functions as well as salient comorbidity-related features achieve satisfactory predictive performance for predicting ED incidences.

Factorization machines and deep views-based co-training for improving answer quality prediction in online health expert question-answering services

In online health expert question-answering (HQA) services, it is significant to automatically determine the quality of the answers. There are two prominent challenges in this task. First, the answers are usually written in short text, which makes it difficult to absorb the text semantic information. Second, it usually lacks sufficient labeled data but contains a huge amount of unlabeled data. To tackle these challenges, we propose a novel deep co-training framework based on factorization machines (FM) and deep textual views to intelligently and automatically identify the quality of HQA systems. More specifically, we exploit additional domain-specific semantic information from domain-specific word embeddings to expand the semantic space of short text and apply FM to excavate the non-independent interaction relationships among diverse features within individual views for improving the performance of the base classifier via co-training. Our learned deep textual views, the convolutional neural networks (CNN) view which focuses on extracting local features using convolution filters to locally model short text and the dependency-sensitive convolutional neural networks (DSCNN) view which focuses on capturing long-distance dependency information within the text to globally model short text, can then overcome the challenge of feature sparseness in the short text answers from the doctors. The developed co-training framework can effectively mine the highly non-linear semantic information embedded in the unlabeled data and expose the highly non-linear relationships between different views, which minimizes the labeling effort. Finally, we conduct extensive empirical evaluations and demonstrate that our proposed method can significantly improve the predictive performance of the answer quality in the context of HQA services.

Design of a Clinical Decision Support System for Fracture Prediction Using Imbalanced Dataset

More than 1 billion people suffer from chronic respiratory diseases worldwide, accounting for more than 4 million deaths annually. Inhaled corticosteroid is a popular medication for treating chronic respiratory diseases. Its side effects include decreased bone mineral density and osteoporosis. The aims of this study are to investigate the association of inhaled corticosteroids and fracture and to design a clinical support system for fracture prediction. The data of patients aged 20 years and older, who had visited healthcare centers and been prescribed with inhaled corticosteroids within 2002-2010, were retrieved from the National Health Insurance Research Database (NHIRD). After excluding patients diagnosed with hip fracture or vertebrate fractures before using inhaled corticosteroid, a total of 11645 patients receiving inhaled corticosteroid therapy were included for this study. Among them, 1134 (9.7%) were diagnosed with hip fracture or vertebrate fracture. The statistical results showed that demographic information, chronic respiratory diseases and comorbidities, and corticosteroid-related variables (cumulative dose, mean exposed daily dose, follow-up duration, and exposed duration) were significantly different between fracture and nonfracture patients. The clinical decision support systems (CDSSs) were designed with integrated genetic algorithm (GA) and support vector machine (SVM) by training and validating the models with balanced training sets obtained by random and cluster-based undersampling methods and testing with the imbalanced NHIRD dataset. Two different objective functions were adopted for obtaining optimal models with best predictive performance. The predictive performance of the CDSSs exhibits a sensitivity of 69.84-77.00% and an AUC of 0.7495-0.7590. It was concluded that long-term use of inhaled corticosteroids may induce osteoporosis and exhibit higher incidence of hip or vertebrate fractures. The accumulated dose of ICS and OCS therapies should be continuously monitored, especially for patients with older age and women after menopause, to prevent from exceeding the maximum dosage.