A theoretical analysis of bagging as a linear combination of classifiers

Dynamic weighted residual ensemble learning for hyperspectral image classification driven by features and samples

Dynamic ensemble selection has emerged as a promising approach for hyperspectral image classification. However, selecting relevant features and informative samples remains a pressing challenge. To address this issue, we introduce two novel dynamic residual ensemble learning methods. The first proposed method is called multi-features driven dynamic weighted residuals ensemble learning (MF-DWRL). This method leverages various combinations of features to construct classifier pools that incorporate feature differences. The K-Nearest Neighbors algorithm is employed to establish the region of competence (RoC) in the dynamic ensemble selection process. By assessing the performance of the RoC, the feature sets that yield the highest classification accuracy are identified as the optimal feature combinations. Additionally, the classification accuracy is utilized as prior information to guide the residual adjustments of each classifier. The second method, known as features and samples double-driven dynamic weighted residual ensemble learning (FS-DWRL), further enhances the performance of the ensemble. This approach not only considers the selection of feature combinations but also takes into account the informative samples. By jointly optimizing the feature and sample selection processes, FS-DWRL achieves superior classification accuracy compared to existing state-of-the-art methods. To evaluate the effectiveness of the proposed methods, three hyperspectral datasets from China-WHU-Hi-HanChuan, WHU-Hi-LongKou, and WHU-Hi-HongHu-are used for classification experiments. For these datasets, the proposed methods achieve the highest classification accuracies of 90.57 %, 98.77 %, and 91.08 %, respectively. The MF-DWRL and FS-DWRL methods exhibit significant improvements in classification accuracy.

Analysis of Parkinson's Disease Using an Imbalanced-Speech Dataset by Employing Decision Tree Ensemble Methods

Parkinson's disease (PD) currently affects approximately 10 million people worldwide. The detection of PD positive subjects is vital in terms of disease prognostics, diagnostics, management and treatment. Different types of early symptoms, such as speech impairment and changes in writing, are associated with Parkinson disease. To classify potential patients of PD, many researchers used machine learning algorithms in various datasets related to this disease. In our research, we study the dataset of the PD vocal impairment feature, which is an imbalanced dataset. We propose comparative performance evaluation using various decision tree ensemble methods, with or without oversampling techniques. In addition, we compare the performance of classifiers with different sizes of ensembles and various ratios of the minority class and the majority class with oversampling and undersampling. Finally, we combine feature selection with best-performing ensemble classifiers. The result shows that AdaBoost, random forest, and decision tree developed for the RUSBoost imbalanced dataset perform well in performance metrics such as precision, recall, F1-score, area under the receiver operating characteristic curve (AUROC) and the geometric mean. Further, feature selection methods, namely lasso and information gain, were used to screen the 10 best features using the best ensemble classifiers. AdaBoost with information gain feature selection method is the best performing ensemble method with an F1-score of 0.903.

Traffic Flow Prediction: An Intelligent Scheme for Forecasting Traffic Flow Using Air Pollution Data in Smart Cities with Bagging Ensemble

Traffic flow prediction is the most critical part of any traffic management system in a smart city. It can help a driver to pick the most optimized way to their target destination. Air pollution data are often connected with traffic congestion and there exists plenty of research on the connection between air pollution and traffic congestion using different machine learning approaches. A scheme for efficiently predicting traffic flow using ensemble techniques such as bagging and air pollution has not yet been introduced. Therefore, there is a need for a more accurate traffic flow prediction system for the smart cities. The aim of this research is to forecast traffic flow using pollution data. The contribution is twofold: Firstly, a comparison has been made using different simple regression techniques to find out the best-performing model. Secondly, bagging and stacking ensemble techniques have been used to find out the most accurate model of the two comparisons. The results show that the K-Nearest Neighbors (KNN) bagging ensemble provides far better results than all the other regression models used in this study. The experimental results show that the KNN bagging ensemble model reduces the error rate in predicting the traffic congestion by more than 30%.

Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers

The number of component classifiers chosen for an ensemble greatly impacts the prediction ability. In this paper, we use a geometric framework for a priori determining the ensemble size, which is applicable to most of the existing batch and online ensemble classifiers. There are only a limited number of studies on the ensemble size examining majority voting (MV) and weighted MV (WMV). Almost all of them are designed for batch-mode, hardly addressing online environments. Big data dimensions and resource limitations, in terms of time and memory, make the determination of ensemble size crucial, especially for online environments. For the MV aggregation rule, our framework proves that the more strong components we add to the ensemble, the more accurate predictions we can achieve. For the WMV aggregation rule, our framework proves the existence of an ideal number of components, which is equal to the number of class labels, with the premise that components are completely independent of each other and strong enough. While giving the exact definition for a strong and independent classifier in the context of an ensemble is a challenging task, our proposed geometric framework provides a theoretical explanation of diversity and its impact on the accuracy of predictions. We conduct a series of experimental evaluations to show the practical value of our theorems and existing challenges.

Dysphonic Voice Pattern Analysis of Patients in Parkinson's Disease Using Minimum Interclass Probability Risk Feature Selection and Bagging Ensemble Learning Methods

Analysis of quantified voice patterns is useful in the detection and assessment of dysphonia and related phonation disorders. In this paper, we first study the linear correlations between 22 voice parameters of fundamental frequency variability, amplitude variations, and nonlinear measures. The highly correlated vocal parameters are combined by using the linear discriminant analysis method. Based on the probability density functions estimated by the Parzen-window technique, we propose an interclass probability risk (ICPR) method to select the vocal parameters with small ICPR values as dominant features and compare with the modified Kullback-Leibler divergence (MKLD) feature selection approach. The experimental results show that the generalized logistic regression analysis (GLRA), support vector machine (SVM), and Bagging ensemble algorithm input with the ICPR features can provide better classification results than the same classifiers with the MKLD selected features. The SVM is much better at distinguishing normal vocal patterns with a specificity of 0.8542. Among the three classification methods, the Bagging ensemble algorithm with ICPR features can identify 90.77% vocal patterns, with the highest sensitivity of 0.9796 and largest area value of 0.9558 under the receiver operating characteristic curve. The classification results demonstrate the effectiveness of our feature selection and pattern analysis methods for dysphonic voice detection and measurement.

Recursive Feature Elimination with Ensemble Learning Using SOM Variants

To uncover an appropriate latent subspace for data representation, we propose in this paper a new extension of the random forests method which leads to the unsupervised feature selection called Feature Selection with Random Forests (RFS) based on SOM variants that evaluates the out-of-bag feature importance from a set of partitions. Every partition is created using a several bootstrap samples and a random features subset. We obtain empirical results on 19 benchmark datasets specifying that RFS, boosted with a recursive feature elimination (RFE) method, can lead to important enhancement in terms of clustering accuracy with a very restricted subset of features.

Simulations are performed on nine different benchmarks, including face data, handwritten digit data, and document data. Promising experimental results and theoretical analysis prove the efficiency and effectiveness of the proposed method for feature selection in comparison with competitive representative algorithms.

Analysis and Classification of Stride Patterns Associated with Children Development Using Gait Signal Dynamics Parameters and Ensemble Learning Algorithms

Measuring stride variability and dynamics in children is useful for the quantitative study of gait maturation and neuromotor development in childhood and adolescence. In this paper, we computed the sample entropy (SampEn) and average stride interval (ASI) parameters to quantify the stride series of 50 gender-matched children participants in three age groups. We also normalized the SampEn and ASI values by leg length and body mass for each participant, respectively. Results show that the original and normalized SampEn values consistently decrease over the significance level of the Mann-Whitney U test (p < 0.01) in children of 3-14 years old, which indicates the stride irregularity has been significantly ameliorated with the body growth. The original and normalized ASI values are also significantly changing when comparing between any two groups of young (aged 3-5 years), middle (aged 6-8 years), and elder (aged 10-14 years) children. Such results suggest that healthy children may better modulate their gait cadence rhythm with the development of their musculoskeletal and neurological systems. In addition, the AdaBoost.M2 and Bagging algorithms were used to effectively distinguish the children's gait patterns. These ensemble learning algorithms both provided excellent gait classification results in terms of overall accuracy (≥90%), recall (≥0.8), and precision (≥0.8077).

Hybrid adaptive classifier ensemble

Traditional random subspace-based classifier ensemble approaches (RSCE) have several limitations, such as viewing the same importance for the base classifiers trained in different subspaces, not considering how to find the optimal random subspace set. In this paper, we design a general hybrid adaptive ensemble learning framework (HAEL), and apply it to address the limitations of RSCE. As compared with RSCE, HAEL consists of two adaptive processes, i.e., base classifier competition and classifier ensemble interaction, so as to adjust the weights of the base classifiers in each ensemble and to explore the optimal random subspace set simultaneously. The experiments on the real-world datasets from the KEEL dataset repository for the classification task and the cancer gene expression profiles show that: 1) HAEL works well on both the real-world KEEL datasets and the cancer gene expression profiles and 2) it outperforms most of the state-of-the-art classifier ensemble approaches on 28 out of 36 KEEL datasets and 6 out of 6 cancer datasets.

Bagging and boosting variants for handling classifications problems: a survey

Bagging and boosting are two of the most well-known ensemble learning methods due to their theoretical performance guarantees and strong experimental results. Since bagging and boosting are an effective and open framework, several researchers have proposed their variants, some of which have turned out to have lower classification error than the original versions. This paper tried to summarize these variants and categorize them into groups. We hope that the references cited cover the major theoretical issues, and provide access to the main branches of the literature dealing with such methods, guiding the researcher in interesting research directions.

ELITE: ensemble of optimal input-pruned neural networks using TRUST-TECH

The ensemble of optimal input-pruned neural networks using TRUST-TECH (ELITE) method for constructing high-quality ensemble through an optimal linear combination of accurate and diverse neural networks is developed. The optimization problems in the proposed methodology are solved by a global optimization a global optimization method called TRansformation Under Stability-reTraining Equilibrium Characterization (TRUST-TECH), whose main features include its capability in identifying multiple local optimal solutions in a deterministic, systematic, and tier-by-tier manner. ELITE creates a diverse population via a feature selection procedure of different local optimal neural networks obtained using tier-1 TRUST-TECH search. In addition, the capability of each input-pruned network is fully exploited through a TRUST-TECH-based optimal training. Finally, finding the optimal linear combination weights for an ensemble is modeled as a nonlinear programming problem and solved using TRUST-TECH and the interior point method, where the issue of non-convexity can be effectively handled. Extensive numerical experiments have been carried out for pattern classification on the synthetic and benchmark datasets. Numerical results show that ELITE consistently outperforms existing methods on the benchmark datasets. The results show that ELITE can be very promising for constructing high-quality neural network ensembles.