Customer churn modeling in telecommunication using a novel multi-objective evolutionary clustering-based ensemble learning

Customer churn prediction is vital for organizations to mitigate costs and foster growth. Ensemble learning models are commonly used for churn prediction. Diversity and prediction performance are two essential principles for constructing ensemble classifiers. Therefore, developing accurate ensemble learning models consisting of diverse base classifiers is a considerable challenge in this area. In this study, we propose two multi-objective evolutionary ensemble learning models based on clustering (MOEECs), which are include a novel diversity measure. Also, to overcome the data imbalance problem, another objective function is presented in the second model to evaluate ensemble performance. The proposed models in this paper are evaluated with a dataset collected from a mobile operator database. Our first model, MOEEC-1, achieves an accuracy of 97.30% and an AUC of 93.76%, outperforming classical classifiers and other ensemble models. Similarly, MOEEC-2 attains an accuracy of 96.35% and an AUC of 94.89%, showcasing its effectiveness in churn prediction. Furthermore, comparison with previous churn models reveals that MOEEC-1 and MOEEC-2 exhibit superior performance in accuracy, precision, and F-score. Overall, our proposed MOEECs demonstrate significant advancements in churn prediction accuracy and outperform existing models in terms of key performance metrics. These findings underscore the efficacy of our approach in addressing the challenges of customer churn prediction and its potential for practical application in organizational decision-making.

A classification algorithm based on dynamic ensemble selection to predict mutational patterns of the envelope protein in HIV-infected patients

BACKGROUND

Therapeutics against the envelope (Env) proteins of human immunodeficiency virus type 1 (HIV-1) effectively reduce viral loads in patients. However, due to mutations, new therapy-resistant Env variants frequently emerge. The sites of mutations on Env that appear in each patient are considered random and unpredictable. Here we developed an algorithm to estimate for each patient the mutational state of each position based on the mutational state of adjacent positions on the three-dimensional structure of the protein.

METHODS

We developed a dynamic ensemble selection algorithm designated k-best classifiers. It identifies the best classifiers within the neighborhood of a new observation and applies them to predict the variability state of each observation. To evaluate the algorithm, we applied amino acid sequences of Envs from 300 HIV-1-infected individuals (at least six sequences per patient). For each patient, amino acid variability values at all Env positions were mapped onto the three-dimensional structure of the protein. Then, the variability state of each position was estimated by the variability at adjacent positions of the protein.

RESULTS

The proposed algorithm showed higher performance than the base learner and a panel of classification algorithms. The mutational state of positions in the high-mannose patch and CD4-binding site of Env, which are targeted by multiple therapeutics, was predicted well. Importantly, the algorithm outperformed other classification techniques for predicting the variability state at multi-position footprints of therapeutics on Env.

CONCLUSIONS

The proposed algorithm applies a dynamic classifier-scoring approach that increases its performance relative to other classification methods. Better understanding of the spatiotemporal patterns of variability across Env may lead to new treatment strategies that are tailored to the unique mutational patterns of each patient. More generally, we propose the algorithm as a new high-performance dynamic ensemble selection technique.

Multicluster Class-Balanced Ensemble

Ensemble classifiers using clustering have significantly improved classification and prediction accuracies of many systems. These types of ensemble approaches create multiple clusters to train the base classifiers. However, the problem with this is that each class might have many clusters and each cluster might have different number of samples, so an ensemble decision based on large number of clusters and different number of samples per class within a cluster produces biased and inaccurate results. Therefore, in this article, we propose a novel methodology to create an appropriate number of strong data clusters for each class and then balance them. Furthermore, an ensemble framework is proposed with base classifiers trained on strong and balanced data clusters. The proposed approach is implemented and evaluated on 24 benchmark data sets from the University of California Irvine (UCI) machine learning repository. An analysis of results using the proposed approach and the existing state-of-the-art ensemble classifier approaches is conducted and presented. A significance test is conducted to further validate the efficacy of the results and a detailed analysis is presented.

Multiple Elimination of Base Classifiers in Ensemble Learning Using Accuracy and Diversity Comparisons

When generating ensemble classifiers, selecting the best set of classifiers from the base classifier pool is considered a combinatorial problem and an efficient classifier selection methodology must be utilized. Different researchers have used different strategies such as evolutionary algorithms, genetic algorithms, rule-based algorithms, simulated annealing, and so forth to select the best set of classifiers that can maximize overall ensemble classifier accuracy. In this article, we present a novel classifier selection approach to generate an ensemble classifier. The proposed approach selects classifiers in multiple rounds of elimination. In each round, a classifier is given a chance to be selected to become a part of the ensemble, if it can contribute to the overall ensemble accuracy or diversity; otherwise, it is put back into the pool. Each classifier is given multiple opportunities to participate in rounds of selection and they are discarded only if they have no remaining chances. The process is repeated until no classifier in the pool has any chance left to participate in the round of selection. To test the efficacy of the proposed approach, 13 benchmark datasets from the UCI repository are used and results are compared with single classifier models and existing state-of-the-art ensemble classifier approaches. Statistical significance testing is conducted to further validate the results, and an analysis is provided.

Designing machine learning workflows with an application to topological data analysis

In this paper we define the concept of the Machine Learning Morphism (MLM) as a fundamental building block to express operations performed in machine learning such as data preprocessing, feature extraction, and model training. Inspired by statistical learning, MLMs are morphisms whose parameters are minimized via a risk function. We explore operations such as composition of MLMs and when sets of MLMs form a vector space. These operations are used to build a machine learning workflow from data preprocessing to final task completion. We examine the Mapper Algorithm from Topological Data Analysis as an MLM, and build several workflows for binary classification incorporating Mapper on Hospital Readmissions and Credit Evaluation datasets. The advantage of this framework lies in the ability to easily build, organize, and compare multiple workflows, and allows joint optimization of parameters across multiple steps in an application.

Pruning High-Similarity Clusters to Optimize Data Diversity when Building Ensemble Classifiers

Diversity is a key component for building a successful ensemble classifier. One approach to diversifying the base classifiers in an ensemble classifier is to diversify the data they are trained on. While sampling approaches such as bagging have been used for this task in the past, we argue that since they maintain the global distribution, they do not create diversity. Instead, we make a principled argument for the use of [Formula: see text]-means clustering to create diversity. Expanding on previous work, we observe that when creating multiple clusterings with multiple [Formula: see text] values, there is a risk of different clusterings discovering the same clusters, which would in turn train the same base classifiers. This would bias the ensemble voting process. We propose a new approach that uses the Jaccard Index to detect and remove similar clusters before training the base classifiers, not only saving computation time, but also reducing classification error by removing repeated votes. We empirically demonstrate the effectiveness of the proposed approach compared to the state of the art on 19 UCI benchmark datasets.

Hybrid Incremental Ensemble Learning for Noisy Real-World Data Classification

Traditional ensemble learning approaches explore the feature space and the sample space, respectively, which will prevent them to construct more powerful learning models for noisy real-world dataset classification. The random subspace method only search for the selection of features. Meanwhile, the bagging approach only search for the selection of samples. To overcome these limitations, we propose the hybrid incremental ensemble learning (HIEL) approach which takes into consideration the feature space and the sample space simultaneously to handle noisy dataset. Specifically, HIEL first adopts the bagging technique and linear discriminant analysis to remove noisy attributes, and generates a set of bootstraps and the corresponding ensemble members in the subspaces. Then, the classifiers are selected incrementally based on a classifier-specific criterion function and an ensemble criterion function. The corresponding weights for the classifiers are assigned during the same process. Finally, the final label is summarized by a weighted voting scheme, which serves as the final result of the classification. We also explore various classifier-specific criterion functions based on different newly proposed similarity measures, which will alleviate the effect of noisy samples on the distance functions. In addition, the computational cost of HIEL is analyzed theoretically. A set of nonparametric tests are adopted to compare HIEL and other algorithms over several datasets. The experiment results show that HIEL performs well on the noisy datasets. HIEL outperforms most of the compared classifier ensemble methods on 14 out of 24 noisy real-world UCI and KEEL datasets.

Wind Power Prediction Using Cluster Based Ensemble Regression

Accurate prediction of wind power is of vital importance for demand management. In this paper, we adopt a cluster-based ensemble framework to predict wind power. Natural groups/clusters exist in datasets and learning algorithms benefit from group/cluster wise learning — a philosophy that is not well explored for wind power prediction. The research presented in this paper investigates this philosophy to predict wind power by using an ensemble of regression models on natural clusters within wind data. We have conducted a series of experiments on a large number of locations across Australia and analyzed the existence of clusters within wind data, suitability of linear and nonlinear regression models for the proposed framework, and how well the cluster-based ensemble performs against the situation when no clustering is done. Experimental results demonstrate prediction improvement as high as 17.94% through the usage of the cluster-based ensemble regression algorithm.

A New Kind of Nonparametric Test for Statistical Comparison of Multiple Classifiers Over Multiple Datasets

Nonparametric statistical analysis, such as the Friedman test (FT), is gaining more and more attention due to its useful applications in a lot of experimental studies. However, traditional FT for the comparison of multiple learning algorithms on different datasets adopts the naive ranking approach. The ranking is based on the average accuracy values obtained by the set of learning algorithms on the datasets, which neither considers the differences of the results obtained by the learning algorithms on each dataset nor takes into account the performance of the learning algorithms in each run. In this paper, we will first propose three kinds of ranking approaches, which are the weighted ranking approach, the global ranking approach (GRA), and the weighted GRA. Then, a theoretical analysis is performed to explore the properties of the proposed ranking approaches. Next, a set of the modified FTs based on the proposed ranking approaches are designed for the comparison of the learning algorithms. Finally, the modified FTs are evaluated through six classifier ensemble approaches on 34 real-world datasets. The experiments show the effectiveness of the modified FTs.

Multimodal Degradation Prognostics Based on Switching Kalman Filter Ensemble

For accurate prognostics, users have to determine the current health of the system and predict future degradation pattern of the system. An increasingly popular approach toward tackling prognostic problems involves the use of switching models to represent various degradation phases, which the system undergoes. Such approaches have the advantage of determining the exact degradation phase of the system and being able to handle nonlinear degradation models through piecewise linear approximation. However, limitations of such existing methods include, limited applicability due to the discretization of predicted remaining useful life, insufficient robustness due to the use of single models and others. This paper circumvents these limitations by proposing a hybrid of ensemble methods with switching methods. The proposed method first implements a switching Kalman filter (SKF) to classify between various linear degradation phases, then predict the future propagation of fault dimension using appropriate Kalman filters for each phase. This proposed method achieves both continuous and discrete prediction values representing the remaining life and degradation phase of the system, respectively. The proposed framework is shown via a case study on benchmark simulated aeroengine data sets. The evaluation of the proposed framework shows that the proposed method achieves better accuracy and robustness against noise compared with other methods reported in the literature. The results also indicate the effectiveness of the SKF in detecting the switching point between various degradation modes.

A Cluster-Based Data Balancing Ensemble Classifier for Response Modeling in Bank Direct Marketing

The aim of direct marketing is to find the right customers who are most likely to respond to marketing campaign messages. In order to detect which customers are most valuable, response modeling is used to classify customers as respondent or non-respondent using their purchase history information or other behavioral characteristics. Data mining techniques, including effective classification methods, can be used to predict responsive customers. However, the inherent problem of imbalanced data in response modeling brings some difficulties into response prediction. As a result, the prediction models will be biased towards non-respondent customers. Another problem is that single models cannot provide the desired high accuracy due to their internal limitations. In this paper, we propose an ensemble classification method which removes imbalance in the data, using a combination of clustering and under-sampling. The predictions of multiple classifiers are combined in order to achieve better results. Using data from a bank’s marketing campaigns, this ensemble method is implemented on different classification techniques and the results are evaluated. We also evaluate the performance of this ensemble method against two alternative ensembles. The experimental results demonstrate that our proposed method can improve the performance of the response models for bank direct marketing by raising prediction accuracy and increasing response rate.

Hybrid adaptive classifier ensemble

Traditional random subspace-based classifier ensemble approaches (RSCE) have several limitations, such as viewing the same importance for the base classifiers trained in different subspaces, not considering how to find the optimal random subspace set. In this paper, we design a general hybrid adaptive ensemble learning framework (HAEL), and apply it to address the limitations of RSCE. As compared with RSCE, HAEL consists of two adaptive processes, i.e., base classifier competition and classifier ensemble interaction, so as to adjust the weights of the base classifiers in each ensemble and to explore the optimal random subspace set simultaneously. The experiments on the real-world datasets from the KEEL dataset repository for the classification task and the cancer gene expression profiles show that: 1) HAEL works well on both the real-world KEEL datasets and the cancer gene expression profiles and 2) it outperforms most of the state-of-the-art classifier ensemble approaches on 28 out of 36 KEEL datasets and 6 out of 6 cancer datasets.

Heuristic ternary error-correcting output codes via weight optimization and layered clustering-based approach

One important classifier ensemble for multiclass classification problems is error-correcting output codes (ECOCs). It bridges multiclass problems and binary-class classifiers by decomposing multiclass problems to a serial binary-class problems. In this paper, we present a heuristic ternary code, named weight optimization and layered clustering-based ECOC (WOLC-ECOC). It starts with an arbitrary valid ECOC and iterates the following two steps until the training risk converges. The first step, named layered clustering-based ECOC (LC-ECOC), constructs multiple strong classifiers on the most confusing binary-class problem. The second step adds the new classifiers to ECOC by a novel optimized weighted (OW) decoding algorithm, where the optimization problem of the decoding is solved by the cutting plane algorithm. Technically, LC-ECOC makes the heuristic training process not blocked by some difficult binary-class problem. OW decoding guarantees the nonincrease of the training risk for ensuring a small code length. Results on 14 UCI datasets and a music genre classification problem demonstrate the effectiveness of WOLC-ECOC.

RELATIONSHIP BETWEEN DATA SIZE, ACCURACY, DIVERSITY AND CLUSTERS IN NEURAL NETWORK ENSEMBLES

This paper presents an approach for analyzing relationships between data size, cluster, accuracy and diversity in neural network ensembles. The main objective of this research is to find out the influence of data size such as number of patterns, number of inputs and number of classes on various parameters such as clusters, accuracy and diversity of a neural network ensemble. The proposed approach is based on splitting data sets into different groups using the data size, clustering data and conducting training and testing of neural network ensembles. The test data is same for all groups and used to test all trained ensembles. The experiments have been conducted on 15 UCI machine learning benchmark datasets and results are presented in this paper.

CLUSTER BASED ENSEMBLE CLASSIFIER GENERATION BY JOINT OPTIMIZATION OF ACCURACY AND DIVERSITY

This paper presents an algorithm to generate ensemble classifier by joint optimization of accuracy and diversity. It is expected that the base classifiers in an ensemble are accurate and diverse (i.e., complementary in terms of errors) among each other for the ensemble classifier to be more accurate. We adopt a multi-objective evolutionary algorithm (MOEA) for joint optimization of accuracy and diversity on our recently developed nonuniform layered cluster oriented ensemble classifier (NULCOEC). In NULCOEC, the data set is partitioned into a variable number of clusters at different layers. Base classifiers are then trained on the clusters at different layers. The performance of NULCOEC is a function of the vector of the number of layers and clusters. The research presented in this paper investigates the implication of applying MOEA to generate NULCOEC. Accuracy and diversity of the ensemble classifier is expressed as a function of layers and clusters. A MOEA then searches for the combination of layers and clusters to obtain the nondominated set of (accuracy, diversity). We have obtained the results of single objective optimization (i.e., optimizing either accuracy or diversity) and compared them with the results of MOEA on sixteen UCI data sets. The results show that the MOEA can improve the performance of ensemble classifier.

SSC: a classifier combination method based on signal strength

We propose a new classifier combination method, the signal strength-based combining (SSC) approach, to combine the outputs of multiple classifiers to support the decision-making process in classification tasks. As ensemble learning methods have attracted growing attention from both academia and industry recently, it is critical to understand the fundamental issues of the combining rule. Motivated by the signal strength concept, our proposed SSC algorithm can effectively integrate the individual vote from different classifiers in an ensemble learning system. Comparative studies of our method with nine major existing combining rules, namely, geometric average rule, arithmetic average rule, median value rule, majority voting rule, Borda count, max and min rule, weighted average, and weighted majority voting rules, is presented. Furthermore, we also discuss the relationship of the proposed method with respect to margin-based classifiers, including the boosting method (AdaBoost.M1 and AdaBoost.M2) and support vector machines by margin analysis. Detailed analyses of margin distribution graphs are presented to discuss the characteristics of the proposed method. Simulation results for various real-world datasets illustrate the effectiveness of the proposed method.