Class-Imbalanced Deep Learning via a Class-Balanced Ensemble

Optimizing lipocalin sequence classification with ensemble deep learning models

Deep learning (DL) has become a powerful tool for the recognition and classification of biological sequences. However, conventional single-architecture models often struggle with suboptimal predictive performance and high computational costs. To address these challenges, we present EnsembleDL-Lipo, an innovative ensemble deep learning framework that combines Convolutional Neural Networks (CNNs) and Deep Neural Networks (DNNs) to enhance the identification of lipocalin sequences. Lipocalins are multifunctional extracellular proteins involved in various diseases and stress responses, and their low sequence similarity and occurrence in the 'twilight zone' of sequence alignment present significant hurdles for accurate classification. These challenges necessitate efficient computational methods to complement traditional, labor-intensive experimental approaches. EnsembleDL-Lipo overcomes these issues by leveraging a set of PSSM-based features to train a large ensemble of deep learning models. The framework integrates multiple feature representations derived from position-specific scoring matrices (PSSMs), optimizing classification performance across diverse sequence patterns. The model achieved superior results on the training dataset, with an accuracy (ACC) of 97.65%, recall of 97.10%, Matthews correlation coefficient (MCC) of 0.95, and area under the curve (AUC) of 0.99. Validation on an independent test set further confirmed the robustness of the model, yielding an ACC of 95.79%, recall of 90.48%, MCC of 0.92, and AUC of 0.97. These results demonstrate that EnsembleDL-Lipo is a highly effective and computationally efficient tool for lipocalin sequence identification, significantly outperforming existing methods and offering strong potential for applications in biomarker discovery.

Ensemble Prototype Network For Weakly Supervised Temporal Action Localization

Weakly supervised temporal action localization (TAL) aims to localize the action instances in untrimmed videos using only video-level action labels. Without snippet-level labels, this task should be hard to distinguish all snippets with accurate action/background categories. The main difficulties are the large variations brought by the unconstraint background snippets and multiple subactions in action snippets. The existing prototype model focuses on describing snippets by covering them with clusters (defined as prototypes). In this work, we argue that the clustered prototype covering snippets with simple variations still suffers from the misclassification of the snippets with large variations. We propose an ensemble prototype network (EPNet), which ensembles prototypes learned with consensus-aware clustering. The network stacks a consensus prototype learning (CPL) module and an ensemble snippet weight learning (ESWL) module as one stage and extends one stage to multiple stages in an ensemble learning way. The CPL module learns the consensus matrix by estimating the similarity of clustering labels between two successive clustering generations. The consensus matrix optimizes the clustering to learn consensus prototypes, which can predict the snippets with consensus labels. The ESWL module estimates the weights of the misclassified snippets using the snippet-level loss. The weights update the posterior probabilities of the snippets in the clustering to learn prototypes in the next stage. We use multiple stages to learn multiple prototypes, which can cover the snippets with large variations for accurate snippet classification. Extensive experiments show that our method achieves the state-of-the-art weakly supervised TAL methods on two benchmark datasets, that is, THUMOS'14, ActivityNet v1.2, and ActivityNet v1.3 datasets.

PAFormer: Anomaly Detection of Time Series With Parallel-Attention Transformer

Time-series anomaly detection is a critical task with significant impact as it serves a pivotal role in the field of data mining and quality management. Current anomaly detection methods are typically based on reconstruction or forecasting algorithms, as these methods have the capability to learn compressed data representations and model time dependencies. However, most methods rely on learning normal distribution patterns, which can be difficult to achieve in real-world engineering applications. Furthermore, real-world time-series data is highly imbalanced, with a severe lack of representative samples for anomalous data, which can lead to model learning failure. In this article, we propose a novel end-to-end unsupervised framework called the parallel-attention transformer (PAFormer), which discriminates anomalies by modeling both the global characteristics and local patterns of time series. Specifically, we construct parallel-attention (PA), which includes two core modules: the global enhanced representation module (GERM) and the local perception module (LPM). GERM consists of two pattern units and a normalization module, with attention weights that indicate the relationship of each data point to the whole series (global). Due to the rarity of anomalous points, they have strong associations with adjacent data points. LPM is composed of a learnable Laplace kernel function that learns the neighborhood relevancies through the distributional properties of the kernel function (local). We employ the PA to learn the global-local distributional differences for each data point, which enables us to discriminate anomalies. Finally, we propose a two-stage adversarial loss to optimize the model. We conduct experiments on five public benchmark datasets (real-world datasets) and one synthetic dataset. The results show that PAFormer outperforms state-of-the-art baselines.

SCCAM: Supervised Contrastive Convolutional Attention Mechanism for Ante-Hoc Interpretable Fault Diagnosis With Limited Fault Samples

In real industrial processes, fault diagnosis methods are required to learn from limited fault samples since the procedures are mainly under normal conditions and the faults rarely occur. Although attention mechanisms have become increasingly popular for the task of fault diagnosis, the existing attention-based methods are still unsatisfying for the above practical applications. First, pure attention-based architectures like transformers need a substantial quantity of fault samples to offset the lack of inductive biases thus performing poorly under limited fault samples. Moreover, the poor fault classification dilemma further leads to the failure of the existing attention-based methods to identify the root causes. To develop a solution to the aforementioned problems, we innovatively propose a supervised contrastive convolutional attention mechanism (SCCAM) with ante-hoc interpretability, which solves the root cause analysis problem under limited fault samples for the first time. First, accurate classification results are obtained under limited fault samples. More specifically, we integrate the convolutional neural network (CNN) with attention mechanisms to provide strong intrinsic inductive biases of locality and spatial invariance, thereby strengthening the representational power under limited fault samples. In addition, we ulteriorly enhance the classification capability of the SCCAM method under limited fault samples by employing the supervised contrastive learning (SCL) loss. Second, a novel ante-hoc interpretable attention-based architecture is designed to directly obtain the root causes without expert knowledge. The convolutional block attention module (CBAM) is utilized to directly provide feature contributions behind each prediction thus achieving feature-level explanations. The proposed SCCAM method is testified on a continuous stirred tank heater (CSTH) and the Tennessee Eastman (TE) industrial process benchmark. Three common fault diagnosis scenarios are covered, including a balanced scenario for additional verification and two scenarios with limited fault samples (i.e., imbalanced scenario and long-tail scenario). The effectiveness of the presented SCCAM method is evidenced by the comprehensive results that show our method outperforms the state-of-the-art methods in terms of fault classification and root cause analysis.

Deep learning-based prediction of post-pancreaticoduodenectomy pancreatic fistula

Postoperative pancreatic fistula is a life-threatening complication with an unmet need for accurate prediction. This study was aimed to develop preoperative artificial intelligence-based prediction models. Patients who underwent pancreaticoduodenectomy were enrolled and stratified into model development and validation sets by surgery between 2016 and 2017 or in 2018, respectively. Machine learning models based on clinical and body composition data, and deep learning models based on computed tomographic data, were developed, combined by ensemble voting, and final models were selected comparison with earlier model. Among the 1333 participants (training, n = 881; test, n = 452), postoperative pancreatic fistula occurred in 421 (47.8%) and 134 (31.8%) and clinically relevant postoperative pancreatic fistula occurred in 59 (6.7%) and 27 (6.0%) participants in the training and test datasets, respectively. In the test dataset, the area under the receiver operating curve [AUC (95% confidence interval)] of the selected preoperative model for predicting all and clinically relevant postoperative pancreatic fistula was 0.75 (0.71-0.80) and 0.68 (0.58-0.78). The ensemble model showed better predictive performance than the individual ML and DL models.

Class-imbalanced complementary-label learning via weighted loss

Complementary-label learning (CLL) is widely used in weakly supervised classification, but it faces a significant challenge in real-world datasets when confronted with class-imbalanced training samples. In such scenarios, the number of samples in one class is considerably lower than in other classes, which consequently leads to a decline in the accuracy of predictions. Unfortunately, existing CLL approaches have not investigate this problem. To alleviate this challenge, we propose a novel problem setting that enables learning from class-imbalanced complementary labels for multi-class classification. To tackle this problem, we propose a novel CLL approach called Weighted Complementary-Label Learning (WCLL). The proposed method models a weighted empirical risk minimization loss by utilizing the class-imbalanced complementary labels, which is also applicable to multi-class imbalanced training samples. Furthermore, we derive an estimation error bound to provide theoretical assurance. To evaluate our approach, we conduct extensive experiments on several widely-used benchmark datasets and a real-world dataset, and compare our method with existing state-of-the-art methods. The proposed approach shows significant improvement in these datasets, even in the case of multiple class-imbalanced scenarios. Notably, the proposed method not only utilizes complementary labels to train a classifier but also solves the problem of class imbalance.

Linear-exponential loss incorporated deep learning for imbalanced classification

The class imbalance issue is a pretty common and enduring topic all the time. When encountering unbalanced data distribution, conventional methods are prone to classify minority samples as majority ones, which may cause severe consequences in reality. It is crucial yet challenging to cope with such problems. In this paper, inspired by our previous work, we borrow the linear-exponential (LINEX) loss function in statistics into deep learning for the first time and extend it into a multi-class form, denoted as DLINEX. Compared with existing loss functions in class imbalance learning (e.g., the weighted cross entropy-loss and the focal loss), DLINEX has an asymmetric geometry interpretation, which can adaptively focus more on the minority and hard-to-classify samples by solely adjusting one parameter. Besides, it simultaneously achieves between and within class diversities via caring about the inherent properties of each instance. As a result, DLINEX achieves 42.08% G-means on the CIFAR-10 dataset at the imbalance ratio of 200, 79.06% G-means on the HAM10000 dataset, 82.74% F1 on the DRIVE dataset, 83.93% F1 on the CHASEDB1 dataset and 79.55% F1 on the STARE dataset The quantitative and qualitative experiments convincingly demonstrate that DLINEX can work favorably in imbalanced classifications, either at the image-level or the pixel-level.

Consumer-grade UAV imagery facilitates semantic segmentation of species-rich savanna tree layers

Conventional forest inventories are labour-intensive. This limits the spatial extent and temporal frequency at which woody vegetation is usually monitored. Remote sensing provides cost-effective solutions that enable extensive spatial coverage and high sampling frequency. Recent studies indicate that convolutional neural networks (CNNs) can classify woody forests, plantations, and urban vegetation at the species level using consumer-grade unmanned aerial vehicle (UAV) imagery. However, whether such an approach is feasible in species-rich savanna ecosystems remains unclear. Here, we tested whether small data sets of high-resolution RGB orthomosaics suffice to train U-Net, FC-DenseNet, and DeepLabv3 + in semantic segmentation of savanna tree species. We trained these models on an 18-ha training area and explored whether models could be transferred across space and time. These models could recognise trees in adjacent (mean F1-Score = 0.68) and distant areas (mean F1-Score = 0.61) alike. Over time, a change in plant morphology resulted in a decrease of model accuracy. Our results show that CNN-based tree mapping using consumer-grade UAV imagery is possible in savanna ecosystems. Still, larger and more heterogeneous data sets can further improve model robustness to capture variation in plant morphology across time and space.

Searching for Optimal Oversampling to Process Imbalanced Data: Generative Adversarial Networks and Synthetic Minority Over-Sampling Technique

Classification problems due to data imbalance occur in many fields and have long been studied in the machine learning field. Many real-world datasets suffer from the issue of class imbalance, which occurs when the sizes of classes are not uniform; thus, data belonging to the minority class are likely to be misclassified. It is particularly important to overcome this issue when dealing with medical data because class imbalance inevitably arises due to incidence rates within medical datasets. This study adjusted the imbalance ratio (IR) within the National Biobank of Korea dataset “Epidemiologic data of Parkinson’s disease dementia patients” to values of 6.8 (raw data), 9, and 19 and compared four traditional oversampling methods with techniques using the conditional generative adversarial network (CGAN) and conditional tabular generative adversarial network (CTGAN). The results showed that when the classes were balanced with CGAN and CTGAN, they showed a better classification performance than the more traditional oversampling techniques based on the AUC and F1-score. We were able to expand the application scope of GAN, widely used in unstructured data, to structured data. We also offer a better solution for the imbalanced data problem and suggest future research directions.

MCFF-MTDDI: multi-channel feature fusion for multi-typed drug-drug interaction prediction

Adverse drug-drug interactions (DDIs) have become an increasingly serious problem in the medical and health system. Recently, the effective application of deep learning and biomedical knowledge graphs (KGs) have improved the DDI prediction performance of computational models. However, the problems of feature redundancy and KG noise also arise, bringing new challenges for researchers. To overcome these challenges, we proposed a Multi-Channel Feature Fusion model for multi-typed DDI prediction (MCFF-MTDDI). Specifically, we first extracted drug chemical structure features, drug pairs' extra label features, and KG features of drugs. Then, these different features were effectively fused by a multi-channel feature fusion module. Finally, multi-typed DDIs were predicted through the fully connected neural network. To our knowledge, we are the first to integrate the extra label information into KG-based multi-typed DDI prediction; besides, we innovatively proposed a novel KG feature learning method and a State Encoder to obtain target drug pairs' KG-based features which contained more abundant and more key drug-related KG information with less noise; furthermore, a Gated Recurrent Unit-based multi-channel feature fusion module was proposed in an innovative way to yield more comprehensive feature information about drug pairs, effectively alleviating the problem of feature redundancy. We experimented with four datasets in the multi-class and the multi-label prediction tasks to comprehensively evaluate the performance of MCFF-MTDDI for predicting interactions of known-known drugs, known-new drugs and new-new drugs. In addition, we further conducted ablation studies and case studies. All the results fully demonstrated the effectiveness of MCFF-MTDDI.

Classifier Ensemble Based on Multiview Optimization for High-Dimensional Imbalanced Data Classification

High-dimensional class imbalanced data have plagued the performance of classification algorithms seriously. Because of a large number of redundant/invalid features and the class imbalanced issue, it is difficult to construct an optimal classifier for high-dimensional imbalanced data. Classifier ensemble has attracted intensive attention since it can achieve better performance than an individual classifier. In this work, we propose a multiview optimization (MVO) to learn more effective and robust features from high-dimensional imbalanced data, based on which an accurate and robust ensemble system is designed. Specifically, an optimized subview generation (OSG) in MVO is first proposed to generate multiple optimized subviews from different scenarios, which can strengthen the classification ability of features and increase the diversity of ensemble members simultaneously. Second, a new evaluation criterion that considers the distribution of data in each optimized subview is developed based on which a selective ensemble of optimized subviews (SEOS) is designed to perform the subview selective ensemble. Finally, an oversampling approach is executed on the optimized view to obtain a new class rebalanced subset for the classifier. Experimental results on 25 high-dimensional class imbalanced datasets indicate that the proposed method outperforms other mainstream classifier ensemble methods.

Improving the Prediction of Persistent High Healthcare Utilizers: Using an Ensemble Methodology

Background

A small proportion of high-need patients persistently use the bulk of health care services and incur disproportionate costs. Population health management (PHM) programs often refer to these patients as persistent high utilizers (PHUs). Accurate PHU prediction enables PHM programs to better align scarce health care resources with high-need PHUs while generally improving outcomes. While prior research in PHU prediction has shown promise, traditional regression methods used in these studies have yielded limited accuracy.

Objective

We are seeking to improve PHU predictions with an ensemble approach in a retrospective observational study design using insurance claim records.

Methods

We defined a PHU as a patient with health care costs in the top 20% of all patients for 4 consecutive 6-month periods. We used 2013 claims data to predict PHU status in next 24 months. Our study population included 165,595 patients in the Johns Hopkins Health Care plan, with 8359 (5.1%) patients identified as PHUs in 2014 and 2015. We assessed the performance of several standalone machine learning methods and then an ensemble approach combining multiple models.

Results

The candidate ensemble with complement naïve Bayes and random forest layers produced increased sensitivity and positive predictive value (PPV; 49.0% and 50.3%, respectively) compared to logistic regression (46.8% and 46.1%, respectively).

Conclusions

Our results suggest that ensemble machine learning can improve prediction of care management needs. Improved PPV implies reduced incorrect referral of low-risk patients. With the improved sensitivity/PPV balance of this approach, resources may be directed more efficiently to patients needing them most.