A Robust Alternative to the Schemper-Henderson Estimator of Prediction Error

The natural course of binge-eating disorder: findings from a prospective, community-based study of adults

BACKGROUND

Epidemiological data offer conflicting views of the natural course of binge-eating disorder (BED), with large retrospective studies suggesting a protracted course and small prospective studies suggesting a briefer duration. We thus examined changes in BED diagnostic status in a prospective, community-based study that was larger and more representative with respect to sex, age of onset, and body mass index (BMI) than prior multi-year prospective studies.

METHODS

Probands and relatives with current DSM-IV BED (n = 156) from a family study of BED ('baseline') were selected for follow-up at 2.5 and 5 years. Probands were required to have BMI > 25 (women) or >27 (men). Diagnostic interviews and questionnaires were administered at all timepoints.

RESULTS

Of participants with follow-up data (n = 137), 78.1% were female, and 11.7% and 88.3% reported identifying as Black and White, respectively. At baseline, their mean age was 47.2 years, and mean BMI was 36.1. At 2.5 (and 5) years, 61.3% (45.7%), 23.4% (32.6%), and 15.3% (21.7%) of assessed participants exhibited full, sub-threshold, and no BED, respectively. No participants displayed anorexia or bulimia nervosa at follow-up timepoints. Median time to remission (i.e. no BED) exceeded 60 months, and median time to relapse (i.e. sub-threshold or full BED) after remission was 30 months. Two classes of machine learning methods did not consistently outperform random guessing at predicting time to remission from baseline demographic and clinical variables.

CONCLUSIONS

Among community-based adults with higher BMI, BED improves with time, but full remission often takes many years, and relapse is common.

Optimizing cardiovascular disease mortality prediction: a super learner approach in the tehran lipid and glucose study

BACKGROUND & AIM

Cardiovascular disease (CVD) is the most important cause of death in the world and has a potential impact on health care costs, this study aimed to evaluate the performance of machine learning survival models and determine the optimum model for predicting CVD-related mortality.

METHOD

In this study, the research population was all participants in Tehran Lipid and Glucose Study (TLGS) aged over 30 years. We used the Gradient Boosting model (GBM), Support Vector Machine (SVM), Super Learner (SL), and Cox proportional hazard (Cox-PH) models to predict the CVD-related mortality using 26 features. The dataset was randomly divided into training (80%) and testing (20%). To evaluate the performance of the methods, we used the Brier Score (BS), Prediction Error (PE), Concordance Index (C-index), and time-dependent Area Under the Curve (TD-AUC) criteria. Four different clinical models were also performed to improve the performance of the methods.

RESULTS

Out of 9258 participants with a mean age of (SD; range) 43.74 (15.51; 20-91), 56.60% were female. The CVD death proportion was 2.5% (228 participants). The death proportion was significantly higher in men (67.98% M, 32.02% F). Based on predefined selection criteria, the SL method has the best performance in predicting CVD-related mortality (TD-AUC > 93.50%). Among the machine learning (ML) methods, The SVM has the worst performance (TD-AUC = 90.13%). According to the relative effect, age, fasting blood sugar, systolic blood pressure, smoking, taking aspirin, diastolic blood pressure, Type 2 diabetes mellitus, hip circumference, body mss index (BMI), and triglyceride were identified as the most influential variables in predicting CVD-related mortality.

CONCLUSION

According to the results of our study, compared to the Cox-PH model, Machine Learning models showed promising and sometimes better performance in predicting CVD-related mortality. This finding is based on the analysis of a large and diverse urban population from Tehran, Iran.

SurvBenchmark: comprehensive benchmarking study of survival analysis methods using both omics data and clinical data

Survival analysis is a branch of statistics that deals with both the tracking of time and the survival status simultaneously as the dependent response. Current comparisons of survival model performance mostly center on clinical data with classic statistical survival models, with prediction accuracy often serving as the sole metric of model performance. Moreover, survival analysis approaches for censored omics data have not been thoroughly investigated. The common approach is to binarize the survival time and perform a classification analysis.

Here, we develop a benchmarking design, SurvBenchmark, that evaluates a diverse collection of survival models for both clinical and omics data sets. SurvBenchmark not only focuses on classical approaches such as the Cox model but also evaluates state-of-the-art machine learning survival models. All approaches were assessed using multiple performance metrics; these include model predictability, stability, flexibility, and computational issues. Our systematic comparison design with 320 comparisons (20 methods over 16 data sets) shows that the performances of survival models vary in practice over real-world data sets and over the choice of the evaluation metric. In particular, we highlight that using multiple performance metrics is critical in providing a balanced assessment of various models. The results in our study will provide practical guidelines for translational scientists and clinicians, as well as define possible areas of investigation in both survival technique and benchmarking strategies.

Fitting and Cross-Validating Cox Models to Censored Big Data With Missing Values Using Extensions of Partial Least Squares Regression Models

Fitting Cox models in a big data context -on a massive scale in terms of volume, intensity, and complexity exceeding the capacity of usual analytic tools-is often challenging. If some data are missing, it is even more difficult. We proposed algorithms that were able to fit Cox models in high dimensional settings using extensions of partial least squares regression to the Cox models. Some of them were able to cope with missing data. We were recently able to extend our most recent algorithms to big data, thus allowing to fit Cox model for big data with missing values. When cross-validating standard or extended Cox models, the commonly used criterion is the cross-validated partial loglikelihood using a naive or a van Houwelingen scheme -to make efficient use of the death times of the left out data in relation to the death times of all the data. Quite astonishingly, we will show, using a strong simulation study involving three different data simulation algorithms, that these two cross-validation methods fail with the extensions, either straightforward or more involved ones, of partial least squares regression to the Cox model. This is quite an interesting result for at least two reasons. Firstly, several nice features of PLS based models, including regularization, interpretability of the components, missing data support, data visualization thanks to biplots of individuals and variables -and even parsimony or group parsimony for Sparse partial least squares or sparse group SPLS based models, account for a common use of these extensions by statisticians who usually select their hyperparameters using cross-validation. Secondly, they are almost always featured in benchmarking studies to assess the performance of a new estimation technique used in a high dimensional or big data context and often show poor statistical properties. We carried out a vast simulation study to evaluate more than a dozen of potential cross-validation criteria, either AUC or prediction error based. Several of them lead to the selection of a reasonable number of components. Using these newly found cross-validation criteria to fit extensions of partial least squares regression to the Cox model, we performed a benchmark reanalysis that showed enhanced performances of these techniques. In addition, we proposed sparse group extensions of our algorithms and defined a new robust measure based on the Schmid score and the R coefficient of determination for least absolute deviation: the integrated R Schmid Score weighted. The R-package used in this article is available on the CRAN, http://cran.r-project.org/web/packages/plsRcox/index.html. The R package bigPLS will soon be available on the CRAN and, until then, is available on Github https://github.com/fbertran/bigPLS.

A validated pediatric disease risk index for allogeneic hematopoietic cell transplantation

A disease risk index (DRI) that was developed for adults with hematologic malignancy who were undergoing hematopoietic cell transplantation is also being used to stratify children and adolescents by disease risk. Therefore, to develop and validate a DRI that can be used to stratify those with AML and ALL by their disease risk, we analyzed 2569 patients aged <18 years with acute myeloid (AML; n = 1224) or lymphoblastic (ALL; n = 1345) leukemia who underwent hematopoietic cell transplantation. Training and validation subsets for each disease were generated randomly with 1:1 assignment to the subsets, and separate prognostic models were derived for each disease. For AML, 4 risk groups were identified based on age, cytogenetic risk, and disease status, including minimal residual disease status at transplantation. The 5-year leukemia-free survival for low (0 points), intermediate (2, 3, 5), high (7, 8), and very high (>8) risk groups was 78%, 53%, 40%, and 25%, respectively (P < .0001). For ALL, 3 risk groups were identified based on age and disease status, including minimal residual disease status at transplantation. The 5-year leukemia-free survival for low (0 points), intermediate (2-4), and high (≥5) risk groups was 68%, 51%, and 33%, respectively (P < .0001). We confirmed that the risk groups could be applied to overall survival, with 5-year survival ranging from 80% to 33% and 73% to 42% for AML and ALL, respectively (P < .0001). This validated pediatric DRI, which includes age and residual disease status, can be used to facilitate prognostication and stratification of children with AML and ALL for allogeneic transplantation.

mlr3proba: An R Package for Machine Learning in Survival Analysis

MOTIVATION

As machine learning has become increasingly popular over the last few decades, so too has the number of machine learning interfaces for implementing these models. Whilst many R libraries exist for machine learning, very few offer extended support for survival analysis. This is problematic considering its importance in fields like medicine, bioinformatics, economics, engineering, and more. mlr3proba provides a comprehensive machine learning interface for survival analysis and connects with mlr3's general model tuning and benchmarking facilities to provide a systematic infrastructure for survival modeling and evaluation.

AVAILABILITY

mlr3proba is available under an LGPL-3 license on CRAN and at https://github.com/mlr-org/mlr3proba, with further documentation at https://mlr3book.mlr-org.com/survival.html.

MEPHAS: an interactive graphical user interface for medical and pharmaceutical statistical analysis with R and Shiny

Background

Even though R is one of the most commonly used statistical computing environments, it lacks a graphical user interface (GUI) that appeals to students, researchers, lecturers, and practitioners in medicine and pharmacy for conducting standard data analytics. Current GUIs built on top of R, such as EZR and R-Commander, aim to facilitate R coding and visualization, but most of the functionalities are still accessed through a command-line interface (CLI). To assist practitioners of medicine and pharmacy and researchers to run most routines in fundamental statistical analysis, we developed an interactive GUI; i.e., MEPHAS, to support various web-based systems that are accessible from laptops, workstations, or tablets, under Windows, macOS (and IOS), or Linux. In addition to fundamental statistical analysis, advanced statistics such as the extended Cox regression and dimensional analyses including partial least squares regression (PLS-R) and sparse partial least squares regression (SPLS-R), are also available in MEPHAS.

Results

MEPHAS is a web-based GUI (https://alain003.phs.osaka-u.ac.jp/mephas/) that is based on a shiny framework. We also created the corresponding R package mephas (https://mephas.github.io/). Thus far, MEPHAS has supported four categories of statistics, including probability, hypothesis testing, regression models, and dimensional analyses. Instructions and help menus were accessible during the entire analytical process via the web-based GUI, particularly advanced dimensional data analysis that required much explanation. The GUI was designed to be intuitive for non-technical users to perform various statistical functions, e.g., managing data, customizing plots, setting parameters, and monitoring real-time results, without any R coding from users. All generated graphs can be saved to local machines, and tables can be downloaded as CSV files.

Conclusion

MEPHAS is a free and open-source web-interactive GUI that was designed to support statistical data analyses and prediction for medical and pharmaceutical practitioners and researchers. It enables various medical and pharmaceutical statistical analyses through interactive parameter settings and dynamic visualization of the results.

Survival forests under test: Impact of the proportional hazards assumption on prognostic and predictive forests for amyotrophic lateral sclerosis survival

We investigate the effect of the proportional hazards assumption on prognostic and predictive models of the survival time of patients suffering from amyotrophic lateral sclerosis. We theoretically compare the underlying model formulations of several variants of survival forests and implementations thereof, including random forests for survival, conditional inference forests, Ranger, and survival forests with L₁ splitting, with two novel variants, namely distributional and transformation survival forests. Theoretical considerations explain the low power of log-rank-based splitting in detecting patterns in non-proportional hazards situations in survival trees and corresponding forests. This limitation can potentially be overcome by the alternative split procedures suggested herein. We empirically investigated this effect using simulation experiments and a re-analysis of the Pooled Resource Open-Access ALS Clinical Trials database of amyotrophic lateral sclerosis survival, giving special emphasis to both prognostic and predictive models.

Quantifying and estimating the predictive accuracy for censored time-to-event data with competing risks

This paper focuses on quantifying and estimating the predictive accuracy of prognostic models for time-to-event outcomes with competing events. We consider the time-dependent discrimination and calibration metrics, including the receiver operating characteristics curve and the Brier score, in the context of competing risks. To address censoring, we propose a unified nonparametric estimation framework for both discrimination and calibration measures, by weighting the censored subjects with the conditional probability of the event of interest given the observed data. The proposed method can be extended to time-dependent predictive accuracy metrics constructed from a general class of loss functions. We apply the methodology to a data set from the African American Study of Kidney Disease and Hypertension to evaluate the predictive accuracy of a prognostic risk score in predicting end-stage renal disease, accounting for the competing risk of pre-end-stage renal disease death, and evaluate its numerical performance in extensive simulation studies.

Review and evaluation of performance measures for survival prediction models in external validation settings

BACKGROUND

When developing a prediction model for survival data it is essential to validate its performance in external validation settings using appropriate performance measures. Although a number of such measures have been proposed, there is only limited guidance regarding their use in the context of model validation. This paper reviewed and evaluated a wide range of performance measures to provide some guidelines for their use in practice.

METHODS

An extensive simulation study based on two clinical datasets was conducted to investigate the performance of the measures in external validation settings. Measures were selected from categories that assess the overall performance, discrimination and calibration of a survival prediction model. Some of these have been modified to allow their use with validation data, and a case study is provided to describe how these measures can be estimated in practice. The measures were evaluated with respect to their robustness to censoring and ease of interpretation. All measures are implemented, or are straightforward to implement, in statistical software.

RESULTS

Most of the performance measures were reasonably robust to moderate levels of censoring. One exception was Harrell's concordance measure which tended to increase as censoring increased.

CONCLUSIONS

We recommend that Uno's concordance measure is used to quantify concordance when there are moderate levels of censoring. Alternatively, Gönen and Heller's measure could be considered, especially if censoring is very high, but we suggest that the prediction model is re-calibrated first. We also recommend that Royston's D is routinely reported to assess discrimination since it has an appealing interpretation. The calibration slope is useful for both internal and external validation settings and recommended to report routinely. Our recommendation would be to use any of the predictive accuracy measures and provide the corresponding predictive accuracy curves. In addition, we recommend to investigate the characteristics of the validation data such as the level of censoring and the distribution of the prognostic index derived in the validation setting before choosing the performance measures.

Boosting the discriminatory power of sparse survival models via optimization of the concordance index and stability selection

Background

When constructing new biomarker or gene signature scores for time-to-event outcomes, the underlying aims are to develop a discrimination model that helps to predict whether patients have a poor or good prognosis and to identify the most influential variables for this task. In practice, this is often done fitting Cox models. Those are, however, not necessarily optimal with respect to the resulting discriminatory power and are based on restrictive assumptions. We present a combined approach to automatically select and fit sparse discrimination models for potentially high-dimensional survival data based on boosting a smooth version of the concordance index (C-index). Due to this objective function, the resulting prediction models are optimal with respect to their ability to discriminate between patients with longer and shorter survival times. The gradient boosting algorithm is combined with the stability selection approach to enhance and control its variable selection properties.

Results

The resulting algorithm fits prediction models based on the rankings of the survival times and automatically selects only the most stable predictors. The performance of the approach, which works best for small numbers of informative predictors, is demonstrated in a large scale simulation study: C-index boosting in combination with stability selection is able to identify a small subset of informative predictors from a much larger set of non-informative ones while controlling the per-family error rate. In an application to discover biomarkers for breast cancer patients based on gene expression data, stability selection yielded sparser models and the resulting discriminatory power was higher than with lasso penalized Cox regression models.

Conclusion

The combination of stability selection and C-index boosting can be used to select small numbers of informative biomarkers and to derive new prediction rules that are optimal with respect to their discriminatory power. Stability selection controls the per-family error rate which makes the new approach also appealing from an inferential point of view, as it provides an alternative to classical hypothesis tests for single predictor effects. Due to the shrinkage and variable selection properties of statistical boosting algorithms, the latter tests are typically unfeasible for prediction models fitted by boosting.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-016-1149-8) contains supplementary material, which is available to authorized users.

Explained variation for recurrent event data

Although there are many suggested measures of explained variation for single-event survival data, there has been little attention to explained variation for recurrent event data. We describe an existing rank-based measure and we investigate a new statistic based on observed and expected event count processes. Both methods can be used for all models. Adjustments for missing data are proposed and demonstrated through simulation to be effective. We compare the population values of the two statistics and illustrate their use in comparing an array of non-nested models for data on recurrent episodes of infant diarrhoea.

Deviance residuals-based sparse PLS and sparse kernel PLS regression for censored data

MOTIVATION

A vast literature from the past decade is devoted to relating gene profiles and subject survival or time to cancer recurrence. Biomarker discovery from high-dimensional data, such as transcriptomic or single nucleotide polymorphism profiles, is a major challenge in the search for more precise diagnoses. The proportional hazard regression model suggested by Cox (1972), to study the relationship between the time to event and a set of covariates in the presence of censoring is the most commonly used model for the analysis of survival data. However, like multivariate regression, it supposes that more observations than variables, complete data, and not strongly correlated variables are available. In practice, when dealing with high-dimensional data, these constraints are crippling. Collinearity gives rise to issues of over-fitting and model misidentification. Variable selection can improve the estimation accuracy by effectively identifying the subset of relevant predictors and enhance the model interpretability with parsimonious representation. To deal with both collinearity and variable selection issues, many methods based on least absolute shrinkage and selection operator penalized Cox proportional hazards have been proposed since the reference paper of Tibshirani. Regularization could also be performed using dimension reduction as is the case with partial least squares (PLS) regression. We propose two original algorithms named sPLSDR and its non-linear kernel counterpart DKsPLSDR, by using sparse PLS regression (sPLS) based on deviance residuals. We compared their predicting performance with state-of-the-art algorithms on both simulated and real reference benchmark datasets.

RESULTS

sPLSDR and DKsPLSDR compare favorably with other methods in their computational time, prediction and selectivity, as indicated by results based on benchmark datasets. Moreover, in the framework of PLS regression, they feature other useful tools, including biplots representation, or the ability to deal with missing data. Therefore, we view them as a useful addition to the toolbox of estimation and prediction methods for the widely used Cox's model in the high-dimensional and low-sample size settings.

AVAILABILITY AND IMPLEMENTATION

The R-package plsRcox is available on the CRAN and is maintained by Frédéric Bertrand. http://cran.r-project.org/web/packages/plsRcox/index.html.

CONTACT

pbastien@rd.loreal.com or fbertran@math.unistra.fr.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

A comparison of estimators to evaluate the discriminatory power of time-to-event models

Discrimination measures for continuous time-to-event outcomes have become an important tool in medical decision making. The idea behind discrimination measures is to evaluate the performance of a prediction model by measuring its ability to distinguish between observations having an event and those having no event. Researchers proposed a variety of approaches to estimate discrimination measures from a set of right-censored data. These approaches rely on different regularity assumptions that are needed to ensure consistency of the respective estimators. Typical examples of regularity assumptions include the proportional hazards assumption in Cox regression and the random censoring assumption. Because regularity assumptions are often violated in practice, conducting a sensitivity analysis of the estimators is of considerable interest. The aim of the paper is to analyze and to compare the most popular estimators of discrimination measures for event time outcomes. On the basis of the results of an extensive simulation study and the analysis of molecular data, we investigate the behavior of the estimators in situations where the underlying regularity assumptions do not hold. We show that violations of the regularity assumptions may induce a nonignorable bias and may therefore result in biased medical decision making.